Hands-on lab: Sentinel as code, end to end

Parts 1 and 2 explained the ideas. Here you do it, on your own subscription, as the implementer: build the workspace from code, ship detections through a reviewed pipeline, switch Sentinel settings on and off from Git, then catch someone changing a rule by hand.

What you will build

The detections watch the Azure Activity log: who created, changed or deleted what in the subscription. It is free to ingest, and it means the lab detects your own actions. The main rule fires when a person, rather than the pipeline, changes an analytic rule. In Lab 6, that person is you.

Lab 0 · Before you start

• An Azure subscription where you are Owner, in a tenant where you can sign in to the Defender portal. A personal pay-as-you-go subscription is ideal. Do not use a work tenant.
• A GitHub account. Azure DevOps is optional, for Lab 8.
• Azure Cloud Shell (Bash) for the one-time commands. Nothing needs installing on your laptop. Installing Terraform locally helps if you want to run terraform validate yourself.
• Cost: the workspace has a 0.5 GB daily ingestion cap, and the Activity log is free to ingest. With no other data sources, expect a few cents. The threat-intelligence connector in Lab 5 adds a small amount of data, still under the cap. Lab 9 deletes everything.

Lab 1 · Repository and bootstrap

Goal: a private repository with the lab files, and the three things the pipeline needs before it can run at all: state storage, an identity, and permissions.

1.1 Create your repository

1. On GitHub, create a new private repository, for example sentinel-as-code-lab. Do not add a README.
2. Copy the lab files into it:

git clone --depth 1 https://github.com/iamkaushiksaha/learning-hub.git /tmp/hub
git clone https://github.com/<you>/sentinel-as-code-lab.git
cp -R /tmp/hub/examples/sentinel-as-code-lab/. sentinel-as-code-lab/
cd sentinel-as-code-lab

# Start with the platform only. The other files come back one lab at a time.
mkdir later
mv terraform/{rules,nrt,watchlists,automation,settings}.tf later/

git add . && git commit -m "Lab 1: scaffold" && git push origin main

1.2 Run the bootstrap in Cloud Shell

1. Open shell.azure.com, choose Bash, and check the right subscription is selected with az account show.
2. Clone your repository there (or upload bootstrap/bootstrap.sh) and run it:

git clone https://github.com/<you>/sentinel-as-code-lab.git && cd sentinel-as-code-lab
./bootstrap/bootstrap.sh github <you>/sentinel-as-code-lab

Read the script before you run it. In order, it:

• registers the Azure resource providers the lab uses. The pipeline will not be allowed to.
• creates an empty rg-sentinel-lab resource group for Terraform to deploy into, and rg-sentinel-lab-tfstate for the state.
• creates the state storage account with key access disabled, public blob access off and versioning on.
• creates the Entra app the pipeline signs in as, with three federated credentials: one for main, one for pull requests, and one for the sentinel-lab environment.
• grants it Contributor on rg-sentinel-lab only, and Storage Blob Data Contributor on the state account.

It ends by printing five values and the teardown commands. Keep that output.

1.3 Configure the repository

1. Settings → Secrets and variables → Actions → Variables tab. Add the five printed values: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID, TFSTATE_RESOURCE_GROUP and TFSTATE_STORAGE_ACCOUNT. Use variables, not secrets: none of them are credentials.
2. Settings → Environments → New environment → sentinel-lab. Tick Required reviewers and add yourself.
3. Settings → Branches → add a protection rule for main: require a pull request before merging, and require the validate status check.

Lab 2 · Platform as code

Goal: the workspace and Sentinel itself, created by the pipeline through a reviewed pull request. This is Phase 0 and Phase 1 from part 2.

1. Read terraform/platform.tf. Three things matter: the workspace with its daily_quota_gb cost guard; azurerm_sentinel_log_analytics_workspace_onboarding, the one resource that means "enable Sentinel"; and local.sentinel_workspace_id, which every later resource uses so it waits for Sentinel to be on.
2. Make any small change on a branch, for example a comment in lab.tfvars, and open a pull request. The Lab 1 push only ran validate, because the variables did not exist yet. This time the whole review flow runs.
3. Watch the checks. validate runs with no Azure access. plan signs in through OIDC and posts a comment ending in Plan: 2 to add, 0 to change, 0 to destroy.
4. Merge. The apply job pauses at Waiting for review. Approve it.

2.1 Connect the Activity log (the second human step)

The detections need the AzureActivity table, which exists only once Activity log data flows into the workspace. That setting lives at subscription scope. The pipeline is deliberately limited to one resource group, so you do this once by hand. Put the reason in your runbook, not just the step.

WS_ID=$(az monitor log-analytics workspace show -g rg-sentinel-lab -n law-sentinel-lab --query id -o tsv)

az monitor diagnostic-settings subscription create \
  --name activity-to-sentinel-lab \
  --location centralindia \
  --workspace "$WS_ID" \
  --logs '[{"category":"Administrative","enabled":true},{"category":"Security","enabled":true},{"category":"Policy","enabled":true}]'

Lab 3 · Analytic rules through CI/CD

Goal: detections deployed only through a pull request, and one written by you.

3.1 Ship the first two rules

1. On a new branch: git mv later/rules.tf later/nrt.tf terraform/.
2. Read content/rules/sentinel-rule-changed-by-person.yaml, then terraform/rules.tf, which turns every YAML file into a rule with for_each. Compare it with nrt.tf, where the rule is written straight in HCL.
3. Push and open a pull request. The plan should say 2 to add: one scheduled rule, one NRT rule. Read the plan as a reviewer would. Is the query what you expect? Are the entity mappings there?
4. Merge and approve.

3.2 Exercise: write a detection yourself

Write a High-severity rule that fires when a resource group is deleted. Work from AzureActivity where OperationNameValue is MICROSOFT.RESOURCES/SUBSCRIPTIONS/RESOURCEGROUPS/DELETE. Map it to tactic Impact and technique T1485, and generate a fresh GUID with uuidgen. Add only a YAML file. You should not need to touch Terraform. The answer is in solutions/lab3-resource-group-deleted.yaml.

3.3 Make them fire

# Fires the resource-group rule (within the hour)
az group create -n rg-lab-scratch -l centralindia
az group delete -n rg-lab-scratch --yes

# Fires the NRT rule (within minutes)
ME=$(az ad signed-in-user show --query id -o tsv)
RG_ID=$(az group show -n rg-sentinel-lab --query id -o tsv)
az role assignment create --assignee-object-id "$ME" --assignee-principal-type User --role Reader --scope "$RG_ID"
az role assignment delete --assignee "$ME" --role Reader --scope "$RG_ID"

Lab 4 · Watchlist and automation rule

Goal: content beyond rules, and a lesson in deployment order.

1. On a branch: git mv later/watchlists.tf later/automation.tf terraform/. Read how watchlists.tf uses csvdecode(), so each CSV row becomes one watchlist item.
2. Pull request → the plan adds the watchlist, one item and the automation rule → merge → approve.
3. Second pull request: replace content/rules/sentinel-rule-changed-by-person.yaml with solutions/lab4-rule-with-watchlist.yaml. The rule now ignores people on the approved-admins watchlist.
4. Read this plan carefully. It should say ~ update in-place with only the query changed, not destroy-and-create. An in-place update keeps the rule's identity and incident history.

Do not add yourself to the CSV yet. If you do, Lab 6's detection will correctly ignore you.

Lab 5 · Turn Sentinel settings on and off from Git

Goal: manage capabilities, not just content, using two more of the 32 resource types.

1. On a branch: git mv later/settings.tf terraform/. Read the comments first.
2. Pull request. The plan adds a Fusion rule and the threat-intelligence connector. Merge and approve.

What just happened with Fusion. Every new workspace already has Fusion enabled, as a rule named BuiltInFusion. The provider writes to that same rule, so Terraform did not create a second one. It adopted the existing setting. From now on, Fusion's on/off state lives in Git.

5.1 Exercise: switch Fusion off and on

1. Pull request setting enabled = false in settings.tf. The plan shows ~ enabled: true → false. Merge and approve, then confirm in Analytics that Fusion shows as disabled.
2. Pull request setting it back to true. The history of who turned detection off, when, and who approved it is now in git log.

WS_ID=$(az monitor log-analytics workspace show -g rg-sentinel-lab -n law-sentinel-lab --query id -o tsv)
az rest --method get   --url "https://management.azure.com$WS_ID/providers/Microsoft.SecurityInsights/dataConnectors?api-version=2023-02-01"   --query "value[].{name:name, kind:kind}" -o table

If the apply failed on the connector instead, read the error, remove that resource, and write down why. Connectors are where Terraform coverage is weakest, and finding that out yourself is part of the lesson.

Not everything is one of the 32. UEBA, anomaly settings and Content Hub solutions have no azurerm resource. Enable UEBA in the portal and add it to your human-steps runbook. That runbook is a deliverable in a real engagement, not an apology.

Lab 6 · Catch drift

Goal: see the whole system respond when someone bypasses it. This is the lab worth describing in an interview.

1. In the portal, open [LAB] Azure role assignment created and change its severity from Low to High. Save. You have just done what an analyst does at 2 a.m. during an incident.
2. Wait up to an hour. [LAB] Sentinel analytic rule changed by a person fires, with you as the account entity. The incident carries the sentinel-as-code-lab label, because your automation rule from Lab 4 labelled it on creation.
3. In GitHub → Actions → sentinel → Run workflow on main. The plan shows ~ severity: "High" → "Low". Terraform has noticed that reality no longer matches the code.
4. Decide. To keep the change, open a pull request putting High in nrt.tf, so the code catches up with reality. To revert it, approve the apply job of the run you just started. If you keep the change, reject that pending apply instead.

You do not have to remember to check. The workflow also runs at 01:30 UTC every night, and fails with a drift error whenever the plan is not empty.

Lab 7 · Roll back, and break it on purpose

7.1 Roll back a bad change

git checkout main && git pull
git log --oneline                      # find the Lab 4 watchlist-query commit
git checkout -b revert-watchlist-query
git revert <commit-sha>
git push -u origin revert-watchlist-query   # PR: the plan shows the old query coming back

Merge and approve: the rule is back to its earlier version in minutes, with a record of why. Then revert the revert.

7.2 Break it on purpose

Open a pull request for each of these and watch where it fails. Nothing should reach Azure:

• Run terraform fmt, then deliberately mis-indent a .tf file: validate fails on format.
• Remove the event_grouping block from nrt.tf: terraform validate fails. NRT rules require it, and the author of this lab found that out exactly this way.
• Set a rule's tactic to Initial Access, with a space: plan fails, because the provider checks tactics against Sentinel's list. Nothing was approved, nothing reached Azure.
• Give a rule a period shorter than its frequency, for example frequency: PT1H and period: PT30M. validate and plan both pass. The provider only refuses it during apply, after you have approved the deployment. A mistake caught after approval is the case for your own validator in front of Terraform, catching it on the pull request instead. Part 1 of the Detection-as-Code series covers one.

Lab 8 · The same pipeline in Azure DevOps

Goal: prove the Terraform does not care which platform runs it. Before you start, disable the GitHub workflow (Actions → sentinel → ⋯ → Disable workflow) so two pipelines never fight over one state file.

1. Sign in at dev.azure.com, create an organisation and link your Azure subscription for billing. That is what unlocks the free Microsoft-hosted pipeline minutes. Create a private project.
2. Repos → Import repository → your GitHub repository URL. Alternatively, keep the code in GitHub and connect it when you create the pipeline.
3. Project settings → Service connections → New → Azure Resource Manager → Workload identity federation (manual). Name it sc-sentinel-lab, enter the subscription, then the client ID and tenant ID from Lab 1. It displays an issuer and a subject. Leave it open.
4. In Cloud Shell, add a fourth federated credential to the existing app with that issuer and subject, then return and verify and save the service connection:

az ad app federated-credential create --id <AZURE_CLIENT_ID> --parameters '{
  "name": "ado-sc-sentinel-lab",
  "issuer": "<issuer shown by Azure DevOps>",
  "subject": "<subject shown by Azure DevOps>",
  "audiences": ["api://AzureADTokenExchange"]
}'

1. Pipelines → Library → variable group sentinel-lab with the same five values as Lab 1, plus SERVICE_CONNECTION_ID: the GUID in the service connection page's URL.
2. Pipelines → Environments → sentinel-lab → Approvals and checks → Approvals → add yourself.
3. Repos → Branches → main → Branch policies: require a minimum number of reviewers, and add Build validation pointing at the pipeline. In Azure Repos, that policy is what runs the pipeline on pull requests.
4. Pipelines → New pipeline → your repository → Existing Azure Pipelines YAML file → /azure-pipelines.yml → Run. Permit access to the service connection, variable group and environment when asked.

Lab 9 · Tear down

az monitor diagnostic-settings subscription delete --name activity-to-sentinel-lab --yes
az group delete --name rg-sentinel-lab --yes --no-wait
az group delete --name rg-sentinel-lab-tfstate --yes --no-wait
az ad app delete --id <AZURE_CLIENT_ID>
az group list --query "[?tags.purpose=='sentinel-as-code-lab'].name" -o tsv   # empty when done

Deleting the resource groups deletes the workspace, every rule and the state together. In a real environment you would run terraform destroy through the pipeline instead, so the deletion is reviewed like any other change.

Troubleshooting

What you can now say, honestly

After completing Labs 1 to 7, these are true statements about your own work, and you can defend each one in detail:

• Built a Microsoft Sentinel deployment as code with Terraform: workspace, Sentinel onboarding, scheduled and NRT analytic rules, watchlists, automation rules, Fusion and a threat-intelligence connector.
• Designed a CI/CD pipeline in GitHub Actions (and Azure DevOps) with OIDC workload identity federation, least-privilege role scoping, plan review on pull requests, an approval gate before deployment, and nightly drift detection.
• Wrote a detection that alerts when analytic rules are changed outside the pipeline, closing the loop between change control and the SOC.

Questions you should now be able to answer without notes: