---
title: "Terraform CI/CD Workflows"
slug: gcp-terraform-workflows
section: GCP Infrastructure
status: stable
description: "The `alliance-gcp-infra` repo ships eight GitHub Actions workflows in `.github/workflows`."
last-updated: 2026-10-07
---

# Terraform CI/CD Workflows

The `alliance-gcp-infra` repo ships eight GitHub Actions workflows in `.github/workflows`. They split into three groups: read-only checks (`terraform-plan-pr`, `terraform-plan-notify-prod`, `terraform-drift-detect`), apply workflows (`terraform-apply-dev`, `-qa`, `-prod`, `-mcp`), and a notifier (`notify-platform-pulse`). Only four environments are wired into CI: development, qa, production and mcp. For the environments themselves see `/docs/gcp-overview`, and for the branching model that feeds them see `/docs/git-branching-release-model` and `/docs/onboarding-cicd-deployment`.

Every apply workflow authenticates to Google through Workload Identity Federation (WIF) with no stored key. The apply pool is locked to this repo on `refs/heads/main`, so a feature branch or fork cannot authenticate as an apply identity. The identity model is covered in `/docs/gcp-iam-and-workload-identity`.

## Workflow files

| Workflow file | Name | Triggers |
| --- | --- | --- |
| `notify-platform-pulse.yml` | Notify Platform Pulse |  |
| `terraform-apply-dev.yml` | terraform-apply-dev |  |
| `terraform-apply-mcp.yml` | terraform-apply-mcp |  |
| `terraform-apply-prod.yml` | terraform-apply-prod |  |
| `terraform-apply-qa.yml` | terraform-apply-qa |  |
| `terraform-drift-detect.yml` | terraform-drift-detect |  |
| `terraform-plan-notify-prod.yml` | terraform-plan-notify-prod |  |
| `terraform-plan-pr.yml` | terraform-plan-pr |  |

## Workflow summary

| Workflow | Trigger | Identity (WIF pool, service account) | Writes? |
|---|---|---|---|
| `terraform-plan-pr` | pull request touching `environments/{development,qa,production,mcp}/**` or `modules/**` | pool `github-actions-iac`, SA `terraform-ci-viewer-<dev\|qa\|prod\|mcp>` | No, comments the plan on the PR |
| `terraform-apply-dev` | push to `main` on `environments/development/**` or `modules/**`; manual dispatch with optional `targets` | pool `github-actions-apply-dev`, SA `terraform-tf` in `skilled-script-469819-f2` | Yes, automatic |
| `terraform-apply-qa` | push to `main` on `environments/qa/**` or `modules/**`; manual dispatch with optional `targets` | pool `github-actions-apply-qa`, SA `terraform-tf` in `quality-assurance-496022` | Yes, automatic |
| `terraform-apply-prod` | manual dispatch only; inputs `confirm` and optional `target` | pool `github-actions-apply-prod`, SA `terraform-tf` in `production-499116` | Yes, after typed phrase |
| `terraform-apply-mcp` | manual dispatch only; input `confirm` | pool `github-actions-apply-mcp`, SA `terraform-tf` in `alliance-mcp-services` | Yes, after typed phrase |
| `terraform-plan-notify-prod` | push to `main` on `environments/production/**` or `modules/**` | pool `github-actions-iac`, SA `terraform-ci-viewer-prod` | No, updates a tracking issue |
| `terraform-drift-detect` | cron `17 9 * * *` (09:17 UTC) and manual dispatch | pool `github-actions-iac`, viewer SA per environment | No, updates a tracking issue |
| `notify-platform-pulse` | `workflow_run` completed | none to GCP; uses the gateway PAT | No |

## Apply workflows and the circuit breaker

All four apply workflows share one shape. They run `terraform plan -detailed-exitcode` first and save the plan. Exit code 1 fails the job. Exit code 2 (changes) goes to a gate step that counts `delete` actions in `terraform show -json`. If the count is above `DESTROY_THRESHOLD`, the job fails before `terraform apply` and a human must review `plan_output.txt` and apply by hand. Exit code 0 does nothing. The applied plan is the saved one, so what was gated is what runs. Each workflow serialises runs with a `concurrency` group (queued, not cancelled), uploads the plan as a 30-day artifact, and writes its result to a single labelled tracking issue (`terraform-apply-dev`, `-qa`, `-prod`, `-mcp`).

| Environment | `DESTROY_THRESHOLD` | Gate on the run |
|---|---|---|
| development | 3 | none, applies on merge |
| qa | 3 | none, applies on merge |
| production | 1 | typed phrase `apply-to-production` in `confirm`, checked before checkout or GCP auth |
| mcp | 1 | typed phrase `apply-to-mcp` in `confirm`, checked before checkout or GCP auth |

The breaker is a heuristic. A reviewed destroy below the threshold still applies unattended in dev and qa. The prod `target` input scopes plan and apply to one address, which is the intended way to get under the threshold when an unrelated destroy is also pending. The dev and qa `targets` input does the same for space-separated lists. Production uses manual dispatch because the org's GitHub plan rejects the required-reviewers rule on Environments (noted in `terraform-apply-prod.yml`). The mcp workflow copies that pattern because a non-destructive but breaking change to the gateway would pass a destroy-count check.

Environment-specific steps:
- development guards on the repo secret `DEV_DRIZZLE_MIGRATOR_JOB_ARGS` before any terraform command. A missing value would otherwise blank the live drizzle-migrator job args, so the job fails loudly instead.
- development, qa and production fetch `github-agent-workflow-app-id` and `github-agent-workflow-app-private-key` from Secret Manager in `skilled-script-469819-f2`, mint a token for `n8n-workflows`, and install a git `insteadOf` rewrite. They also fetch GKE credentials for `globus-cluster-development`, `-qa` and `-prod`. Both exist for the `null_resource` provisioners that run kubectl and clone `n8n-workflows`.
- mcp has no GKE cluster and no provisioner, so it has neither step.
- qa surfaces a failed plan as an annotation. When the plan exits 1, the plan step greps the first `Error:` blocks from `plan_output.txt` (up to 60 lines, clipped to 1800 characters) and emits them as a `::error title=terraform plan errors::` annotation. The refresh output is long enough that job-log views truncate the tail where the error sits, and annotations are readable through the check-runs API without downloading the plan artifact. Dev, prod and mcp do not have this step.
- The dev and qa "Report result" step checks the Apply step outcome. The prod and mcp report scripts do not: in both, a code 2 run that passed the breaker is reported as "Applied successfully" even if Apply failed. The prod script goes further: its final branch is a bare else, so any run that is not code 0, 1 or blocked (including a run where the plan step never produced an exit code) is also reported as "Applied successfully" (see Known gaps).

## Read-only workflows

`terraform-plan-pr` detects changed environments with `git diff`. A change under `modules/**` plans all four environments, since consumers are not resolved. It passes the `DEV_DRIZZLE_MIGRATOR_JOB_ARGS` secret as the drizzle migrator variable for development and `[]` for the others, and edits one comment per environment using a hidden marker. `terraform-plan-notify-prod` plans production on every relevant merge and updates the tracking issue titled "Terraform apply-prod report (manual trigger)", so a pending apply is visible. It is the same issue that `terraform-apply-prod` writes to, so whichever ran last wins. `terraform-drift-detect` plans the four live environments nightly in a matrix and writes job summaries and 14-day artifacts. Its tracking issue (label `terraform-drift`) contains only a link to the run, not the drift itself. `notify-platform-pulse` runs on `workflow_run` completion. It posts to Platform Pulse through the composite action `.github/actions/notify-platform-pulse` when any other workflow fails, and on success for the four apply workflows; cancelled runs do not trigger it. The action sends an HTML bot-channel message through the Alliance MCP Gateway using the secret `PLATFORM_PULSE_GATEWAY_PAT`, and only logs a warning if the post fails, so a failed notification does not fail the run.

## State backends

| Environment | Bucket | Prefix | In CI |
|---|---|---|---|
| development | `dev-tfstate-skilled-script-469819-f2` | `alliance-gcp-infra` | plan, apply, drift |
| qa | `qa-tfstate-quality-assurance-496022` | `alliance-gcp-infra` | plan, apply, drift |
| production | `prod-tfstate-production-499116` | `alliance-gcp-infra` | plan, manual apply, drift |
| mcp | `mcp-tfstate-alliance-mcp-services` | `alliance-gcp-infra` | plan, manual apply, drift |
| vertex | `dev-tfstate-skilled-script-469819-f2` | `alliance-gcp-infra/vertex` | none |
| platform-services | `platform-services-tfstate-alliance-platform-services` | `alliance-gcp-infra` | none |

vertex and platform-services have no apply workflow and are excluded from plan-on-PR and drift detection. The plan-on-PR and drift workflow headers describe them as unapplied draft IaC, and say platform-services' bucket did not exist and vertex's state was empty. Their `backend.tf` files carry a "bucket does not exist yet" note. See `/docs/gcp-env-vertex` and `/docs/gcp-env-platform-services`. The same note in the development, production and mcp `backend.tf` files is stale, since CI initialises against those buckets.

## Recovering a stuck pipeline

- Missing CI secret: the dev apply stops at the guard step with a message naming `DEV_DRIZZLE_MIGRATOR_JOB_ARGS`. Create it under repo Settings, Secrets and variables, Actions, then re-run. The drift and plan-on-PR jobs for development read the same secret but fall back to `[]` when it is empty, so without it their development plan runs against blank job args and can show spurious changes rather than failing outright. Nothing in the workflows can create the secret.
- Circuit breaker tripped: read the `plan_output.txt` artifact. If the destroys are intended, re-run through manual dispatch with a narrower target, or apply by hand. Do not raise the threshold to get past a single run.
- Orphaned state lock: a cancelled or crashed job can leave a lock object in the state bucket, and later plans fail with a lock error. Confirm no run is active (the `concurrency` group queues runs), then clear the lock with `terraform force-unlock` and the lock ID from the error message. Neither the workflows nor the repo automate this.
- Partial apply: dev and qa say "state may be partially applied" when Apply fails after a clean plan. Re-run the workflow; the next plan shows what is left.
- First-time access for mcp: the mcp workflow notes that its apply identity cannot create itself, so the first apply for that environment needed someone with standing GCP credentials.

## Gotcha: green plan check, failed plan

In `terraform-plan-pr` and the drift job, the plan step ends with `exit 0` so the comment step always runs. A failed plan therefore does not fail the check. It posts a comment whose heading ends in "(Plan failed)" and the check stays green; read the comment body, not the check colour. The viewer service accounts hold `roles/viewer` and Secret Manager viewer, plus `roles/storage.objectUser` on their own environment's state bucket, added 2026-09-02 because plan needs to write the state lock object. Drift detection has the same property: a failed plan appears only in the job summary and the artifact.

## Known gaps

- Lock recovery steps above are standard Terraform practice, not documented in the repo.
- The repo README is partly stale (it still describes development and production as outside Terraform). The workflows and `.tf` files contradict it.
- The prod and mcp "Report result" scripts report success for a code 2 run that passed the breaker without checking the Apply outcome, and the prod script also reports success when the plan step produced no exit code. The dev and qa workflows check it; their comments cite a 2026-09-28 qa run where a failed Apply was reported as success.
- Drift is detected nightly, but nothing routes it to Platform Pulse. The drift workflow comments say Teams wiring is a deferred fast-follow (the failure notifier only covers workflow failures, and a drift run with a failed plan still concludes green).
- No existing Nexus page I searched mentions these workflows, so there are no contradictions to report. `/docs/disaster-recovery-plan` and `/docs/repo-inventory` were not read in full.