--- title: "GCP Runbooks" slug: gcp-runbooks section: GCP Infrastructure status: stable description: "The `docs/runbooks` directory of `alliance-gcp-infra` holds four Markdown runbooks, and `scripts/n8n-sync` holds the n8n deploy tooling used by the n8n-related steps." last-updated: 2026-10-07 --- # GCP Runbooks The `docs/runbooks` directory of `alliance-gcp-infra` holds four Markdown runbooks, and `scripts/n8n-sync` holds the n8n deploy tooling used by the n8n-related steps. They differ in status: one is a living checklist, one is a reference for a finished build, one is an unexecuted plan, and one mixes a one-time rebuild procedure with standing operational notes. Read the status line of each section before following it. For the wider recovery picture see `/docs/disaster-recovery-plan`; for how changes reach each environment see `/docs/gcp-terraform-workflows` and `/docs/git-branching-release-model`. ## Runbooks ### post-deploy-verification **Status:** living checklist, still applicable. **Purpose.** Prove that state matches reality after a `terraform apply`, not just that apply exited 0. **When to use.** After every apply in `environments/qa` (and development or production once the same changes are promoted), scoped to the domain that changed. The QA wipe checklist calls it as its verify step. **Steps in brief.** 1. Run `terraform plan` straight after apply; it must show no diff. 2. Run `gcloud asset search-all-resources` for the project and diff it against `terraform state list` for the domain; nothing should be left over from the old manually created resources. 3. Service accounts: list them and confirm only the expected new account(s) exist, and that project IAM bindings resolve to them with none dangling on a deleted account. 4. Secrets: list secrets for the environment, confirm each has a version (re-uploaded per the wipe checklist), and confirm the consuming service logs show no `PERMISSION_DENIED` or `NOT_FOUND` from Secret Manager after its next restart. 5. n8n (the "PR1 acceptance gate"): pods in namespace `n8n` are Running with no CrashLoopBackOff, the nightly `dev-to-qa-sync-agent` workflow executes without a credentials error, and a summary is posted to Platform Pulse as an HTML bot-channel message. **Preconditions.** A completed `terraform apply` in the environment being checked; step 5 also assumes the Platform Pulse posting rules for bot-channel messages. **Gotchas.** An apply that exits 0 is not proof on its own; the clean plan and the asset-versus-state diff are the evidence. The n8n step also asks the operator to remind the owner about two anomalies deferred from PR1 (`globus-mcp-qa`, which breaks the one-shared-MCP-instance policy, and missing `ppp2-participant-server-qa` / `ppp2-participant-client-qa` Cloud Run services); treat those as dated notes, not current findings. **Done.** Clean post-apply plan, zero asset-versus-state drift, expected accounts and secrets present, n8n healthy. ### qa-wipe-checklist **Status:** the wipe procedure is historical (it supported the 2026-07-12 QA rebuild into Terraform and later per-domain rebuilds). The SendGrid rotation and n8n out-of-band sections are standing operational notes still worth reading. **Purpose.** Rebuild one QA domain (service accounts, secrets, Cloud SQL, Cloud Run, GKE, DNS or load balancer) through this repo safely. **When to use.** Only when deliberately deleting and recreating a QA domain. Never more than one domain per PR. **Steps in brief.** 1. Run `terraform plan` for the domain's module blocks in `environments/qa`; any change to resources you did not intend to touch means stop and fix. 2. Get explicit sign-off on the plan; approval must not be assumed. 3. Re-confirm live state with a live call just before deleting, delete only what the approved plan recreates, and for secrets dump every existing value first because deleted secret history cannot be recovered. 4. Apply, then re-upload the dumped secret values (the new containers start empty). 5. Manual, not Terraform-owned: update Workload Identity bindings and the Kubernetes ServiceAccount annotation for any GKE workload using the rebuilt account. This repo deliberately does not own in-cluster manifests. 6. Run `post-deploy-verification` and the asset-versus-state diff for `quality-assurance-496022`. **Standing notes in the same file.** - SendGrid key rotation: `globus--sendgrid-api-key` feeds three senders (the n8n SendGrid credential, the Globus Hub direct mail path, and the Keycloak applicant-portal SMTP relay). Keycloak stores the SMTP password in realm config at apply time and does not re-read the secret, so after rotation `PPPToo/terraform/keycloak` must be re-applied for that environment or verification and reset emails fail silently. Verify with a throwaway `execute-actions-email` expecting HTTP 204. - n8n in QA had no dedicated service account at the time; the Workload Identity binding for `n8n-runtime-qa` was a first-time binding. The `n8n` Deployment, its `n8n-kc-admin` Kubernetes Secret and its cross-service URL env vars are populated out of band and must be recreated by hand on any rebuild. - Dev n8n carries the Platform Pulse monitors (excluded from the QA manifest so they do not double-fire), posting through the gateway with a least-privilege robot PAT held in the `n8n-monitors-gateway-pat` secret. **Risks.** The checklist's destructive steps depend on human sign-off, not tooling. Note that the file hardcodes QA-specific URLs and a Keycloak admin pattern; the file itself lists a dedicated `n8n-provisioner` Keycloak client and ESO sync as owed work. **Done.** Zero drift for the domain and all dependent services healthy. ### postgres-version-alignment **Status:** plan only. The file states that as of 2026-08-01 no decision was made and no migration was run. Merging the document does not close its Asana ticket. **Purpose.** Resolve the Postgres major-version mismatch: `globus-db-qa` is Postgres 16 while `globus-db-development` is Postgres 15. **When to use.** When capacity is scheduled for the migration. The recommendation is to upgrade development and production in place to 16 (non-destructive, restart-based) rather than downgrade QA by dump and restore. **Steps in brief.** 1. Phase 0, blocking: someone with personal `gcloud` credentials lists Cloud SQL instances in `production-499116` to learn prod's real version, because the automation identity sees zero instances there. 2. Phase 1: snapshot dev, patch it to `POSTGRES_16` in a quiet window, run a BackOfficePrototype regression, update `environments/development/main.tf` and confirm a zero-diff plan. 3. Phase 2: soak dev for roughly one to two weeks. 4. Phase 3: after Phase 0 and 2 sign-off, production upgrade in a maintenance window with a full backup first, smoke test, then update the correct Terraform resource. 5. Phase 4: re-list instances in all three projects and confirm all are `POSTGRES_16`. **Risks.** The production Terraform block for the Globus database is described in the runbook as an unapplied template copied from QA whose name differs from the live instance; do not plan or apply prod Cloud SQL until Phase 0 is settled. Major-version changes are deliberately never triggered from a PR-only workflow. **Done.** All three environments report the same version and the ticket's acceptance criteria are updated. Until then the mismatch stands. ### sandbox-n8n-setup **Status:** reference for a completed build (provisioned 2026-09-04, DNS and certificate live 2026-09-08), plus an open follow-up list. **Purpose.** Record the final state of the isolated sandbox n8n: project `skilled-script-469819-f2`, cluster `globus-cluster-development`, namespace `n8n-sandbox`, Cloud SQL `n8n-db-sandbox`, runtime service account `n8n-runtime-sandbox`, static IP `n8n-sandbox-ip`, and domain `sandbox.n8n.allianceabroad.com`. The authoritative source is `environments/development/n8n-sandbox-standalone.tf`. **When to use.** To understand or repair the sandbox, or to test workflow changes before they touch shared instances. **Steps in brief (operational).** - DNS is on GoDaddy, not Cloud DNS, and the A record was created by hand; GoDaddy API access is not currently automatable. - Check the managed certificate with `kubectl get managedcertificate n8n-sandbox-cert -n n8n-sandbox`. If it regresses, confirm public DNS, DNSSEC and the two load balancer forwarding rules, and as a last resort recreate the `ManagedCertificate` object after waiting several hours. - The owner password lives in the `n8n-sandbox-owner-password` secret; it is not in the file. - Two API-key secrets exist: `globus-n8n-sandbox-api-key` (read and control scope, used by the integration test harness in `n8n-workflows`) and `n8n-sandbox-deploy-api-key` (broader scope, used by the baseline deploy). - The 37 curated workflows come from `environments/development/n8n-sync-sandbox/` through `deploy-n8n-baseline.cjs`; the runbook records 37 of 37 active and 0 unbound credentials as of 2026-09-08 (a dated observation, not a live check). **Gotchas.** The sandbox keeps the NetSuite workflows that prod excludes, since it uses NetSuite's sandbox account. **Done.** `https://sandbox.n8n.allianceabroad.com/healthz` returns 200 and the certificate is Active. Open follow-ups listed in the file: decommission the older instance that occupied the "sandbox" identity in `n8n-workflows` tooling, widen integration fixtures beyond three, design a sandbox-to-development promotion path, and reconcile the two parallel n8n deploy mechanisms. ## n8n sync tooling and manifests `scripts/n8n-sync` contains two Node scripts, both of which refuse to run unless the current `kubectl` context contains the `--cluster` value: - `deploy-n8n-baseline.cjs` is a non-destructive, create-only deployer. It clones the source repo, requires a 40-character pinned `sourceCommit`, preflights every file (duplicate ids or names, credential references that must exist live with a matching type, sub-workflow references that must resolve), imports only workflows absent from the live n8n, publishes those meant to be active, restarts the Deployment, and verifies one active `[Globus] Dispatcher: Event Router` with a POST `globus-events` webhook. `--dry-run` stops before importing. - `import-credentials.cjs` is the additive companion: it creates credentials missing by id, never overwrites or deletes, and is safe to repeat. Per the baseline script's header, the baseline deployer does not import credentials itself. The manifests under `environments/*/n8n-sync*` are the git-tracked curation decisions these scripts read: - `workflow-manifest.json`: source repo and branch, the pinned `sourceCommit`, the `files` list, and alias maps (`workflowIdAliases`, `credentialIdAliases`, `nodeCredentialBindings`). Present for qa, production, production's three isolated manifests (email template, external notification, SendGrid webhook receiver) and the development sandbox. Singleton platform-ops and monitoring agents are deliberately excluded so they do not double-fire per environment. - `credential-manifest.json`: credential entities keyed by the ids the workflows reference. It stores no secret values, only Secret Manager secret names (or literals), and a null secret creates an empty shell. Present for development, qa, production and the sandbox. Terraform wires these in through `null_resource` blocks whose triggers hash the manifest and script, so editing either re-runs the deploy on the next apply (see `environments/qa/main.tf` and `environments/production/main.tf`). The development `n8n-sync` directory has no workflow manifest (only the sandbox directory does); development workflow content comes from the `n8n-workflows` repository's own deploy action, and `environments/development/n8n-sync/sync-n8n-credentials.js` only imports credentials, rebinds nodes and activates workflows. `environments/production/n8n-sync/pre-wipe-archive-2026-09-03` is a historical export of the 81 workflows and 7 credential records (metadata only) that production n8n held before it was wiped and rebuilt from the manifests. ## Known gaps - The runbooks are written by and for specific people and dates; none was re-verified against live state for this page. - `qa-wipe-checklist.md` and the production archive README refer to `sync-n8n-workflows.js`, and `terraform-apply-qa.yml` and `terraform-apply-prod.yml` comments mention it, but that file is not in `scripts/n8n-sync`; `deploy-n8n-baseline.cjs` describes it as a retired delete-first mechanism. Treat those references as stale. - The `README.md` of the repo is partly stale about which environments Terraform manages; the runbooks assume all three are. - The production live Cloud SQL version could not be confirmed from the sources. - The relationship between the two n8n deploy mechanisms (this repo versus `n8n-workflows`) is an open question recorded in the sandbox runbook.