---
title: "Disaster Recovery Plan"
slug: disaster-recovery-plan
section: GCP Infrastructure
status: draft
description: "Baseline disaster-recovery plan for Alliance Abroad's GCP-hosted systems — backups, RTO/RPO targets, and restore procedure."
last-updated: 2026-10-07
---

# Disaster Recovery Plan

Baseline disaster-recovery plan for Alliance Abroad's GCP-hosted systems — backups, RTO/RPO targets, and restore procedure. This page is built incrementally; each section below lands as its own PR.

## Status

> **Living document — Part 3 of an ongoing DR plan.** Tracked by Asana ticket [Disaster recovery?](https://app.asana.com/1/817430812898727/project/1218177098088763/task/1218180528183023). Part 1 covered the production rollback strategy (blue-green deployment) and the load balancer decision behind it. Part 2 added the backup/restore capability audit. Part 3 adds formal RTO/RPO targets, tiered by service criticality — this closes out the ticket's original acceptance criteria. Remaining work (GKE Gateway API rollout, the actual PITR change, a restore drill) is now execution, not planning — see What's Next.

## Why This Matters Now

Related: [Git Branching & Release Model](/docs/git-branching-release-model) (draft) moves general work to continuous deployment straight to `main` → production, with no intermediate `development`/QA environment to catch a bad change before real users see it. That raises the cost of "no fast rollback" from an inconvenience to a real production risk — the only thing standing between a bad merge and every user becomes CI + review, not a staging buffer.

Blue-green deployment is the mitigation: a second, fully-provisioned environment stays idle (or takes a small percentage of traffic) alongside the live one. A bad deploy is a traffic flip back to the known-good version — no redeploy, no restore-from-backup cycle. This closes the specific gap discussed with Bryan Mishkin (Teams DM, 2026-09-10/11) about `n8n-workflows`' `deploy-prod.yml`: it deploys straight to live `n8n.allianceabroad.com` on merge to `main`, with no approval gate and no rollback path today.

## Blue-Green Deployment for Production

**Proposed for all production services**, not just n8n — the same "merge = live" exposure applies to any Cloud Run or GKE service once the branching/release model above ships.

- **Blue** = whatever is currently serving live traffic. **Green** = the newly deployed version, validated before (or while) it takes real traffic. Roles swap every release — there's no fixed "blue service" vs "green service."
- Rollback = flip traffic back to blue. No redeploy, no waiting for a new revision to become healthy — this is the direct RTO improvement DR needs.
- Blue-green does **not** cover database DR by itself — Cloud SQL/Postgres needs point-in-time recovery and/or cross-region replication running independently. Treat compute rollback (this section) and data recovery (the audit below) as two separate tracks.

## Load Balancer Candidates

Two candidates to use **now** — both map onto infrastructure Alliance already runs, no new operational surface:

| # | Candidate | Fit | Why |
| --- | --- | --- | --- |
| 1 | **GKE Gateway API** | BackOfficePrototype / Globus (GKE) | Google's supported successor to `Ingress` (now maintenance-mode, no new features). Weighted traffic splitting is a first-class `HTTPRoute` field — flip an `HTTPRoute` weight to cut over, no mesh required. Closes the rollback gap GKE workloads have today. |
| 2 | **Cloud Run native traffic splitting** | PPPToo / participant server, MCP servers, HostPortal (Cloud Run) | Already built in — `gcloud run services update-traffic --to-revisions=OLD=90,NEW=10`. Zero new infrastructure; this is a policy/process adoption, not a build. |

**Fallback candidate — only if #1 and #2 prove insufficient:**

| # | Candidate | When to reach for it |
| --- | --- | --- |
| 3 | **Service mesh (Istio / Anthos Service Mesh)** | Only if GKE workloads outgrow simple weighted splitting — e.g. need header-based routing, traffic mirroring, or automated canary analysis (Flagger). Meaningfully more operational overhead (sidecars, mesh control plane) than #1, so it's a deliberate upgrade, not a starting point. |

## Backup & Restore Capability Audit

> **Audited 2026-09-22 — real configuration, not a proposal.** Pulled directly via `gcloud sql instances describe` across development, QA, and prod for every Cloud SQL instance Alliance runs.

| System | Automated backups | PITR | Availability | RPO today |
| --- | --- | --- | --- | --- |
| `globus-db-prod` (SEVIS/compliance data) | ✅ | ❌ | ZONAL (no HA) | ~24h (daily snapshot only) |
| `globus-db-qa` | ✅ | ❌ | ZONAL | ~24h |
| `globus-db-development` | ✅ | ❌ | ZONAL | ~24h |
| `n8n-db-prod` / `n8n-db-qa` / `n8n-db-dev` | ✅ | ❌ | ZONAL | ~24h |
| `keycloak-db-prod` / `keycloak-db-qa` | ✅ | ❌ | ZONAL | ~24h |
| `participant-keycloak-db-development` | ✅ | ✅ (the one exception) | ZONAL | near-zero |
| `mcp-db-development`, `npe-j1-visa-postgres-db`, `n8n-db-sandbox`, `n8n-dev-db` | ✅ | ❌ | ZONAL | ~24h |

**The finding that matters:** every production database — including `globus-db-prod`, which holds SEVIS/government compliance data — has automated daily backups but **no point-in-time recovery**. RPO is "lose up to a day of data" everywhere except one dev instance. Every instance is also **ZONAL, not regional** — no automatic failover; a zone outage means a manual restore from the last daily snapshot, and that restore process has never been drilled.

**Secrets** — Secret Manager's built-in version history is the restore mechanism, already structurally in place (confirmed ~90 real secrets in the prod project, versioned by default). No gap here, just unverified in practice — nobody's tested rolling back a bad secret version.

**Configs** — Terraform + git (`alliance-gcp-infra`) is the backup, but known drift between Terraform code and live infrastructure state means a restore-from-git isn't guaranteed to reproduce what's actually running today.

**Recommendation:** enable PITR on `globus-db-prod` first, given it's the compliance-critical instance — then roll out to the rest of the prod tier (`n8n-db-prod`, `keycloak-db-prod`), then QA/dev. This is a `gcloud sql instances patch` flag change per instance (enable PITR + set transaction log retention), not an architecture change — should be one of the fastest wins in this whole plan.

**Separate, non-DR finding surfaced during this audit:** `keycloak-db-prod` and `keycloak-db-qa` reported `requireSsl: false` at audit time — Cloud SQL SSL was not enforced. Terraform in `alliance-gcp-infra` now sets `ssl_mode = "ENCRYPTED_ONLY"` (TLS required, no client cert) on both instances; confirm the live setting matches. Tracked independently of this doc since it's a security gap, not a DR one.

## Formal RTO/RPO Targets

> **Grounded in real system behavior, not arbitrary numbers.** Each tier below is derived from what was actually found investigating SEVIS's own submission semantics ([SEVIS & Government](/docs/sevis-government)), the portal architecture's single-source-of-truth rule, and Alliance's own recent incident history — not an industry-checklist default.

| Tier | Systems | RPO target | RTO target | Why |
| --- | --- | --- | --- | --- |
| **1 — Critical** | `globus-db-prod` (Globus/BackOfficePrototype: applicants, compliance, SEVIS data) | **≤15 min** | **≤4 hours** | SEVIS itself is asynchronous/batch — submissions move through `draft → confirmed → sent` on staff-triggered batch windows, not real-time, so a short outage doesn't itself violate a government deadline. What's expensive to lose is the *compliance data feeding those batches* — host verifications, background-check results, passport OCR/identity confirmation, staff approvals — all of which are costly and slow to re-collect. A `sevis_checkin_due_days`-scale deadline (days, not hours) is the real regulatory clock; 4h RTO comfortably clears it. |
| **2 — Important** | `keycloak-db-prod`/`qa` (auth/session state); Applicant/Host/Partner portal layers | **≤1 hour** | **≤2 hours** | These portals hold **no database of their own** — Gate 0 Sub-check E in `pr-review` enforces this as an Inviolable Law; all compliance data reads/writes flow through Globus's API. Portal-layer state is auth sessions and UI cache, not the system of record — recoverable independently of Tier 1, and user-facing enough (mid-application applicants, active hosts) to warrant a tighter RTO than Tier 1 despite lower data stakes. |
| **3 — Standard** | `n8n-db-prod`/`qa`/`dev` (workflow automation) | **≤4 hours** | **≤8 hours** | n8n already has its own resilience layer independent of the database — `eventRetryWorker` and `webhookRetryWorker` (see [Batch Jobs](/docs/batch-jobs)) retry failed deliveries from source events, so a workflow-state loss is often replayable rather than truly lost. Lower priority than Tiers 1–2, but not lowest: per the revised [Git Branching & Release Model](/docs/git-branching-release-model), Platform Pulse alerting and security-remediation workflows are moving to the prod n8n instance specifically because they can't tolerate silent breakage — that shifts n8n-prod's real-world criticality above a pure "internal tooling" tier. |
| **4 — Best-effort** | MCP servers, dev/sandbox instances, internal platform tooling | **≤24h (current state)** | **≤24h (best-effort, untested)** | Internal-only, no direct external user or compliance impact. Current ~24h RPO from daily snapshots is already sufficient here — no change needed. |

**The gap between today and target, stated plainly:** Tier 1's ≤15 min RPO target requires the PITR change already recommended above — today's actual RPO for `globus-db-prod` is ~24h, **96x worse than target**. Tier 1's ≤4h RTO target is currently **unverified** — every instance is ZONAL with no drilled restore procedure, so the real RTO today is unknown, not "currently 4h and needs tightening." The Sep 11 2026 production outage (missing `SEVIS_PROGRAM_SPONSOR_ID`/`SEVIS_BATCH_USER_ID` secrets causing pods to fail to start — see [SEVIS & Government](/docs/sevis-government)) is the closest real precedent Alliance has to a Tier 1 incident, and it was resolved via a fast redeploy with corrected secrets, not a database restore — consistent with the idea that most realistic Tier 1 incidents are config/deploy problems the blue-green work above already addresses, while the *database* RPO gap remains the one genuinely unaddressed risk.

## What's Next

Execution items only — the planning phase of this ticket is done:
- **Enable PITR on `globus-db-prod`**, then the rest of the prod tier, per the Backup & Restore recommendation above — closes the single largest gap-to-target (Tier 1 RPO).
- **Run a first restore/rollback drill** once PITR lands, to convert Tier 1's RTO target from aspirational to verified.
- GKE Gateway API adoption plan for BackOfficePrototype specifically (blue-green build item).
- (Tracked separately, not DR) Confirm `keycloak-db-prod`/`keycloak-db-qa` live SSL enforcement matches Terraform (`ENCRYPTED_ONLY`).