---
title: "Git Branching & Release Model"
slug: git-branching-release-model
section: Developer Onboarding
status: stable
description: "Trunk-based branching with no long-lived integration branch."
last-updated: 2026-10-07
---

# Git Branching & Release Model

Trunk-based branching with no long-lived integration branch. Every branch is cut from `main`. A feature branch deploys straight to a pooled **development** environment (3 slots). Ready features are merged into a named `release/<name>` branch, which deploys to a pooled **QA** environment (3 slots). A `release/<name>` → `main` merge, or a `hotfix/*` → `main` merge, is the only way into production, and it goes through a blue-green cutover.

## Status

> **Implemented and verified in the `branching-sandbox` repo; adopted by WildPacksApplicationForm (2026-10-03), PPPToo (2026-10-04), HostPortalPrototype (2026-10-04) and BackOfficePrototype (2026-10-06). Revised 2026-10-06.** This page describes what the sandbox (`Alliance-Strategies/branching-sandbox`, with its infrastructure in `alliance-gcp-infra`) does and what the four converted repos inherited from it. It supersedes the 2026-09-22 revision, which kept a permanent `development` branch that features merged into. The model implements the opposite: **`development` is an environment, not a branch.** Sections below are marked **Implemented**, **Verified** (run end to end and observed), or **Not implemented** (design intent only), so nobody mistakes one for the other. The **Migration Plan** further down is done for all four repos; the items still open for BackOfficePrototype are listed under Phase 4.
>
> The sandbox is a **stateless static page**. It exercises branching, pooling, deploys and cutover. It does **not** exercise databases, migrations, or irreversible side effects (such as BackOfficePrototype/Globus SEVIS submissions, which a canary or rollback cannot undo). Those were the adoption risks for BackOfficePrototype, listed under Adoption Risks and worked through in the Migration Plan; BackOfficePrototype went live on the GKE edition of the model on 2026-10-06 (production cutover 2026-10-05).
>
> **PPPToo is live on this model since 2026-10-04** (Asana 1219058470201463): `main` is the default branch and production; `development`/`qa` are frozen legacy branches; `feature/*` pushes deploy to `dev-{1,2,3}.applicants.allianceabroad.com` (API at `dev-N-api.…`), `release/*` pushes to `qa-{1,2,3}.applicants.allianceabroad.com`, and `release/*` → `main` merges run a Cloud Run revision blue-green on the existing `ppp2-participant-{server,client}-prod` pair (server first, then client). All five proof runs (feature → dev slot, feature → release, release → QA slot, release → main blue-green with a forced-rollback test, hotfix → main) were observed live on 2026-10-04; see `docs/sandbox-model-proof-2026-10-04.md` in the repo.
>
> **HostPortalPrototype is live on this model since 2026-10-04 20:17 UTC** (Asana 1219139535095088): same shape as PPPToo with Auth0 instead of Keycloak. Pools `host-portal-{server,client}-{dev,qa}-{1,2,3}` in `us-central1` behind `dev-N` / `dev-N-api` / `qa-N` / `qa-N-api` `.hosts.allianceabroad.com`; production stays on `host-company-portal-{server,client}-prod` in `northamerica-northeast1` behind its existing load balancer; the first production blue-green (main `b27a182`) cut over both services with the previous revisions kept as `blue`. Rulesets: Branch names 24467828, Protect main 24464581, Protect release branches 24467830, Freeze legacy branches 24467833. Proven live: feature → dev slot, feature → release checks, release → main blue-green plus its automatic rollback (the first attempt failed the canary and restored traffic, see Known Limitations), hotfix → no deploy, protected-path block, branch-name allow-list. Still pending at the time of writing: the first `release/*` → QA slot deploy (its PR awaits a second approver), the two consumer PRs (vertex-agents #268, itrequestportal #111), and the dev Auth0 slot callbacks (the dev tenant's management clients lack `update:clients`).

## Principle

- `main` is the source of truth and maps 1:1 to production.
- **Every branch is cut from `main`. Nothing is ever cut from another branch.**
- There is no `development` branch. `development` is the name of an environment.
- Environments are deploy targets chosen by the branch name, not branches themselves.
- Production is reached only by a blue-green cutover.

## Branch Types

| Branch | Cut from | Merges into | Lifetime | Deploys to |
| --- | --- | --- | --- | --- |
| `main` | — | — | Permanent | Production, via blue-green cutover (on every push to `main`). |
| `feature/<asana-ticket-number>-<short-name>` | `main` | a `release/<name>` branch | Until merged into a release | Development, one of 3 pool slots (on every push). |
| `release/<name>` | `main` | `main` | Until merged into `main` | QA, one of 3 pool slots (on every push). |
| `hotfix/<asana-ticket-number>-<short-name>` | `main` | `main` | Very short-lived | Production only when its PR merges to `main`. It can also be tried in a development slot (push or dispatch) and a QA slot (manual dispatch); proven in the sandbox only, see [Hotfix Dev and QA Slots](#hotfix-dev-and-qa-slots). Everywhere else a hotfix push still deploys nothing. |

Example: `feature/1218773865500269-branching-model`.

## Environments

| Environment | Branch trigger | Pooled | Hostnames (sandbox / PPPToo / HostPortal) | GCP project |
| --- | --- | --- | --- | --- |
| development | `feature/**` | Yes, 3 slots, 30 min reservation | `dev-{1,2,3}.sandbox.allianceabroad.com` / `dev-{1,2,3}.applicants.allianceabroad.com` + `dev-N-api.applicants.…` / `dev-{1,2,3}.hosts.allianceabroad.com` + `dev-N-api.hosts.…` | `skilled-script-469819-f2` |
| QA | `release/**` | Yes, same mechanism, 3 slots | `qa-{1,2,3}.sandbox.allianceabroad.com` / `qa-{1,2,3}.applicants.allianceabroad.com` + `qa-N-api.applicants.…` / `qa-{1,2,3}.hosts.allianceabroad.com` + `qa-N-api.hosts.…` | `quality-assurance-496022` |
| production | `main` | No, single service with weighted traffic | `sandbox.allianceabroad.com` / `applicants.allianceabroad.com` / `hosts.allianceabroad.com` + `hosts.wildpacks.com` (existing LB in front of the existing pair) | `production-499116` |

Pool slots live in `us-central1` (Cloud Run domain mappings are not allowed in `northamerica-northeast1`). The sandbox's Artifact Registry is `northamerica-northeast1`; PPPToo keeps its existing `applicants-portal` registry in `us-east1`; HostPortal keeps `host-company-portal` (`us-east1`, dev and prod) and `portal` (`us-central1`, QA). PPPToo and HostPortal production stay in `northamerica-northeast1` behind their existing load balancers: the serverless NEG targets the service, so revision traffic splitting works there without touching the balancer. HostPortal's slots are mapped on `hosts.allianceabroad.com` only; the client chooses its brand from the hostname, so the WildPacks brand is not exercised on a slot.

## Flow

```
main ──┬── feature/<ticket>-<name> ──► development pool slot (dev-1..3)
       │            │
       │            └── PR into a release branch  (PR review agent: wired for PPPToo, HostPortal after vertex-agents #268)
       │
       ├── release/<name> ──► QA pool slot (qa-1..3) ──► PR to main ──► blue-green ──► prod
       │
       └── hotfix/<ticket>-<name> ──► PR to main ──► blue-green ──► prod
              │                                           │
              │                                           └── hotfix-backport.yml: one feature/<GID>-backport-* PR per open release
              ├── push / dispatch ──► development pool slot   (sandbox only so far)
              └── manual dispatch ──► QA pool slot, 2 h hold  (sandbox only so far)
```

1. **Build a feature.** Cut `feature/<ticket>-<name>` from `main` and push. It deploys to a development pool slot.
2. **Cut a release.** Cut `release/<name>` from `main`. Open a PR from each ready feature branch into it.
3. **QA.** Every push to the release branch deploys it to a QA pool slot.
4. **Ship.** Open a `release/<name>` → `main` PR. When it merges, `main` deploys to production via blue-green.
5. **Hotfix.** Cut `hotfix/<ticket>-<name>` from `main`, PR it straight to `main`. It does not have to pass through development or QA, but where the repo supports it (today only the sandbox) it can be tried in a dev slot and a QA slot first.

Because feature branches are cut from `main` and merged into a release (not into a shared branch), a feature only meets other features inside the release branch it is merged into.

## Pools — Development and QA (Implemented, Verified)

Development and QA each have three pre-provisioned Cloud Run services (`sandbox-web-<env>-1`, `-2`, `-3`; for PPPToo and HostPortal a server+client pair per slot, `ppp2-{server,client}-<env>-N` / `host-portal-{server,client}-<env>-N`, with the labels on the server service) with their own hostname. The service labels are the entire slot state, so there is no registry file or extra infrastructure:

- `sandbox-owner` — the **ownership key**: `b-` plus the 40 hex characters of the SHA-256 of the full ref (`refs/heads/<branch>`). Two different branch names can never produce the same key, so one branch cannot renew or evict another's slot. This is the only label the allocator trusts.
- `sandbox-branch` — sanitized, truncated branch name, **diagnostics only** (it is always a valid label value, so it can collide or be truncated without consequence).
- `sandbox-reserved-until` — unix epoch when the reservation expires.

A service that has no `sandbox-owner` label (for example one deployed before this scheme existed) counts as unowned and is eligible for the next claim. The page rendered into each slot HTML-escapes the branch name, so a hostile branch name cannot inject markup.

On each push, `scripts/allocate-slot.sh` picks a slot in this order:

1. A slot this branch already holds — **renews** the reservation (30 minutes from now).
2. A slot whose reservation has expired (or never existed), oldest expiry first — **evicts** the previous holder.
3. Otherwise **fails the deploy**: every slot is reserved by another branch. The job prints each slot's holder and expiry.

After claiming, it re-reads the labels and retries (up to 5 attempts) if another branch won the race. The reservation is renewed by pushing, not by time passing.

## CI/CD Trigger Logic (Implemented)

One workflow, `.github/workflows/deploy.yml`, handles `release/**` and `main`; `feature/**` goes through `deploy-feature-request.yml` → the reusable `trusted-deploy-dev.yml@main` (see Identity).

| Trigger | Action |
| --- | --- |
| Push to `feature/**` | Build image, claim a development slot, deploy to it. |
| Push to `release/**` | Build image, claim a QA slot, deploy to it. |
| Push to `main` | Build image, run the production blue-green deploy. |
| Push to `hotfix/**` | Sandbox: build image, claim a development slot through `trusted-deploy-dev.yml@main` (proven 2026-10-12). Every other repo: nothing (its dev rail still accepts `feature/**` only until its hotfix-paths PR is live; see Hotfix Dev and QA Slots). |
| Manual dispatch of `deploy-hotfix-qa-request.yml` at a `hotfix/**` branch | Sandbox: claim a QA slot for 2 h through `trusted-deploy-qa.yml@main` (proven 2026-10-12). The job is skipped on any other ref type. |
| Push that only changes `README.md` or `docs/**` | Nothing (`paths-ignore`). A push with no file changes also does not deploy — including creating a branch at a commit that already exists, so the first slot deploy of a fresh branch needs a real commit. |
| A newer push to the same `feature/**` or `release/**` branch while a deploy is running | The older run is cancelled (concurrency group per ref). Different branches deploy in parallel. |
| A newer push to `main` while a production deploy is running | **Not cancelled.** The run queues behind it (`cancel-in-progress` is false for `main`), because cancelling a cutover mid-canary could strand a split. |

Each job has a 20 minute timeout; the production cutover step has its own 15 minute timeout. Images are built per environment from the branch commit, so the image that reaches production is a fresh build of `main`, not the QA image. A `quality` job (the repository quality gate) and, in the portal repos, a `tests` job gate every deploy.

## Production Blue-Green (Implemented, Verified)

Production is one Cloud Run service per deployable (`sandbox-web-prod`; PPPToo: `ppp2-participant-server-prod` then `ppp2-participant-client-prod`; HostPortal: `host-company-portal-server-prod` then `host-company-portal-client-prod`). Blue-green is Cloud Run revision traffic splitting on that service — **no load balancer change is needed**, even where one exists in front.

The cutover is its own script, `scripts/blue-green.sh`, with a test suite that runs it against a fake `gcloud` and `curl`. The portal copies take `PROBE_PATH` (`/api/v1/version` or `/api/version` for the server, `/` for the client), `PUBLIC_URL` and `DEPLOY_ARGS` (the server's authoritative `--env-vars-file` and, for HostPortal, `--update-secrets` list).

1. **Record the exact pre-deploy allocation** (for example `rev-a=60,rev-b=40`), not an assumed `blue=100`. **Blue** is the revision serving the most traffic.
2. The new build is deployed as **green** with `--no-traffic` behind a `g-<sha12>` tag (the first 12 hex characters of the commit).
3. **Smoke test:** the green revision's private tag URL must return a page containing the full commit SHA.
4. **Canary:** 10% of traffic to green. The 10% is taken from the revision serving the most traffic, and every other serving revision keeps exactly the share it had (a 60/40 start becomes green 10, 50, 40). If the largest revision has less than 10%, it is drained and the remainder comes from the next largest. The service URL is probed for at least 30 seconds, every probe must succeed, and **green must actually be seen serving** (at least 5 responses carrying the new commit). A canary that only ever hit blue fails as inconclusive.
5. **Cutover:** 100% to green. The previous leader keeps the `blue` tag as the instant rollback target.
6. **Prune:** only the newest 5 `g-*` tags are kept (`KEEP_GREEN_TAGS`); older ones are removed. A tag on a serving revision is never removed and a failed run prunes nothing.
7. **Failure handling:** from the first traffic change until the 100% update succeeds, any exit restores the recorded allocation — a failed probe, an inconclusive canary, a timeout, or SIGINT/SIGTERM from a cancelled run. Every probe is time-bounded, so a signal is acted on within about 15 seconds. The workflow also has a safety step that repeats the restore if the script itself was killed outright.

Verified (sandbox): a deliberately broken change failed at the smoke test (step 3), the broken revision stayed at 0%, and production kept serving the previous revision. A fixing hotfix then cut over cleanly and moved the `blue` tag. Verified (PPPToo, 2026-10-04): two live cutovers on the real portal (90/10 canary → 100%, previous revision tagged `blue`, both services), and a forced-rollback run with an unreachable SHA that failed the smoke test and never touched traffic. Verified (HostPortal, 2026-10-04): the first cutover attempt failed at canary probe 1 because the then-current revision had no `/api/version` route (every probe of the public URL 404ed), and the script restored the recorded allocation exactly; after the probe route was present on the serving revision, the re-run cut over both services (server `00010`, client `00007`) with the previous revisions tagged `blue`.

## Identity and Infrastructure (Implemented)

Infrastructure is Terraform in `alliance-gcp-infra`: `environments/{development,qa,production}/branching-sandbox.tf` for the sandbox; `environments/{development,qa}/ppp2-pool.tf` + `environments/production/ppp2-deploy-identity.tf` for PPPToo; `environments/{development,qa}/host-portal-pool.tf` + `environments/production/host-portal-deploy-identity.tf` for HostPortal. Per environment:

- An Artifact Registry repo (`sandbox`; PPPToo reuses `applicants-portal`; HostPortal reuses `host-company-portal` / `portal`).
- A runtime service account (no roles beyond what the app needs; PPPToo's server runtime gets `secretAccessor` on exactly its three secrets; HostPortal reuses its existing per-environment runtime accounts) and a deployer service account (`run.developer` on the pool/prod services, `run.viewer`, AR writer, `actAs` on the runtime accounts only; HostPortal's deployer also reads the four build-time client secrets and the private npm registry).
- A Workload Identity pool per repo (`github-actions-sandbox`, `github-actions-ppp2`, `github-actions-hostportal`) that accepts only that repository **and only the matching branch type**: `refs/heads/feature/` in development, `refs/heads/release/` in QA, exactly `refs/heads/main` in production. In development the provider additionally requires `assertion.job_workflow_ref == <repo>/.github/workflows/trusted-deploy-dev.yml@refs/heads/main`, so a feature branch's own workflow can never hold the deploy identity: only the reusable workflow pinned to protected `main` can. In the sandbox (since 2026-10-12) the development provider also accepts `refs/heads/hotfix/` through that same workflow, and the QA provider has a second path for a `hotfix/` ref that is valid only when `job_workflow_ref == <repo>/.github/workflows/trusted-deploy-qa.yml@refs/heads/main`; the production provider is unchanged (`ref == refs/heads/main`), so a hotfix ref can never mint the production deployer.
- Cloud Run services ignore changes to image, env, labels, traffic and revisions, so deploys and Terraform never fight.

**Custom-domain mappings are not in Terraform.** Creating one needs a verified owner of the domain. Since 2026-10-02 `terraform-tf@` in dev and QA is a verified owner of `allianceabroad.com`, so mappings are created with `gcloud beta run domain-mappings create … --impersonate-service-account=terraform-tf@<project>…` (the caller needs `iam.serviceAccountTokenCreator` on that SA) plus a GoDaddy CNAME to `ghs.googlehosted.com`. Production applies are `workflow_dispatch` only with a typed confirmation, one `target` at a time when the plan carries unrelated drift (two parties dispatching production applies at the same time collide on the state lock with HTTP 412; wait and retry).

## Enforcement: Rulesets and the branch-rules Check (Implemented, Verified)

The rules on this page are enforced by repository rulesets, not by this document. No ruleset has bypass actors, so nobody, administrators included, can skip them. Copies are kept in each repo under `.github/rulesets/` as documentation; the live settings are what count.

| Rule | Enforced by |
| --- | --- |
| Only `main`, `feature/<16-digit ticket>-<name>`, `release/<name>`, `hotfix/<16-digit ticket>-<name>` and `dependabot/**` branches can be created | Ruleset "Branch names": a `creation` rule that applies to every ref except those patterns (an allow-list written as exclusions). Verified with real pushes on all three repos. |
| No direct push, force push or delete on `main` and `release/**` | Rulesets "Protect main" and "Protect release branches" (the release ruleset needs `do_not_enforce_on_create: true` on its required checks, or cutting a release branch from `main` is itself rejected) |
| Everything reaches those branches by pull request, with `quality-gate` and `branch-rules` (PPPToo also `build-test`; HostPortal also `typecheck (server)`, `lint (client)`, `secret-scan (gitleaks)`) green and 1 approving review (stale approvals dismissed on a new push), Code Owner review on governance paths | Same rulesets: pull request rule plus required status checks |
| Allowed pull requests: feature → release, release → main, hotfix → main (Dependabot → main) | Required check `branch-rules` (`scripts/check-branch-rules.sh`) |
| Feature and hotfix branches were cut from `main`, not from another branch | Same required check (ancestry test) |
| Changes to workflows, the gate, its config, deploy config, CODEOWNERS, AGENTS.md need the `governance-approved` label from a maintainer **and** a Code Owner review | Required check `quality-gate` (`protected-paths`) + CODEOWNERS |
| Legacy `development`/`qa` branches cannot be pushed to, force-pushed or deleted during the transition | Ruleset "Freeze legacy branches" (PPPToo, WildPacks, HostPortal) |

Rulesets cannot express which branch type may open a pull request into which, or where a branch was cut from; they can only require a named check to pass. Those two rules therefore live in the `branch-rules` check and the rulesets make passing it mandatory. The check runs on `pull_request_target`, so the workflow and script are read from the **base** branch and a pull request cannot weaken its own check; it never executes code from the pull request.

Verified live: a hotfix → release pull request failed `branch-rules` and its merge was refused, while hotfix → main passed (sandbox and PPPToo); a PPPToo PR that edited `deploy/README.md` without the label failed `quality-gate` with `protected-paths` and the file was dropped from the PR rather than the rule lifted; a HostPortal PR that edited `AGENTS.md` without the label failed `quality-gate` with `protected-paths` while `branch-rules` passed (#621, closed as a probe).

Limits worth knowing:

- Ancestry cannot be blocked at push time. A feature cut from the wrong branch can still deploy to a development slot; it can never be merged.
- `branch_name_pattern` rules are accepted by the API on the current plan but are not enforced, which is why branch names use the creation allow-list.
- One approving review is required, and GitHub does not let an author approve their own pull request, so every merge into `main` or a release branch needs a second identity (the PR review agent's App approval, or the `alliance-platform-bot` machine account, which is also a Code Owner). Dependabot branches can target `main` directly because Dependabot cannot know about release branches.
- Making a new check required needs the workflow to exist on the target branches first, and fixing the check itself while it is required needs the requirement lifted temporarily.
- `pull_request_target` workflows only register from the repository **default branch**: during a conversion, switch the default branch before expecting the checks to run.

## Hotfix Workflow

1. **Rollback vs. hotfix decision.** If the issue has no destructive or irreversible migration attached, flip traffic back to the `blue` revision first — this resolves user impact immediately. Only proceed straight to a hotfix if rollback is unsafe.
2. **Branch upstream-first:** `git checkout -b hotfix/<ticket>-<slug> main`.
3. Smallest possible diff, plus a regression test that fails without the fix.
4. **PR into `main`.** The required checks and one approving review apply like any other PR into `main`. The PR review agent reviews hotfix PRs on PPPToo and HostPortal (their base-branch lists include `main`).
5. Merge to `main` → blue-green cutover (Implemented, Verified).
6. **Backport** the fix by opening one PR per open `release/*` branch (not `feature/*`; no auto-merge; a conflicting backport stays open, labelled, with a Platform Pulse post): **Decided 2026-10-08, implemented as `hotfix-backport.yml`** (see Open Questions): it runs after the hotfix PR merges to `main`, which is why a `hotfix/*` → `release/*` PR is never opened by hand (`branch-rules` rejects that route). It is on `main` in the sandbox and WildPacksApplicationForm and is on the release branches of PPPToo, HostPortalPrototype and BackOfficePrototype, reaching their `main` with the next release. Where it is not yet on `main`, a release branch cut before the hotfix does not contain the fix; merging `main` into the release branch is the manual way. Review the backport PRs as a mirror of the merged hotfix, and close and reopen each once so its required checks start.
7. Verify in production and write a short post-incident note.

## Hotfix Dev and QA Slots

> **Verified in `branching-sandbox` on 2026-10-12 (Asana 1219393647120312). Not live in any other repo yet.** A hotfix used to be testable nowhere before production. The same trusted-workflow pattern that protects the development rail now lets a hotfix branch claim a pool slot, without giving it any route to production.

| Path | Trigger | Runs | Hold |
| --- | --- | --- | --- |
| Development slot | Push to `hotfix/<16-digit ticket>-*`, or "Run workflow" on `deploy-feature-request.yml` at the branch | `trusted-deploy-dev.yml@main`; its caller guard accepts `feature/*` or `hotfix/*` | 30 minutes |
| QA slot | Manual dispatch of `deploy-hotfix-qa-request.yml` at the branch (never on push: QA slots are shared with release branches) | `trusted-deploy-qa.yml@main`; its caller guard accepts `hotfix/*` only | 2 hours (a `release/*` branch holds 24 hours) |

Trust boundary (Workload Identity, `alliance-gcp-infra` #384, applied to development and QA):

- **Development:** ref `refs/heads/feature/` **or** `refs/heads/hotfix/`, and `job_workflow_ref` exactly `trusted-deploy-dev.yml@refs/heads/main`.
- **QA:** the existing release path (`refs/heads/release/` and `deploy.yml@refs/heads/release/`) is unchanged; a second path accepts `refs/heads/hotfix/` only with `job_workflow_ref` exactly `trusted-deploy-qa.yml@refs/heads/main`.
- **Production:** unchanged, exactly `refs/heads/main` through `deploy.yml@refs/heads/main`. A hotfix ref cannot mint the production deployer; only the merge to `main` deploys.

Proof (sandbox, throwaway branch `hotfix/1219393647120312-slot-proof`, after `release/2026-q4-hotfix-slots` reached `main` as `b6dd1d0`):

1. **Development:** the dispatched request ran through `trusted-deploy-dev.yml@main`, claimed `sandbox-web-dev-1`, labelled it with the hotfix branch and served `b6dd1d0` (30 minute hold).
2. **QA:** the dispatched request ran through `trusted-deploy-qa.yml@main`, claimed `sandbox-web-qa-2` for about 2 hours and served `b6dd1d0`.
3. **Refusals:** the QA request on a `feature/*` ref and on a `release/*` ref was skipped; the development request on a `release/*` ref failed at "Verify caller context" before any cloud credential was requested; no other slot changed.
4. **Production:** untouched by every hotfix run; only the merge to `main` ran the blue-green.

A `workflow_dispatch` only works on a ref that already contains the workflow file, so the refusal checks needed a throwaway release branch cut from `main`.

Rollout to the real repos, as of 2026-10-12. **None of these is live yet.** A hotfix path needs both the infrastructure change and the release on `main`, because the trusted workflows are pinned `@main`; until then a hotfix branch cut from `main` does not even contain the new workflows, and once it does, authentication fails closed at Workload Identity until the infrastructure change is applied.

| Repo | Repo change (merged into its release branch) | Infrastructure PR (`alliance-gcp-infra`) | Live when |
| --- | --- | --- | --- |
| PPPToo | #453, `release/2026-10-ppp2-pilot` | #387, open | #387 merged and applied (scoped), and the release reaches `main` |
| HostPortalPrototype | #635, `release/2026-10-hostportal-pilot` | #385, open | same |
| WildPacksApplicationForm | #127, `release/2026-q4-form-validation` | #386, open | same |
| BackOfficePrototype (GKE) | #2837, `release/2026-10-globus-pilot` | #388, open | same |

**Infrastructure applies are scoped for now.** The full `terraform-apply-dev` and `terraform-apply-qa` plans already fail before this change (development since at least 2026-10-06; QA wants to destroy the `prevent_destroy` Cloud SQL instance `module.globus_db_qa`), so apply-on-merge applies nothing. Each of these changes is applied with `workflow_dispatch` and `targets=<address>` (for the sandbox: `google_iam_workload_identity_pool_provider.sandbox`), then checked with `gcloud iam workload-identity-pools providers describe`. Never force an unscoped apply while the QA plan carries the database destroy (tracked in Asana 1219396006336148).

## Verified vs. Not Implemented

| Behaviour | State |
| --- | --- |
| Feature push deploys to a dev slot; page shows branch and commit | Verified (sandbox, PPPToo, HostPortal) |
| Re-push of the same branch keeps its slot and extends the reservation | Verified |
| A new branch takes an expired slot (eviction) | Verified |
| A 4th branch with all 3 slots reserved is rejected, other slots untouched | Verified |
| Release push deploys to a QA slot | Verified (sandbox, PPPToo); HostPortal pending its first release push |
| Release → `main` merge runs the production blue-green cutover | Verified (sandbox, PPPToo: two live cutovers 2026-10-04; HostPortal: one live cutover 2026-10-04) |
| Hotfix PR to `main` deploys through the same cutover | Verified (PPPToo: docs-only hotfix proved the route and the `paths-ignore` no-deploy; HostPortal: hotfix push produced no run) |
| Failed smoke test rolls back; production keeps serving the old revision | Verified (sandbox, PPPToo prod server) |
| Canary that never sees green, a failed probe, and a cancelled run each restore the exact pre-deploy allocation | Failed probe verified **live** on HostPortal prod (2026-10-04, first cutover attempt); the other two by the script test suite |
| Successful canary preserves a mixed starting allocation | Verified by the script test suite; the live cutovers so far started from a single revision |
| Only the newest 5 `g-*` deploy tags kept after a cutover | Verified live and in the test suite |
| Branch names outside the allowed patterns cannot be created; PR routing and cut-from-`main` enforced; `main` and `release/**` need a PR, green checks and 1 approval | Verified live (see Enforcement) |
| Development WIF identity only obtainable by the reusable workflow pinned to `main` | Implemented and exercised (sandbox, PPPToo, HostPortal); QA/prod providers check repo + ref only |
| PR review agent on feature → release and release/hotfix → main PRs | Implemented for PPPToo 2026-10-04; HostPortal via vertex-agents #268 (merged) |
| Merge queue for concurrent releases | Not implemented |
| Scheduled slot-reconciliation sweep, delete-triggered slot free, "waiting on capacity" notification | Not implemented (a full pool is a failed deploy; expiry is the only slot release) |
| Hotfix backport CI (`hotfix-backport.yml`) | Implemented 2026-10-08/09 and exercised on the sandbox (proof PRs 2026-10-08); on `main` in the sandbox and WildPacksApplicationForm, on the release branches of PPPToo, HostPortalPrototype and BackOfficePrototype (reaches `main` with the next release) |
| Hotfix → development slot and QA slot (dispatch) | Verified in the sandbox 2026-10-12; merged into the release branches of PPPToo, HostPortalPrototype, WildPacksApplicationForm and BackOfficePrototype but **not live** until their infrastructure PRs are applied and the releases reach `main` (see Hotfix Dev and QA Slots) |
| Periodic merge of `main` into long-lived release branches | Not implemented (manual) |
| Adoption by portal repos | WildPacksApplicationForm converted 2026-10-03; **PPPToo converted and verified end to end 2026-10-04**; **HostPortalPrototype converted 2026-10-04** (QA-slot proof and consumer PRs pending); **BackOfficePrototype converted 2026-10-06** (GKE edition: dev pool `globus-dev-N`, QA pool `globus-qa-N`, production cut over to the GKE Gateway blue-green 2026-10-05 08:11 UTC; see Phase 4) |

## Known Limitations

- **A 30-minute reservation is not a long hold.** Earlier drafts assumed a named-initiative release branch holds a QA slot for weeks. In the sandbox the slot is only reserved for 30 minutes after the last push; after that another release can evict it, and the initiative's branch must push again to redeploy. Decided 2026-10-08: long-lived releases get a longer reservation TTL for `release/*` (value to be chosen; not yet implemented).
- A branch carries its own copy of the workflow, so a long-lived branch keeps running an old version of it. Delete branches when done.
- A push with no file changes (an empty commit, or a new branch at an existing commit) does not deploy.
- The release branch is not deleted automatically after it merges to `main` (decided 2026-10-08 to auto-delete; not yet implemented), and its QA slot is only released by expiry or eviction.
- **The portal repos cut the server and client over sequentially, not atomically**: the API must stay backward compatible with the previous client for the seconds between the two cutovers.
- Slots share the environment's identity provider (Keycloak realm for PPPToo, Auth0 tenant for HostPortal) and Globus API; isolation is at the Cloud Run service level only (both BFFs are stateless, so this is sufficient there). PPPToo's `GLOBUS_API_BASE_URL` is `https://dev-3.globus.allianceabroad.com` in development and `https://qa-1.globus.allianceabroad.com` in QA.
- **First-cutover bootstrap.** The canary probes the public URL, which mostly hits the *old* revision. If the old revision does not serve the probe route (`/api/version`), every probe fails and the cutover correctly rolls back forever. Ship the probe route through the old deploy path first, or pin traffic to a revision that has it, before the first blue-green. After one blue-green the service's traffic is pinned to a revision, so a legacy `gcloud run deploy` no longer moves traffic to `LATEST` on its own.

## Adoption Risks (Portal Repos)

- **Stateful services.** Pool slots for BackOfficePrototype/Globus need their own data (a database per slot, migrations, secrets), which the sandbox does not model. PPPToo and HostPortal are stateless BFFs, which is why they could adopt the mechanism as-is.
- **Irreversible side effects.** BackOfficePrototype/Globus carries compliance-sensitive actions (SEVIS submissions) that a canary or rollback cannot undo. A blue-green cutover protects the deploy, not the data.
- **GKE workloads** (BackOfficePrototype/Globus) cannot use Cloud Run traffic splitting. They need GKE Gateway API with weighted `HTTPRoute` routing, since `Ingress` has no equivalent. That was the real build item; it shipped as BackOfficePrototype's `deploy-prod-bluegreen.yml`, and production cut over to it on 2026-10-05.
- **Cloud Run workloads** (PPPToo participant server, HostPortal) adopt the sandbox mechanism closest to as-is: done for both. See [Disaster Recovery Plan](/docs/disaster-recovery-plan) for the load-balancer decision behind the GKE case.
- **Migration from the current pipeline.** Portal repos used to promote `development` → `qa` → `main` through the nightly promotion agent. Moving to this model retires the `development` branch there, which touches the promotion agents, the PR review agent's gates, branch protections, and the IT request portal's deployment-request lists. Done for all four repos: WildPacks and PPPToo (vertex-agents #249/#264, itrequestportal #89), HostPortal (vertex-agents #268, itrequestportal #111) and BackOfficePrototype (vertex-agents #285, itrequestportal #124, both merged 2026-10-06). The only repo the promotion agents still promote is `n8n-workflows`.

## Migration Plan: BackOfficePrototype, PPPToo, HostPortalPrototype

> **PPPToo: done 2026-10-04 (Phase 2 below). HostPortalPrototype: done 2026-10-04 (Phase 3 below). BackOfficePrototype: done 2026-10-06 (Phase 4 below, with the retirement items still open listed there).** The repo comparison table below is the read-only inspection of 2026-10-01 and is kept as the baseline; all three repo columns describe the state *before* conversion. Sizes (S/M/L/XL) are relative guides, not estimates; run `estimate-my-workload` before committing dates.

### Where each repo is today, compared with the sandbox

| Dimension | Sandbox | BackOfficePrototype (BOP) | PPPToo (before 2026-10-04) | HostPortalPrototype (before 2026-10-04) |
| --- | --- | --- | --- | --- |
| Integration base | `main`; everything cut from `main` | `development` integrates; `qa` is a separately maintained branch; `main` default. Hotfixes also land on `main` directly | `development` is the **default branch**; `main` is 123 commits ahead of and 3 behind `development` | `main` default, but `main` carries a **prod-only baseline** (retired GKE manifests, prod Cloud Run workflow), so dev and QA are not `main`-derived |
| Deploy triggers | `feature/**`, `release/**`, `main` | push to `development`, `qa`, `main` (fixed names) | same fixed names | same fixed names |
| Deploy workflows | one, parameterised by branch | three, plus legacy `cloudbuild*.yaml` | three, plus an unused Cloud Deploy pilot and Keycloak image workflows | three that diverge (PostHog steps, secrets, ports), plus legacy workflows |
| Runtime | Cloud Run, `us-central1` | **GKE**, `us-central1`, kustomize directory per environment, GCE Ingress + ManagedCertificate | Cloud Run (server, client, Keycloak) in `northamerica-northeast1`, behind a per-environment global load balancer | Cloud Run behind a per-environment load balancer with serverless NEGs. Dev and prod `northamerica-northeast1`, QA `us-central1`; service names, regions and registries differ per environment |
| Environments | pooled: 3 dev, 3 QA, 1 prod | one fixed environment each | one fixed environment each | one fixed environment each; QA also has leftover GKE scaffolding |
| Prod release | blue-green revisions, smoke test, canary, rollback | RollingUpdate; no rollback job (QA's rollback and E2E jobs exist but are disabled with `if: false`); `kubectl rollout undo` by hand | 100% to the new revision; only the Keycloak image pilot uses `--no-traffic` plus tag promotion | 100% to the new revision; the "3 failing prod deploys" seen on 2026-10-01 were fixed by #614 before conversion (prod matched `main`) |
| State | none | Cloud SQL per environment; migrations run on **every deploy before rollout**, not backward-compatible by policy, no down-migration | no database in the BFF; Keycloak has its own; sessions are bearer tokens | no own database found; calls the Globus API; session secret |
| Identity / hostname coupling | n/a | Keycloak/Auth0 issuer and redirects in manifests; hostnames in `ingress.yaml`; PPPToo origins hard-coded in code | Keycloak redirect URIs, `CORS_ORIGINS`, `KEYCLOAK_ISSUER` fixed per environment; client API URL **baked in at build** | Auth0 tenant per environment; the SPA redirects to `window.location.origin`, so every host must be in Auth0's callback list; CORS is a literal list |
| Merge gates | none | strong (rulesets plus classic protection); `qa` has no required checks | `build-test` required on `development` only; **`main` unprotected**; deploys do not wait for CI | **no required checks at all**; CI runs on PRs only |
| Promotion | none | agent promotes `development` → `qa`; `qa` → `main` by merge-commit PR | same pattern, merge-commit ruleset on `qa` | same pattern; `main` vs `qa` shows a permanent "diverged" signal from the prod-only baseline |
| Image | built per environment | built per environment on the runner; Vite keys baked in, so an image cannot be promoted | built per environment; API URL baked in | built per environment |
| Pipeline length | ~2 min | 9–16 min | ~1.5–2 min | ~3–4 min |

### Decisions needed before any repository moves

1. **Slot hostnames behind load balancers.** Cloud Run domain mappings are not allowed in `northamerica-northeast1`, where PPPToo and HostPortal live. Options: (a) host rules plus a wildcard certificate on each existing load balancer, with one serverless NEG per slot; (b) put pool services in `us-central1` and use domain mappings as the sandbox does; (c) fewer slots with static hostnames. BOP needs the GKE equivalent (an Ingress or Gateway host rule and certificate per slot). *PPPToo and HostPortal chose (b) for dev/QA and left prod where it is.*
2. **Artifact strategy.** The sandbox builds per environment, so production ships a fresh, untested build of `main`. BOP and PPPToo bake environment values into the bundle. Decide whether to accept build-per-environment or to move runtime configuration out of the bundle so one artifact can be promoted. *PPPToo and HostPortal accepted build-per-environment; the image carries the commit SHA (`GIT_SHA` / `VITE_GIT_SHA` build args) and the cutover verifies it.*
3. **Slot data isolation (BOP).** A database or schema per slot on the shared Cloud SQL instance, versus slots sharing the environment's database.
4. **Identity redirects.** Whether wildcard callback and redirect URIs are acceptable in Auth0 and Keycloak; otherwise per-slot URIs must be registered and managed in Terraform. *PPPToo and HostPortal: fixed slot hostnames, so literal entries per environment, no wildcards. HostPortal's Auth0 entries are registered by the Management API rather than Terraform; the dev tenant's management clients lack the `update:clients` scope and need a one-time grant.*
5. **Re-baselining `main`.** How `development` and `qa` history is reconciled into `main` for each repo, without rewriting history. *PPPToo and HostPortal: a last legacy promotion carried the tooling to `main`; `main` already contained everything else, so the switch was a ruleset change (default branch was already `main` on HostPortal).*
6. **The open model questions** already tracked in the Open Questions section: hotfix review, backport target, long-lived release slot hold.
7. **Side-effect isolation in pools (BOP).** How non-production slots are guaranteed never to receive production SEVIS, SendGrid or n8n credentials.

### Order, and why

1. **Phase 0 — Baseline and prerequisites (all repos).** Fix what is already broken or stale, so the migration starts from known state.
2. **Phase 1 — Shared foundation.** Extract the sandbox mechanism into reusable parts and make the decisions above.
3. **Phase 2 — PPPToo (pilot, size M). Done 2026-10-04.** Healthiest pipeline of the three, no database in the BFF, Cloud Run.
4. **Phase 3 — HostPortalPrototype (size M). Done 2026-10-04.** Also stateless and Cloud Run; followed the pilot the same day.
5. **Phase 4 — BackOfficePrototype (size XL, last). Done 2026-10-06.** GKE, a database per environment, deploy-time migrations and irreversible SEVIS side effects. It needed the most new capability and carried the most risk, so it went last, on proven shared parts.
6. **Phase 5 — Retire the old model.** Only after each repo has run at least one full release cycle on the new one.

### Phase 0 — Baseline and prerequisites (size S–M per repo)

- **HostPortal (done 2026-10-04):** prod already matched `main` (#614 had fixed the failing deploy); the QA GKE namespace was empty; the Deploy QA workflow had been red for 11 days only because its post-deploy Playwright job lacked repo secrets (now skipped unless `vars.HOSTPORTAL_QA_E2E_ENABLED`); `main` protected; 69 merged branches pruned (338 left, mostly squash-merged batches).
- **BOP:** decide whether to re-enable or delete the disabled QA E2E and rollback jobs; resolve the `SevisAlpha` branch references in workflows; refresh `docs/environment-promotion-strategy.md`, which still calls production "future".
- **PPPToo (done 2026-10-04):** `main` protected; the Platform Pulse notifier's parse failure (442 consecutive phantom failures) fixed; the unused Cloud Deploy pilot disabled; 38 merged branches pruned (159 unmerged left for a prune rule).
- **All repos:** prune stale branches (100+ in BOP) so rulesets and agents are not reasoning over noise.
- **Verify ownership *(unverified)*:** which load balancers, certificates and Workload Identity pools are managed in `alliance-gcp-infra` and which are not.

Exit: each repo's three deploy workflows are green and match; `main` is protected; the decisions above are recorded.

### Phase 1 — Shared foundation (size M)

- Extract the sandbox `deploy.yml` logic into one **reusable workflow** (branch type → environment, build, claim slot, deploy; blue-green on `main`) that each repo calls with its own parameters (service names, regions, projects, health URL). *Not done yet: WildPacks, PPPToo and HostPortal each carry a verbatim copy of the sandbox scripts; the PR review agent flagged the duplication on alliance-gcp-infra #349 and it is tracked for after the pilots' first release cycle.*
- Extract `allocate-slot.sh`, generalised so the slot lock lives in labels on Cloud Run services, and as an equivalent lease for GKE (see BOP).
- Add a Terraform module in `alliance-gcp-infra` for a pool: N pre-provisioned services or namespaces, per-slot hostnames, per-environment Workload Identity condition restricted by branch type (`feature/`, `release/`, `main`) as in the sandbox.
- Add the new ruleset pattern, copied from the sandbox (`.github/rulesets/*.json`): protect `main` and `release/**` (PR required, no force-push or delete, required checks, 1 approving review), add the branch-name creation allow-list, and add the `branch-rules` workflow. Make deploys depend on CI passing. Note that a `branch_name_pattern` rule is accepted but **not enforced** on the current GitHub plan, so the creation allow-list plus the required check is the working pattern.

Exit: the reusable workflow and pool module are proven on the sandbox, with the slot-hostname and artifact decisions made.

### Phase 2 — PPPToo (pilot) — Done 2026-10-04

What was done, in order (Asana 1219058470201463 and its five phase subtasks hold the evidence):

1. **Phase 0.** `main` protected; `notify-platform-pulse.yml` given its `workflows:` list; Cloud Deploy pilot disabled; 38 merged branches pruned.
2. **Tooling (PPPToo #429).** The sandbox quality gate, `branch-rules`, pool/blue-green scripts and the three sandbox-model workflows landed as ordinary code on the legacy integration branch. Deviations recorded in the config: `max-lines-app` is a **ratchet at 2400** (31 files exceeded 300), the console rule is scoped to the client, the field-injection regex is anchored to line start, `deploy/**` / `terraform/**` / both Dockerfiles are protected paths. `GET /api/v1/version` and `<meta name="build-sha">` were added as the cutover probes. `blue-green.sh` gained `PROBE_PATH`, `PUBLIC_URL`, `DEPLOY_ARGS` so it runs once per service of the pair.
3. **Infrastructure (alliance-gcp-infra #349, #350).** Dev/QA pools `ppp2-{server,client}-{dev,qa}-{1,2,3}` in `us-central1`; WIF pool `github-actions-ppp2` per project (dev pinned to `trusted-deploy-dev.yml@main`); deployers `ppp2-deployer-{dev,qa,prod}`; prod identity scoped to the existing pair. Twelve domain mappings created by impersonating the verified `terraform-tf@`; twelve GoDaddy CNAMEs. Keycloak slot redirect URIs registered in dev and QA (the Terraform admin client had to be granted `manage-clients` first; now declared in the module).
4. **Switch.** One last legacy promotion (`development` → `qa` → `main`, PPPToo #434/#435) carried the tooling to `main`; default branch switched to `main`; `release/2026-10-ppp2-pilot` cut; rulesets created in order (Branch names → Protect main / Protect release → Freeze legacy branches); legacy deploy workflows disabled; promotion agents, review-eligibility agent, branch-drift agent and the IT request portal updated (vertex-agents #264, itrequestportal #89); `deploy-qa` / `deploy-prod` / `task-workflow` / `pr-review` skills updated.
5. **Proof.** feature → dev slot 1, feature → release (#437), release → QA slot 1, release → main blue-green (#439; the earlier push to `main` had already cut over the tooling release), forced rollback on the prod server, hotfix → main (#436, no deploy). The PR review agent reviewed the first PRs on the new base branches after its redeploy.

Left open from the pilot: branch prune pass 2 (159 unmerged branches, rule to be decided); the dev Keycloak Terraform apply is blocked by an empty `globus-development-sendgrid-api-key` secret; `deploy/README.md` still describes the legacy rail until a governed PR updates it; the shared pool module (Phase 1) is still to be extracted.

### Phase 3 — HostPortalPrototype — Done 2026-10-04

What was done, in order (Asana 1219139535095088 and its five phase subtasks hold the evidence; it is a subtask of the rollout ticket 1219058294224793):

1. **Phase 0.** Ruleset "Protect main" created (PR required, no bypass); 69 merged branches pruned; the 11-day-red Deploy QA diagnosed as a missing E2E secret and gated on a repo variable; prod confirmed equal to `main`.
2. **Tooling (HostPortalPrototype #617).** Same vendored gate and scripts as PPPToo. Repo-specific: `scripts/build-images.sh` runs the Vite build first (the client Dockerfile expects a prebuilt bundle) with Auth0/PostHog values read from Secret Manager per `deploy/env/<env>.build`; `deploy/env/<env>.yaml` + `.server-secrets` + `.client-secrets` mirror the live services' env (`CORS_ALLOWED_ORIGINS` rewritten per slot by `deploy-slot.sh`); `GET /api/version` added next to `/api/health`; `max-lines-app` ratchet 2400; the Globus-URL rule allows the development URL (the code's fallback) and forbids QA/prod URLs; nine debug `console.log` calls removed to satisfy the client console rule. Two CodeQL "high" findings in the vendored gate (file-system race in `gate.mjs`, incomplete escaping in `report.mjs`) were fixed here and the fixes copied back to the skill templates (backport to the other repos tracked as Asana 1219139538996025).
3. **Infrastructure (alliance-gcp-infra #353).** Dev/QA pools `host-portal-{server,client}-{dev,qa}-{1,2,3}` in `us-central1` running as the existing per-environment runtime accounts; WIF pool `github-actions-hostportal` per project (dev pinned to `trusted-deploy-dev.yml@main`, prod to `refs/heads/main`); deployers `hostportal-deployer-{dev,qa,prod}` with `secretAccessor` on the four build-time secrets and reader on the private npm registry. Dev and QA applied by CI; prod applied as 14 single-`target` dispatches. Twelve domain mappings (impersonating `terraform-tf@`), twelve GoDaddy CNAMEs. QA Auth0 client given the three slot hosts; **dev Auth0 still open** (no management client with `update:clients`).
4. **Switch.** Last legacy promotions `development` → `qa` → `main` (#618, #619) carried the tooling to `main`; `release/2026-10-hostportal-pilot` cut; rulesets Branch names 24467828 → Protect main 24464581 (updated with checks, 1 approval, Code Owner) → Protect release branches 24467830 → Freeze legacy branches 24467833; legacy rulesets 18085163 / 20686782 deleted; legacy workflows Deploy Development / Deploy QA / Deploy to Production (Cloud Run) disabled; `governance-approved` label created. Consumer changes prepared as vertex-agents #268 and itrequestportal #111 (both awaiting a second approver); the `deploy-qa` / `deploy-prod` / `task-workflow` / `pr-review` / `qa-promotion-conflict-resolution` / `sandbox-branching-model` skills updated and synced.
5. **Proof.** feature → dev slot 1 (`feature/1219141530843533-pool-proof`, `/api/version` returns the exact SHA); feature → release PR #620 passes `quality-gate`, `branch-rules` and the `ci` jobs; release → main blue-green on `b27a182` for both services (first attempt rolled back correctly on the canary; see Known Limitations), previous revisions tagged `blue`; hotfix push produced no run; protected-path PR #621 blocked by `quality-gate`; branch-name allow-list rejected four bad names. Pending: the release → QA slot deploy (needs #620 merged) and the consumer PR merges.

Left open from the conversion: dev Auth0 slot callbacks (scope grant or dashboard edit); the repo's `README.md` / `.github/rulesets/*.json` still carry the pre-switch wording until a governed PR updates them; Phase 5 retirement after a trust period; the branch-prune rule for the ~330 squash-merged legacy branches.

### Phase 4 — BackOfficePrototype — Done 2026-10-06

BOP was the hard one: it is the only repo on GKE, with a database per environment and irreversible side effects. It is live on the GKE edition of the model:

- `feature/<16-digit Asana GID>-*` branches deploy to a **development pool slot** `globus-dev-N` (`dev-N.globus.allianceabroad.com`) through `deploy-feature-request.yml` / `trusted-deploy-dev.yml`.
- `release/*` branches (current: `release/2026-10-globus-pilot`) deploy to a **QA pool slot** `globus-qa-N` (`qa-N.globus.allianceabroad.com`) through `deploy-release-qa.yml`.
- A `release/*` → `main` pull request (1 approval, Code Owner review, the `governance-approved` label for protected paths) runs `deploy-prod-bluegreen.yml`, a GKE Gateway blue-green cutover. Production cut over to it on 2026-10-05 at 08:11 UTC. `hotfix/<GID>-*` → `main` goes through the same cutover.
- `deploy-development.yml`, `deploy-qa.yml` and `deploy-production.yml` were disabled on 2026-10-05. The `development` and `qa` branches are dead rails: nothing deploys from them, `qa`…`main` is permanently diverged, and the 18 pull requests still targeting `development` on 2026-10-06 must be retargeted to the release branch.
- The promotion agents no longer promote BOP (vertex-agents #285, 2026-10-06: the dev-to-qa-sync-agent dispatches `deploy-release-qa.yml` at a release branch instead, and the qa-to-prod-sync-agent's default promotion list is empty); the Tech Request Portal treats BOP as a sandbox-model service (itrequestportal #124); form-engine bumps arrive as `feature/1219202098622754-form-engine-<ver>` branches cut from `main` into the newest release branch (form-engine-portable #183).

Still open: rulesets on `release/**`, `feature/**` and `hotfix/**`; the freeze ruleset on `development` and `qa` (Asana 1219141533420592); retiring the frozen legacy GKE deployments; and the consumer decision for the legacy hostnames `development.globus` / `qa.globus`, which still point at the frozen legacy ingresses (their cutover to the environment-index board was reverted on 2026-10-05).

The plan as written before the conversion, kept for the record:

1. **Re-baseline.** `development`, `qa` and `main` all have diverging history and `qa` has no required checks. Reconcile into `main` by merge and add a `release/*` ruleset (merge-commit only, same required checks as `main`).
2. **One workflow on GKE.** Keep `kubectl apply -k`, but fold the three per-environment manifest directories (`k8s/{development,qa,prod}`, which hard-code namespace, project, database, service account, IP and hostname) into a shared base with overlays. A slot becomes an overlay.
3. **Pool on GKE.** Cloud Run labels are not available, so the slot lock becomes a lease on a namespace or ConfigMap (the same holder-and-expiry idea). Pre-provision slot namespaces, each with its own ExternalSecrets, hostname and certificate (an Ingress `ManagedCertificate` per slot, or one wildcard certificate), and DNS.
4. **Slot data.** Per decision 3, a database or schema per slot on the shared Cloud SQL instance, created from a template snapshot, with migrations run as a Job per slot. Dev and QA also run a seed step today, which must target the slot's database only.
5. **Blue-green on GKE.** Add GKE Gateway API with a weighted `HTTPRoute` and two Deployments (blue and green) behind one route; promotion is a weight change and rollback is flipping it back. This is new cluster capability, not a policy switch.
6. **Migration policy.** Migrations run before rollout on every deploy and have no down path, so old pods run against the new schema during a cutover. Adopt **expand/contract**: a new migration must be backward compatible with the previous release, enforced with a CI check alongside the existing migration-idempotency gate.
7. **Side-effect isolation.** Enforce `SEVIS_ENVIRONMENT=beta` in every pool slot and lint overlays so they cannot reference production SEVIS, SendGrid or n8n credentials. Define how a cutover guarantees SEVIS submissions are not run twice or dropped, since a rollback cannot undo them.
8. **Artifact.** Decision 2 matters most here: the image bakes Vite keys for Google Maps and PostHog, so QA's artifact cannot be promoted to production.

Exit: feature, release and prod cutover run on BOP with a verified rollback, an expand/contract check in CI, and slot databases isolated from shared environments.

### Phase 5 — Retire the old model

- Delete the `development` branch (and `qa` where it is no longer a deploy source) in each migrated repo, and remove the three fixed-name deploy workflows after one release cycle.
- Disable the development → QA sync agent for each migrated repo at its cutover and update the `/deploy-qa` and `/deploy-prod` skills; freeze the agent per repo during the transition so it does not promote into a repo that no longer uses those branches. *Done for all four repos (vertex-agents #249 WildPacks, #264 PPPToo, #268 HostPortal, #285 BackOfficePrototype on 2026-10-06); the only repo the promotion agents still promote is `n8n-workflows`, and the qa-to-prod-sync-agent's default promotion list is now empty.*
- Extend the PR review agent's repo list and update its gates, update `task-workflow` and `pr-review`, and update the promotion documentation. *Skills done for PPPToo, HostPortal and (2026-10-06) BackOfficePrototype; HostPortal's eligibility config landed in #268.*
- Keep the old workflows disabled, not deleted, until the repo has completed a release cycle on the new model.

### Risks and mitigations

| Risk | Mitigation |
| --- | --- |
| BOP production: irreversible SEVIS side effects and a migration with no down path | Migrate BOP last; expand/contract migrations; enforce non-prod SEVIS isolation; rehearse a cutover and rollback on QA first |
| Wildcard certificates, DNS and load balancer rules for slot hostnames | Decided on the pilot: `us-central1` slots with domain mappings and fixed hostnames; prod untouched behind its LB |
| Auth0 and Keycloak redirect sprawl | Fixed per-slot entries in Terraform/tfvars (Keycloak) or via the Management API (Auth0); no wildcards |
| The promotion agents keep promoting during the transition | Freeze the agent per repo at its cutover (done for WildPacks, PPPToo, HostPortal via frozen branches + #268, and BackOfficePrototype via #285) |
| Branch rules block the first release-branch merges | Add `release/*` rulesets and review the bypass actors before the first release branch is cut; set `do_not_enforce_on_create` on the release ruleset's checks |
| First production cutover cannot pass its canary | Ship the probe route through the old path before the first blue-green (HostPortal lesson) |
| Pool cost and quota (up to 6 extra slot environments per repo) | Start with fewer slots on the pilot; size the pool from observed occupancy |
| Production state drifts from `main` while repos are re-baselined | Phase 0 confirms prod equals `main` before any re-baseline |

### What would tell us it worked

For each repo: a feature branch deploys to a pool slot; a release branch deploys to a QA slot; a `release/*` → `main` merge cuts over with a verified rollback; no `development` branch exists; deploys wait for CI; and a hotfix to `main` deploys through the same cutover. **PPPToo met all of these on 2026-10-04. HostPortalPrototype met all but the release → QA slot deploy on 2026-10-04 (pending its first release-branch merge). BackOfficePrototype runs feature, release and production deploys on the model since 2026-10-06, but its `development` and `qa` branches still exist as dead rails until the freeze ruleset lands (Asana 1219141533420592).**

## n8n-workflows: A Deliberately Different Pattern

> **Not affected by the model above.** n8n-workflows' three-branch pipeline (`development` / `qa` / `main`, one n8n instance per branch) is independently justified by n8n's own source-control feature (one instance paired with one git branch per environment) and by the wider n8n user base's practice at scale. Its branches mean something different underneath: each one mirrors a specific **live running n8n instance's** state, not a deployable codebase promoted through generic environments.

### Why the pattern is different here

n8n's branches aren't a deployable-codebase-promoted-through-environments story — each branch is a version-controlled mirror of one specific **live running n8n instance's** state (workflows, tags, credential/variable stubs), and a PR is the review gate for promoting a change from one running system to another.

### Sandbox: An Isolated Tier Upstream of `development`

As of 2026-09-08 (instance) / 2026-09-10 (git branch + tooling), n8n-workflows also has a `sandbox` tier — a fully isolated GKE-based n8n instance (namespace `n8n-sandbox`) with its own git branch (`sandbox`) and its own integration-test harness (`scripts/test_integration.py`, run via the `Test Sandbox` Action). It sits **upstream of** the three-branch pipeline above, not inside it. (This is unrelated to the `branching-sandbox` repo described on this page.)

- **Development and isolated testing happen in sandbox first.** Workflow authors build and iterate against the sandbox n8n instance. Sandbox's integration tests fire real synthetic events at a running instance end-to-end, which the pipeline's own gates (`validate_workflow_contracts.py`, `validate_references.py`) don't do.
- **Only reviewed, ready workflows are exported to `development` for integrated testing.** The `Export Sandbox Workflows` Action snapshots sandbox's live state to the `sandbox` branch and flags any workflow that exists only there. A human runs `Promote Sandbox Workflow` for anything genuinely ready, opening a **review PR against `development`** — never auto-merged.
- **Sandbox is not part of the promotion path and is not a deploy target.** `deploy_workflows.py` refuses `TARGET_ENV=sandbox` on purpose.

### What does need to change: which workflows belong on which instance

1. **Credential/environment scope.** Does the workflow need to reach real production systems? If so, it needs prod-scoped credentials and belongs in the prod instance.
2. **Reliability requirement.** The `development` instance is a low-governance iteration sandbox by design — fine for throwaway experimentation, harmful for anything whose job is to notice when something else is broken.

**Workflows that should move to prod, not stay parked on dev:** Platform Pulse alerting, agent-issue-resolution triggers, and security-remediation workflows.

**What should stay dev-only:** genuinely experimental automations someone is actively building and iterating on, with no operational dependency on them working right now.

## Why This Isn't GitFlow

Classic GitFlow's problem is a long-lived `develop` branch that accumulates work for a long time before a manual, risky batch-sync into a release branch. Trunk-based development is a rule about **branch lifetime** — short-lived branches, integrated often — not about deleting pre-production environments. This model keeps both properties: there is no long-lived integration branch at all, feature branches are short-lived and deploy immediately to a pooled development environment, the release branch is bounded and disappears when it ships, and the blue-green cutover makes production rollback a traffic flip rather than a redeploy.

## Open Questions

### Decided 2026-10-08 (Asana 1219058395898772)

- **Release branch after it merges to `main`:** auto-delete it. Production `g-<sha>` revision tags stay pruned to the newest 5. **Implemented 2026-10-09** in PPPToo, HostPortalPrototype, WildPacksApplicationForm, BackOfficePrototype and the sandbox (`release-branch-cleanup.yml` deletes the release branch once its PR into `main` merges; the live "Protect release branches" rulesets no longer carry the `deletion` rule).
- **Hotfix PR review:** the PR review agent plus a human. Hotfix PRs into `main` are already in the agent's base-branch list.
- **Hotfix backport target:** one PR per open `release/*` branch, not `feature/*` branches. No auto-merge. When a backport conflicts, leave the PR open, label it and post to Platform Pulse. **Implemented 2026-10-08/09** (`hotfix-backport.yml`): a merged hotfix pushes a `feature/<GID>-backport-*` branch for every open release branch that lacks it and comments the compare links on the hotfix PR, because GitHub Actions cannot open pull requests in this organisation; conflicts are noted and left for a human (Asana 1219058395899893).
- **Long-lived named-initiative releases:** a longer QA-slot reservation TTL for `release/*` (the value is chosen when this is built). No pinned flag and no larger pool. **Implemented, 24 h** (feature branches keep 30 min): in each portal repo's `deploy.yml`, and for BackOfficePrototype in the shared slot allocator `scripts/allocate-slot-gke.sh`. Observed live on BackOfficePrototype: the release branch's QA slot was reserved until 24 h after its last deploy.

### Fixing a file that exists only on a release branch

A fix to a file that lives only on a release branch (an in-flight feature file, the release QA workflow) cannot be cut from `main`, and a branch cut from the release fails the "cut from `main`" rule. The compliant route (proven live on BackOfficePrototype, 2026-10-09; no rule is loosened):

```
git checkout -b feature/<GID>-<slug> $(git merge-base origin/main origin/release/<name>)
git merge --no-ff origin/release/<name>   # must be --no-ff: a fast-forward is the same as cutting from the release
# edit the release-only files, commit, open a PR into release/<name> (the diff shows only the fix)
```

The branch-rules check decides what a branch was cut from by its first-parent lineage and allows merging the pull request's **own base branch** in; history merged in from any other branch is still rejected. PRs into a release branch run the release branch's copy of `scripts/check-branch-rules.sh`, so that copy must be the current one: it is on every sandbox-model repo's release branch as of 2026-10-09 and reaches each repo's `main` with its next release.

### Still open

- Does a feature that misses a release get re-cut from the new `main`, or rebased?
- Merge queue for concurrent releases landing on `main`: needed, and when?
- Is the release batch cadence weekly (earlier assumption) or per feature, and does that change the pool size?
- The remaining migration decisions for BOP (slot data, side-effect isolation) and HostPortal's dev Auth0 callbacks (management-client scope).

## Required Follow-up: Skill Updates

The `task-workflow`, `pr-review`, `deploy-qa`, `deploy-prod` and `qa-promotion-conflict-resolution` skills were updated on 2026-10-02 (WildPacks), 2026-10-04 (PPPToo, HostPortal) and 2026-10-06 (BackOfficePrototype): the methodology is selected per repo, sandbox-model repos cut features from `main` and merge them into the current `release/<name>`, Gate 5 enforces the three allowed routes, and the promotion skills refuse the converted repos. The `sandbox-branching-model` skill holds the templates of every live artifact and the conversion procedure, with the PPPToo and HostPortal learnings recorded.