--- title: "Security & Secrets Handling" slug: onboarding-security-secrets section: Developer Onboarding status: stable description: "The rules that keep credentials, tokens, and applicant data safe across a fairly large, multi-environment ecosystem." last-updated: 2026-10-07 --- # Security & Secrets Handling The rules that keep credentials, tokens, and applicant data safe across a fairly large, multi-environment ecosystem. ## Overview Nothing here is exotic — it's mostly "know which vault a secret lives in, know which token type a task actually needs, and never touch a real person's login to make testing easier." The failures that actually happen are almost always one of those three getting skipped, not a sophisticated attack. ## 1Password Vault Model Secrets are organized per environment/purpose, not pooled into one shared vault: - **Globus Dev vault** — dev credentials, dev E2E bot accounts, dev-scoped API keys. - **Globus QA vault** — the QA equivalents; QA credentials are deliberately separate from dev so a dev secret leak doesn't also compromise QA. - **alliance-mcp-services vault** — credentials for the shared MCP server fleet (Teams, GCP, GoDaddy, etc.), isolated from the Globus dev/QA/prod environments entirely. Never commit a secret to a repo "temporarily" — every credential you'll need for local dev or testing already has a home in one of these vaults; if it doesn't, that's a gap to flag, not a reason to hardcode it. ## GCP Secret Manager & External Secrets Production and QA GCP secrets live in Secret Manager and sync into Kubernetes via the External Secrets Operator (ESO). Two failure modes are common enough to know about in advance: - **Trailing newline contamination** — a secret written with a trailing newline (common when piping from a shell) can silently break auth for the consuming service even though the value "looks right" in the console. - **Stale ESO sync** — a secret updated in Secret Manager doesn't always propagate immediately; a forced re-sync is sometimes needed rather than waiting. Asana credentials in the Hub (`server/asana.ts`) are resolved in order: (1) `ASANA_CLIENT_ID` and `ASANA_CLIENT_SECRET`, used only when both are set, to fetch an OAuth2 client-credentials token that is cached until 60 seconds before expiry; (2) `ASANA_ACCESS_TOKEN`, a direct token; (3) the Replit Connectors token. If the client-credentials request fails once (for example a placeholder id or secret in Secret Manager), it is skipped for the rest of the process lifetime and the next mode is used. The Hub only throws when none of the three is available. ## GitHub Tokens: PAT vs App Not every GitHub write action can be done with a personal OAuth token. Workflow-file changes and certain automated pushes require a **GitHub App token**, which the platform mints on demand — a personal PAT will silently lack the scope and fail in ways that look like a permissions bug rather than a wrong-token-type issue. If an automated push fails specifically on `.github/workflows/*` changes, check the token type before assuming it's an IAM problem. ## Never Touch Personal Accounts > **Standing rule, no exceptions without an explicit ask.** Never reset, rotate, or otherwise mutate the credentials of a real person's personal account — Keycloak, Globus, or anything else — for testing, reseeding, or any automated purpose. Doing this to a real colleague's login locks them out of their actual workday for no benefit; dedicated E2E bot accounts exist in every environment specifically so this is never necessary. The only exception is a person explicitly asking you to reset **their own** login because they're genuinely locked out — that's a user-requested fix, not a testing shortcut. ## PII & Data Handling Applicant data (passports, background checks, biographical fields) has its own handling rules covering redaction, storage, and who can view what — see the PII & Data Handling page under Governance & Operations. If your change touches applicant records in any way, read that page before assuming your existing intuition about "sensitive data" covers it — some of the redaction rules are stricter than they look at first glance. ### Hub encryption and signing secrets Four Hub (BackOfficePrototype) environment variables hold keys that must be stable. In production all four throw when unset; outside production each falls back to a random per-boot key, so anything encrypted or signed under it cannot be decrypted or verified after a restart or by another instance. | Variable | Used for | Notes | | --- | --- | --- | | `PII_ENCRYPTION_KEY` | Encrypts MFA secrets and PII fields (`encryptField` and `decryptField` in server/lib/crypto-utils.ts) | Required in production. Decryption tries the env key and, outside production, the per-boot key too. | | `SEVIS_CERT_ENCRYPTION_KEY` | Encrypts stored SEVIS certificates | Required in production; separate from the PII key. | | `PARTICIPANT_PROFILE_PDF_SECRET` | HMAC-SHA256 signing of the short-lived participant-profile PDF URLs that host emails link to | Required in production so signatures verify across Cloud Run instances. Optional `PARTICIPANT_PROFILE_PDF_TTL_SECONDS` sets the lifetime (default 24 hours, clamped to 60 seconds to 7 days). | | `PORTAL_SESSION_SECRET` | HMAC signing of stateless portal session tokens (`POST /api/portal/session`) | Required in production. Optional `PORTAL_SESSION_TTL_SECONDS` sets the lifetime (default 30 minutes, clamped to 60 seconds to 24 hours). | `SEVIS_CERT_PASSWORD` is separate from the four keys above: it is the password of the SEVIS PKCS#12 certificate, read from the environment and with no per-boot fallback. A certificate upload falls back to it (or to an empty password) when the request carries no `password`, and direct SEVIS credentials use it alongside `SEVIS_CERT_PATH`.