--- title: "Debugging & Observability" slug: onboarding-debugging-observability section: Developer Onboarding status: stable description: "Where to look when something is broken, slow, or silently failing — before you start guessing." last-updated: 2026-10-07 --- # Debugging & Observability Where to look when something is broken, slow, or silently failing — before you start guessing. ## Overview Alliance's observability surface is split across a few tools by layer: application/error logs, cluster-level pod health, in-app session/error tracking, a human-facing audit trail, and a single alerting channel that automated agents post to. Knowing which layer a symptom belongs to saves a lot of dead-end investigation. ## Logs Application logs and error logs for any GCP-hosted service are available per environment through the platform's logging tools — filter by service and time range before assuming a failure is new; many "new" errors have been quietly occurring for a while at low volume. BackOfficePrototype logs are structured JSON from a pino logger, mapped to GCP severity labels. `LOG_LEVEL` sets the level (default `info`; `debug` is typical locally), and only when `NODE_ENV` is `development` does the logger switch to human-readable pino-pretty output. Fields such as passwords, tokens, authorization headers and cookies are redacted. ## Pod & Service Health If a service is crash-looping, failing to start, or can't reach a dependency, the first thing to check is pod status and recent events at the cluster level, followed by Workload Identity configuration if the failure looks auth-related (a pod that can authenticate to nothing is a very common failure shape after a service account or binding change). Cloud Run services follow the same "check the running revision's logs and env" pattern. The Hub's `/api/health` and `/api/globus-status` probes report which tier answered in their `environment` field. That label comes from `CONFIG_ENV` (each tier's ConfigMap sets `development`, `qa` or `production`), falling back to `NODE_ENV` and then to `development` only where no ConfigMap exists (local runs, tests). `NODE_ENV` does not identify the tier, because the image sets it to `production` in every deployed tier, so it no longer drives the label. The field name is unchanged. ## Session & Error Tracking The portals (BackOfficePrototype, PPPToo, HostPortalPrototype) use PostHog for session replay and structured error tracking in the browser layer — useful for reproducing a user-reported bug you can't trigger yourself, since you can often watch the actual session that hit it. ## Audit Trail Every gateway tool call is audited (caller identity, tool, timestamp, allow/deny), and Globus itself has a dedicated Audit Console (see the governance section) for tracking who changed what applicant/application data and when. If you're investigating "who did this and when" rather than "why did this crash," the audit trail is the right tool, not the logs. ## Alerting Surface Automated findings (agent failures, workflow errors, health check failures) post to the **Platform Pulse** Teams channel — never as a direct message. If you're building or modifying an automated agent, its alerts belong there, formatted as HTML (Teams does not render Markdown syntax from the API), with a single 🤖 prefix and a status emoji (🟢/🟡/🔴) indicating severity. One scheduled monitor that follows this pattern is the `[Stability] Email Health` workflow. It runs hourly and probes the email surface for the development and QA environments (production is disabled in the workflow): whether the SendGrid key is valid and has the `mail.send` scope, whether Keycloak admin login and an SMTP send (an `execute-actions-email` call on a throwaway test user) succeed, and whether the n8n SendGrid delivery workflow logged more than three errors in the last hour. On any failure it posts a summary to Platform Pulse through the gateway tool `teams_send_bot_channel_message` (the gateway URL comes from the `GATEWAY_MCP_URL` environment variable, which must be set on the n8n pod or the post fails), and in parallel files one Asana ticket per distinct finding in the Platform Pulse project's New Intake section, titled `[Email Health] — ` (for example SendGrid key invalid, Keycloak SMTP send failing, n8n SendGrid delivery errors) and assigned to Sam. Tickets are deduplicated by exact title against the tasks already in New Intake, so a repeat finding is skipped instead of filed again. The ticket step authenticates to Asana with the `ASANA_PAT` environment variable on the n8n pod. ## CI Build Notifications The `n8n-workflows` repository posts GitHub Actions results to Platform Pulse, so pipeline failures and successful environment deploys show up in the same channel as agent alerts. The workflow `notify-platform-pulse.yml` runs on `workflow_run` completion for `ci`, `Deploy to Dev`, `Deploy to QA`, `Deploy to Production`, `Export Dev Workflows` and `Test Sandbox`. It only notifies when the run's branch is `development`, `qa` or `main`, and only in two cases: | Trigger | Condition | Card label | | --- | --- | --- | | Failure | Any watched workflow concludes `failure` (the notifier never reports on itself) | Build Failure | | Deploy success | `Deploy to Dev`, `Deploy to QA` or `Deploy to Production` concludes `success` | Deployed | The composite action `.github/actions/notify-platform-pulse` builds an HTML card (status emoji, repository, workflow name, branch, triggering actor, link to the run) with a single 🤖 prefix and posts it to the Platform Pulse channel by calling the gateway tool `teams_send_bot_channel_message` with `contentType` set to `html`. It authenticates with a dedicated gateway robot PAT stored as the repository secret `PLATFORM_PULSE_GATEWAY_PAT`. > **Notification failures never fail the build** > > If the gateway call errors, returns a tool error or the request itself fails, the action only emits a GitHub Actions warning annotation. The originating workflow is unaffected. The action also maps a `cancelled` status to a yellow "Build Cancelled" card, but the workflow's `if` condition currently only lets failures and deploy successes through, so cancelled runs are not posted. **Cloud Build Teams notifier.** BackOfficePrototype also ships a Cloud Run service, `cloudbuild-teams-notifier` (`infra/cloudbuild-teams-notifier`), that turns Cloud Build events into Teams cards. It exposes `GET /healthz` and `POST /pubsub` (Pub/Sub push; malformed messages get 400, non-terminal or filtered builds get 204, a failed Teams post gets 502 so Pub/Sub redelivers). Configuration by environment variable: `TEAMS_WEBHOOK_SECRET` is the Secret Manager version resource holding the Teams webhook URL (default `projects/skilled-script-469819-f2/secrets/cloudbuild-teams-webhook-url/versions/latest`), and `TEAMS_WEBHOOK_URL` supplies the URL directly and overrides the secret (local and test use). `GITHUB_TOKEN_SECRET` is the optional Secret Manager resource for a GitHub token used to add commit details to the card, and `GITHUB_TOKEN` supplies a token directly; with neither, cards are posted without commit enrichment. `ALLOWED_BRANCHES` (comma-separated, default all branches) and `NOTIFY_ON_SUCCESS` (default true) filter what is posted. **Portal base URL settings.** Email links that point applicants and hosts at a portal are built from two global settings that administrators edit through `/api/admin/applicant-portal/config` and `/api/admin/host-portal/config` (the admin portal config screens). They are stored in `integration_configs` as `applicant_portal/base_url` and `host_portal/base_url`. A saved value wins over the environment fallback (`APPLICANT_PORTAL_BASE_URL`, then `APPLICANT_PORTAL_URL`, for the applicant portal; `HOST_PORTAL_BASE_URL` for the host portal; a saved host value that resolves to the applicant portal origin is refused at read time and skipped), and clearing the field and saving removes the override and restores the fallback. Editing requires the `system_settings.manage` permission. A value must be an http or https URL whose host is not `localhost` or `127.0.0.1` (otherwise `INVALID_URL`), and a trailing slash is removed on save. The Host Portal URL is additionally rejected with `CROSS_PORTAL_ORIGIN` if it shares the Applicant Portal origin, so host invitations cannot be sent to the applicant portal. These settings are read fresh from the database rather than from the shared integration-config cache, so the config screens show the persisted value even when a different server handles the reload, and the email URL resolvers on other servers pick up a change within about 30 seconds. If a link in an email points at the wrong portal, check these two settings first. **Back-office base URL.** Links that point staff at the back office (for example the staff welcome email) use a third base URL. It resolves to the `integration_configs` entry `platform` / `base_url` when set, else the `PLATFORM_BASE_URL` environment variable, else `PUBLIC_BASE_URL`, else the first entry of `REPLIT_DOMAINS` (as an https URL), else the origin of the current request, and finally `http://localhost:5000`. The environment-stable part is cached for 30 seconds; the request origin is never cached. The signed participant-profile PDF link resolves its own origin separately (see Notification Model).