---
title: "Atlas: LLM-Guided Automated Testing"
slug: atlas-autonomous-testing
section: Governance & Operations
status: stable
description: "Atlas is an advisory testing agent."
last-updated: 2026-10-07
---

# Atlas: LLM-Guided Automated Testing

Atlas is an advisory testing agent. It reads Nexus, designs test scenarios with the Claude API, runs the confirmed ones in a real browser against the development environment, and reports to Platform Pulse. A person decides what happens to every result.

## Overview

Principle: agents recommend, deterministic checks prove, humans decide. The model writes text only and never executes anything. Every scenario passes a linter, every expectation cites a Nexus page, and a failure is a candidate until a person confirms it.

| | |
|---|---|
| Service | `atlas-agent`, Cloud Run (`us-central1`, project `alliance-platform-agents`), IAM-protected, development only |
| Scenario source | Kiwi TCMS plan "Atlas: regression", CONFIRMED cases only ([Test Case Standard](/docs/test-case-standard)) |
| Model | Claude API, scrubbed Nexus documentation only; planning runs on a developer machine, the cloud service makes no model calls |
| Status of results | Advisory. Nothing Atlas produces blocks a merge, a deploy or a promotion |
| Code | `atlas-agent/` in the `vertex-agents` repository (service and a vendored copy of the harness); the harness source is a local repository that is not yet consolidated |

## What Atlas Never Does

- Run against production. Production hostnames are blocked as critical even if added to an allow-list by mistake.
- Run against QA today. The gate accepts QA hosts, but `atlas-agent` is registered for development only and QA has never been run.
- Block, approve, merge or promote anything.
- Confirm its own scenarios. Atlas can propose a case (it lands in Kiwi as PROPOSED); only a named person makes it CONFIRMED, and only a person can designate anything as blocking.
- Use a real person's account. Test identities are dedicated bot accounts; failed sign-ins are never retried.
- Perform SEVIS or DS-2019 writes, payments, role or permission changes, credential resets, configuration publishing, or bulk operations. These are deny-listed in the harness.
- Send applicant, host or other personal data to the model.
- File, update or close tickets. A tracker adapter exists but no client is wired, so filing is refused.

## How a Run Works

| Stage | What happens |
|---|---|
| Fetch | The service reads the CONFIRMED cases of the Kiwi plan. If Kiwi is unreachable or nothing is runnable, the run does not start and Platform Pulse is told why. There is no bundled fallback |
| Lint | Each scenario is checked against the policy gate (allowed hosts, deny-list, identity, fixed step vocabulary) |
| Run | Playwright drives a real browser as a bot identity; every request and redirect passes the gate; a failing scenario gets up to three attempts in total |
| Judge | Deterministic assertions decide the verdict: `pass`, `fail`, `flaky` (passed on a retry), `blocked` or `inconclusive`. The model is not the judge |
| Triage | A deterministic tree classifies a failure as product defect, scenario defect, environment or flaky |
| Report | One Platform Pulse message (bot channel, HTML) and a Kiwi test run with one execution per case. `flaky` is recorded as ERROR, never as a pass |

## How to Trigger It

| Mode | How | Notes |
|---|---|---|
| Manual | `POST /run-live` on the service | Needs a Google ID token and the API key. Takes about 9-10 minutes for the 12-scenario regression set |
| Daily | Cloud Scheduler job `atlas-agent-daily-regression` calls `POST /trigger/schedule` | Weekdays, same set, then posts the summary |
| After a deploy | `POST /trigger/deploy` | The route is live, but no pipeline calls it yet. The first trigger was built for a single development host and was retired when development deploys moved to pool slots; a redesign is tracked |

Only one run executes at a time. Operator procedures (a failed run, a flaky scenario, a missing notification, a rollback) are in `RUNBOOKS.md` in the `atlas-agent` directory of `vertex-agents`.

## Where Evidence Lives

- **Kiwi TCMS**: the test run, one execution per case, with a comment on every non-pass carrying the verdict, attempts, failure class and reason. This is the durable record of pass and fail.
- **Platform Pulse**: the run summary with counts and the not-green scenarios, linking to the Kiwi run.
- **Sealed run record and screenshots**: written by the harness for every run. On a developer machine they are kept under `runs/`. On the cloud service every live run is also copied to a Cloud Storage bucket (`runs/<date>/<run_id>/record.json` plus an `evidence/` folder of screenshots, written once and never overwritten), so a failed run can be explained after the container recycles. Retrieve them with `GET /runs/{run_id}`, `/runs/{run_id}/evidence` and `/runs/{run_id}/evidence/{name}` (API key required); the run id is in the Platform Pulse message. Objects are deleted after 395 days, a proposed retention that is not yet a recorded decision. Runs from before 2026-10-07 were not stored.
- **Build traceability** (commit SHA, image digest, revision, workflow run on each run record) is being added; until then do not assume a run record says which build it tested.

## Guard Rails

- A host allow-list is the source of truth; anything not on it is refused, and production hostnames are refused even if listed.
- The deny-list, allow-list and per-run budget ceilings are policy files changed only by reviewed pull request.
- Prompts are scrubbed before they leave the machine, and a prompt containing a secret pattern is refused rather than sent.
- Bot credentials come from Secret Manager (cloud) or the environment (local), never from a scenario file.
- The harness subprocess gets a clean environment; the service API key is never passed to it.
- Atlas never weakens an assertion to make a scenario pass (healer guard).

## Model and Cost

The Claude API is the model provider, in a dedicated Anthropic workspace named `atlas`. A monthly cap of $150 is approved. The provider-side spend limit in the Anthropic Console is **not yet set**, and the budget alert and kill-switch drill are not yet built, so the cap is currently a policy statement rather than an enforced limit. Every model call is metered against a dated price table and recorded in a cost ledger; a typical planning run costs well under a dollar and total spend to date is a few dollars.

## Autonomy

Decision rights are an earned ladder (policy P-001): rungs 0 to 4 are enabled by policy, but each rung's measured entry criteria are still enforced in code, so in practice only rung 0 (maintenance such as quarantining a flaky scenario) acts today. Rung 5, which covers blocking designations, the Product acceptance suite, budgets, the deny-list, egress, IAM, anything in production, and anything touching SEVIS or government systems, is human-only by design. Every autonomous decision is logged and revertible.

## Ownership and Review

Owner: Platform Engineering. Changes to policy files, the deny-list, the price table or trigger authentication go through pull request review. Promotion of a scenario into the regression plan is a human CONFIRMED decision in Kiwi. An independent Engineering Team review of the original service pull request and its static-analysis suppressions is still outstanding.

## Known Limits

- Atlas has found one real product defect with an AI-designed scenario (a Compliance Console API contract mismatch). That is one data point, not a measured defect-detection rate; its value against real defects is unproven.
- Replay scoring so far used fault-injection fixtures, not the real applications.
- Several Globus staff page-load scenarios are flaky, and their retries are what stretch a run from about 3 minutes to 9-10 minutes.
- Configuration and infrastructure drift (Terraform, secrets, CORS, workload identity) is invisible to browser scenarios.
- Development only: QA and production behaviour are not tested.

## Related Pages

- [Test Case Standard (Kiwi TCMS)](/docs/test-case-standard): how cases are written and logged
- [Testing Practices](/docs/onboarding-testing-practices): what "tested" means for a code change
- [Vertex Agents](/docs/vertex-agents): the agent fleet and registry