---
title: "Test Case Standard (Kiwi TCMS)"
slug: test-case-standard
section: Developer Onboarding
status: stable
description: "How test analysts structure test cases, and how we log them in Kiwi TCMS at `tcms.allianceabroad.com`, the single source of test cases for humans and for the Atlas agent."
last-updated: 2026-10-07
---

# Test Case Standard (Kiwi TCMS)

How test analysts structure test cases, and how we log them in Kiwi TCMS at `tcms.allianceabroad.com`, the single source of test cases for humans and for the Atlas agent.

## Overview

Kiwi TCMS (Kiwi) is the only place a test case lives. Humans write and review cases in the Kiwi web UI; Atlas pulls CONFIRMED cases from a Kiwi test plan before every run and writes a Kiwi test run with one execution per case afterwards. There is no bundled scenario folder in the agent image and no fallback: if Kiwi is unreachable or a plan has no runnable case, the run does not start and Platform Pulse is told why.

This page sets the standard for what a case contains, how cases are grouped, and which Kiwi fields carry which information. It is based on ISO/IEC/IEEE 29119-3, Kiwi's own authoring guidance, and common practitioner guidance (see Sources below). The companion page [Testing Practices](/docs/onboarding-testing-practices) covers what "tested" means for a code change; this page covers the cases themselves.

## Anatomy of a Test Case

ISO/IEC/IEEE 29119-3:2021 (the successor of IEEE 829) gives a test case nine parts. Practitioner guides converge on the same list under slightly different names.

| Part | What it is | Common mistake |
|---|---|---|
| Identifier | Stable, unique, never reused | Encoding the module name in the id so it breaks on reorganisation |
| Title / summary | What is verified, verb plus object, one line, under about 60 characters | "Save attachments", "Login works" |
| Objective | One sentence: the behaviour being checked | Missing, so a reviewer cannot tell what pass means |
| Preconditions / setup | State, data and environment that must already be true, stated precisely (which user, which record, which state) | "User is logged in" without saying who |
| Inputs / test data | The values used, kept separate from the steps so they can vary | Hard-coding data inside step text |
| Steps / actions | Numbered, low level, written for someone with no context, one action per step | Several actions in one step; more than about 15 steps |
| Expected results | One observable, measurable result per step that asserts, never "works fine" | A single expected result at the end of 12 steps |
| Postconditions / breakdown | What state is left and how to clean up; the case should be self-cleaning | Tests that only pass once |
| Environment | Browser, device, environment, configuration | Implied, then the case flakes |
| Traceability | Link to the requirement, story, spec or doc the case proves | Orphan cases nobody can justify deleting |

Priority (business impact) is added by almost every team. The actual result lives on the execution, not the case.

## Writing Rules

- **Atomic.** One behaviour per case. If a feature has three aspects, write three cases, so a failure points at one thing.
- **Independent.** A case must be runnable on its own, in any order, with its own data setup.
- **Deterministic.** No "sometimes". If the environment makes it flaky, fix the precondition or the environment; do not loosen the expected result.
- **Measurable expected results.** Text shown, status code, record count, cookie absent, audit row written.
- **Bounded.** Kiwi's guidance: fewer than 15 steps and executable in under 20 minutes. Split otherwise.
- **Positive and negative.** For every happy path there is at least one invalid-input or forbidden-role case.
- **Named consistently.** Verb plus object in sentence case, the same words for the same thing across cases.
- **Prioritised.** P1 runs every build, P2 every release, P3 when the area changes.
- **Traceable.** Every case cites the requirement or Nexus page it proves; coverage is measured against that.
- **Reviewed.** A second person reads the case before it is trusted. Kiwi models this as PROPOSED to CONFIRMED.
- **Maintained.** Obsolete cases are marked NEED_UPDATE or DISABLED, not left to rot in a suite everyone skips.
- **Shared steps and data separated.** Common sequences (sign in, open record) are referenced, not pasted, so a UI change is one edit.

Format choice: step and expected-result tables suit regression and manual execution; Given/When/Then suits acceptance criteria written with stakeholders. Both fit the same anatomy (Given = preconditions, When = steps, Then = expected results). A common blend is the objective in Given/When/Then and the body as numbered steps.

## Plans, Cases, Builds and Runs

Every test management tool uses the same hierarchy.

| Level | Meaning | Organising rule |
|---|---|---|
| Product and version | The system under test and its release line | One product per system, not per sprint; versions as a controlled list |
| Test plan | Scope, purpose and strategy for an area or type of testing (regression, smoke, release) | Strategy only; steps live in cases |
| Plan membership | Grouping of cases for a purpose | Group by feature area or type; use tags and components for cross-cutting dimensions (browser, role, journey) rather than duplicating cases |
| Test case | One scenario with expected results | Reused across plans and runs, never copied per run |
| Build | The concrete deployed artefact (commit, image, release) | One build per deploy; never reuse one build forever |
| Test run | Execution of selected cases against one build in one environment | One run per build and environment; temporary, not a backlog |
| Test execution | One case inside one run: status, comment, bug links, who, when | Always leave a comment or bug link on anything that is not a pass |

## How Kiwi TCMS Expects It Logged

Kiwi's design principle, from its own introduction: "If it is not in the TCMS then we do not do it, and conversely, if we do it then it is in the TCMS."

**Entities.** Classification, then Product, then Version, then Build. A Test Plan (types include Unit, Integration, Function, System, Acceptance, Smoke, Regression) holds Test Cases. A Test Run (plan + build + manager) holds Test Executions. Cross-cutting: Tag, Component (a feature or module of a product), Category (per product), Priority (P1 to P5), Properties (free key/value pairs on cases, runs and executions), bug and issue-tracker links, attachments.

**Test case fields.** Summary, text (Markdown), notes, requirement, extra link, priority, category, status, is automated, script, arguments, default tester, reviewer, setup and testing duration, components, tags, plans, properties.

**The text field has a built-in four-section template.** Kiwi merged its old separate fields into this layout and the editor pre-fills it:

```
**Setup:**

**Actions:**

**Expected result:**

**Breakdown:**
```

Kiwi's authoring appendix: the summary is verb plus object, under 60 characters, sentence case; actions are written "clearly and at a low level, assume the tester has no experience"; expected results are stated in accurate terms, never "works fine"; cases should be accurate, economical, repeatable, appropriate, traceable and self-cleaning.

**Review workflow.** New cases are PROPOSED. A reviewer checks the environment setup, test data, summary, prerequisites, clear actions with fewer than 15 steps and clear effects, then sets CONFIRMED. Only CONFIRMED cases can be added to a run. NEED_UPDATE and DISABLED cover drift and retirement.

**Runs and executions.** Create a run per build (and per environment if more than one), add the cases, execute, and set the status (IDLE, RUNNING, PAUSED, PASSED, FAILED, BLOCKED, ERROR, WAIVED) with a comment and bug link where relevant. The Telemetry pages then report pass rates per plan, build, tester and over time.

**Dimensions.** Use tags, components, categories and plan membership for browser, role, environment, journey and feature. Do not import a spreadsheet row per variant: normalise first, or the first automation import creates hundreds of near-identical records that nobody trusts.

**Anti-patterns Kiwi users warn about.** A product per sprint; free-text version labels; one build reused forever; steps written in the plan document; the same case duplicated per run; a run kept as a permanent backlog; execution status changed without a comment or bug link; never confirming cases; never closing obsolete plans; letting automation create inconsistent products and versions.

## Conventions for tcms.allianceabroad.com

| Kiwi field | Convention |
|---|---|
| Product | `Alliance Platform` (one product; versions are a controlled list) |
| Test plan | One per purpose, named `Atlas: <set>` for plans Atlas runs (today `Atlas: regression`); human-only plans use a plain name |
| Summary | Verb plus object, sentence case, under 60 characters. Example: "Load applicant intake pages as staff" |
| Text | Kiwi's four sections: Setup (identity, environment, start state), Actions (numbered, one action per line), Expected result (one per asserting step), Breakdown ("read-only, nothing to clean up" or the cleanup) |
| Status | Author as PROPOSED; a second person confirms. Atlas runs CONFIRMED cases only |
| Priority | From risk: impact 3 = P1, impact 2 = P2, else P3 |
| Category | `Atlas` for automated cases, `Manual` for human-only cases, so a run can mix both and the agent materialises only the automated ones |
| Components | One per portal area: Applicants, J-2, Compliance, SEVIS, Admin, Partners, Reporting, Finance, Notifications, Sign-in |
| Tags | Journeys `J01` to `J20`; identity used (`staff-e2e-dev`, `applicant-e2e-dev`); any cross-cutting dimension |
| Requirement | The Nexus page the case proves (its `/docs/<slug>` path), so coverage can be read per page |
| Is automated + script | Set for Atlas cases; the Automation script field holds the harness scenario JSON (`id` of the form `S-Jxx-nnnn`, `steps`). Humans do not edit it by hand |
| Properties | `scenario_id` carries the harness id, so the summary stays human-readable |
| Build | One per deploy, named `<repo> <sha>` for deploy-triggered runs; scheduled runs use the date and time |
| Run | One per build and environment. Environment is a run dimension, not part of the case |
| Execution | Status plus a comment on every non-pass: verdict, attempts, failure class and reason, link to the run record |

Housekeeping: close plans that are no longer run, mark drifted cases NEED_UPDATE instead of editing silently, never reuse a build name across deploys.

## How Atlas Reads and Writes Kiwi

Atlas (see [Vertex Agents](/docs/vertex-agents)) signs in to Kiwi as the `atlas-agent` user over JSON-RPC.

1. **Before a run**: it finds the active plan named `Atlas: <set>`, filters its CONFIRMED cases, and writes each case's Automation script into a fresh directory the Playwright harness reads. Cases whose script is missing, not valid JSON, has no `S-Jxx-nnnn` id or no steps, or that are not flagged automated, are reported as not runnable rather than skipped silently. A duplicate scenario id across cases is also not runnable.
2. **After a run**: it finds or creates the Build for the deploy, creates a Test Run on the plan, adds every materialised case, and sets each execution's status from the Atlas verdict: `pass` = PASSED, `fail` = FAILED, `blocked` = BLOCKED, `flaky` = ERROR (never a pass), unevaluated = IDLE. Not-runnable cases are added as BLOCKED with the reason as a comment. Every non-pass gets a comment with the verdict, attempt count, failure class and reason.
3. **Platform Pulse** receives the summary with a link to the Kiwi run, so the TCMS is the record and Teams is the notification.

Writing a new automated case therefore means: author it in Kiwi with the four-section text for humans, paste the scenario JSON into the Automation script field, tick "is automated", add it to the `Atlas: regression` plan, and have it confirmed. It runs on the next deploy of BackOfficePrototype to development.

## Sources

- ISO/IEC/IEEE 29119-3 test case anatomy: [tessl.io test-case anatomy reference](https://tessl.io/registry/testland/test-case-anatomy-reference)
- Perforce, [Test case best practices](https://www.perforce.com/blog/alm/test-case-best-practices)
- Testiny, [How to write test cases](https://www.testiny.io/how-to-write-test-cases/)
- TestLodge, [How to write test cases](https://www.testlodge.com/resources/learning_center/test_cases/how_to_write_test_cases)
- TestMonitor, [Test case vs test suite vs test run](https://www.testmonitor.com/blog/test-case-test-suite-test-run-whats-the-difference)
- Automation Panda, [BDD 101: the Gherkin language](https://automationpanda.com/2017/01/26/bdd-101-the-gherkin-language/)
- Kiwi TCMS docs: [Introduction](https://kiwitcms.readthedocs.io/en/latest/guide/introduction.html), [Test cases](https://kiwitcms.readthedocs.io/en/latest/guide/testcase.html), [Appendix: writing test cases](https://kiwitcms.readthedocs.io/en/latest/guide/appendix.html)
- Kiwi four-section text template: [migration 0006](https://github.com/kiwitcms/Kiwi/blob/master/tcms/testcases/migrations/0006_merge_text_field_into_testcase_model.py)
- QA Skills, [Kiwi TCMS open-source test management guide](https://qaskills.sh/blog/kiwi-tcms-open-source-test-management-guide)