Regression scenarios

A regression scenario is one standing check: a named thing the platform can run on its own, again and again, after the work that needed it is finished. Most of them are registered for you when a branch merges, and nightly checks is the page that explains when. This one is the lookup table for what a scenario holds.

Scenarios live on the Regression Testing page, on its Scenarios tab.

The record

Field	What it holds
`scenarioId`	The identifier, unique in the workspace. A scenario registered by a merge is `task:` or `huddle:` followed by the id it came from, which is what makes a second merge update it instead of adding a copy
`name`	What the list shows. A registered per-task check is named `<task key> acceptance`, so `SUM-15 acceptance`
`description`	Free text, for a scenario somebody wrote by hand
`kind`	`nightly` or `smoke`. Only `nightly` scenarios run in the nightly sweep, and only a nightly result raises a fix task
`enabled`	Switching it off leaves the record in place and takes it out of the sweep
`config`	Everything about how the scenario runs, as JSON. The only part the editor lets you change
`sourceTaskId`	The task this scenario replays. Written by the platform on the merge, and preserved when you edit the configuration
`huddleId`	The huddle whose merge registered it, when there was one
`createdAt`, `updatedAt`	Timestamps

sourceTaskId and huddleId are lineage rather than settings: they are what links a night’s failure back to the work that shipped it, and what a fix task is filed under. The editor keeps them out of the JSON you edit so a hand save cannot clear them.

The three execution modes

config.exec picks the mode. A configuration that does not name one is command, so a scenario written before the other two modes existed keeps the behavior it had.

`exec`	What it runs	Where it runs	Where it comes from
`command`	One command, with assertions applied to the output it printed	The control plane, with no copy of any repository	Written by hand
`run`	A full agent run against a repository, with optional judges reading the result	A runner	Written by hand, and registered by a merge for a huddle whose final check needs a live agent
`checkout`	A fresh copy of a branch, the setup commands, then the acceptance checks and negative cases the task shipped with. No agent, no commit, no push	A runner	Registered by a merge, one per merged task

Every mode has to be claimed by the nightly sweep and by the dispatcher before it can run at all, so a mode cannot be added and then quietly go nowhere.

A checkout configuration

{
  "exec": "checkout",
  "checkout": {
    "repoRef": "https://github.com/alpenglow/summitline-app.git",
    "branch": "main"
  },
  "prepare": [["pnpm", "install"]],
  "checks": [
    {
      "criterionId": "AC1",
      "argv": ["pnpm", "test", "elevation"],
      "expect": { "exit": 0 }
    },
    {
      "id": "NC1",
      "gate": "scripts/check-elevation.mjs",
      "plant": [["node", "scripts/plant-flat-profile.mjs"]],
      "expect": { "exit": "nonzero" }
    }
  ],
  "acceptanceCriteria": ["The elevation profile renders at full height on Safari"],
  "toolchain": {
    "language": "node",
    "packageManager": "pnpm",
    "prepare": [["pnpm", "install"]],
    "verify": [["pnpm", "test"], ["pnpm", "lint"]],
    "testFileGlobs": ["**/*.test.ts"]
  }
}

Key	What it is
`checkout.repoRef`	The repository to clone
`checkout.branch`	The branch to clone, which is the branch the work merged into. A checkout scenario pins a branch rather than following the default one, so it cannot drift when the default moves
`prepare`	The setup commands, one argv array each. These are what runs before any check
`checks`	The acceptance checks and the negative cases, in one array. An entry with `criterionId` is an acceptance check and has to exit 0; an entry with `id` is a negative case, which plants its violation and then requires the gate to reject it
`acceptanceCriteria`	The criteria the checks are paired with, copied from the task. `AC1` means the first of them, so a criterion and its check cannot drift apart
`toolchain`	How the repository is built and tested, copied from the task. Its `verify` commands are not run by a nightly check: they are your repository’s own gates and they belong to the run and to your CI

Every value in a registered scenario is copied from the merged task. Nothing here is written by an agent at registration time, and nothing is inferred.

What a checkout scenario has to carry

Saving a checkout scenario is refused, with the missing piece named, when any of the following is true. The rule is applied when you save rather than at two in the morning, so a scenario that would fail to start is never stored.

checkout.repoRef or checkout.branch is missing.
checks is empty.
acceptanceCriteria is empty or absent.
The acceptance checks do not cover the criteria: there has to be exactly one AC check per criterion, numbered from the first.
toolchain.verify is empty.

A command scenario may not name a repository

A command scenario runs on the control plane, and it has no copy of any repository to run in. So a command configuration that names one is refused when you save it, rather than accepted and then ignored.

The rule is on the field name and not on the text: repoRef, checkout and run are refused anywhere in the configuration, at any nesting depth, whatever their value. A command whose assertion text happens to quote a git URL is fine, because what the rule is about is a repository named as a setting and then silently dropped.

Next steps

Keep merged work checked nightly: what a merge registers, and what a failing night raises.
Read a run: the same acceptance checks and negative cases, on the way in.
Task states: where a fix task sits once one is raised.