Keep merged work checked nightly

A merge is usually the last time anybody reads the checks that proved a change. The code around it keeps moving, and the criteria it was accepted against are never run again. Nightly checks close that gap. When a branch merges, the command that proved each of its acceptance criteria is registered as a standing check, and it runs again every night against the branch the work landed on.

What a merge registers

Passing a ship verification merges the branch, and that merge registers two kinds of standing check.

• One per merged task, named after the task, so SUM-15 becomes SUM-15 acceptance. It carries the repository, the branch the work landed on, the setup commands the task recorded, the command paired with each of its acceptance criteria, and each of its negative cases. Every one of those values is copied from the task, so a nightly check can only ever run something a person already approved.
• One per huddle, where the huddle’s own final check needed a live agent to answer it rather than a command. That one asks an agent the same question the final check asked, against the same branch.

Two things follow from that:

• Nothing is registered twice. Each check is identified by the task or the huddle it came from, so merging the same work again updates the entry that is already there rather than adding a second one.
• A task with no paired checks registers nothing. Registration needs acceptance criteria, a command for each of them, and the verify commands for the repository. A task that merged without all three is skipped, because there is nothing honest to run nightly. See what a software draft has to carry.

What a nightly run does

Every enabled check runs once a night, in one sweep.

A per-task check gets a fresh copy of the branch at its current head on a runner, and runs three things in order.

Step	What runs
Setup	The setup commands the task recorded, one row each
Acceptance	The command paired with each criterion, one row per criterion, shown as AC1, AC2, and so on
Negative cases	Each case, with its violation planted, one row each. The gate has to reject it

No agent takes part. Nothing is written, nothing is committed, and nothing is pushed: the copy of your code is read and thrown away. The check passes when every command exited the way its criterion said it would, and fails otherwise, naming the criteria that failed.

Your repository’s own test, lint, and build commands are not re-run here, and neither are the checks that measure an agent’s work, because no agent worked here. Those belong to the run that produced the change and to your CI on the pull request.

What a failing night produces

Night	What happens
First	One fix task, filed under the task the check came from, naming the run and the checks that failed
Second	A note on that same fix task, carrying the new run and its failures. No second task
Third	The fix task is assigned to the regression owner, and its priority is raised to high
After that	Notes keep arriving. The assignee and the priority are left alone

The third night is counted from the nights themselves rather than from a tally somebody could reset: the three most recent nightly runs that included this check all have to have failed it. A check registered yesterday cannot escalate tonight, because only one night has run it, and a night the check was switched off is simply not one of the three.

Fixing the work stops the escalation: once the fix task is closed and the check passes again, the next failure starts over at the first night.

Only the nightly sweep raises tasks

Running a check by hand from the Scenarios tab produces a result to read and nothing else. It raises no task, appends no note, and never escalates anything, so trying a check is safe at any hour.

The regression owner

The regression owner is who a failing check escalates to on the third night. It is one setting for the workspace, on the header of the Regression Testing page, and it resolves in this order:

1. The person named in that setting.
2. Otherwise the owner of the project the failing check’s task belongs to.
3. Otherwise nobody. The fix task stays unassigned, and the platform records that it had no one to hand it to.

Set the field, because an unassigned task sits in the queue as long as an assigned one and only one of them gets read. This is a separate setting from the delivery owner who signs off the merge, and one person can hold both.

Where to read the results

The Regression Testing page carries the checks on its Scenarios tab and the nights on its Runs tab. Opening a night lists every check it ran, and each row links to the task the check came from and, when the check failed, to the fix task it raised. A fix task carries the check that raised it, so either end leads to the other.

Next steps

• Read a run: the checks a change passes before it merges.
• Regression scenarios: the record, the three modes, and what each one may carry.
• Ship code from a huddle: the merge that registers all of this.