test-run-rules

• SKILL.mdcore · 14,312 bytes

# Test Run Rules

Apply these rules for the rest of the test run, whether it drives the running app or runs a test target. A scenario they leave **blocked** could not be set up or run; one they leave **inconclusive** ran, but its result cannot be read.

## Isolation

- Isolate shared process state so concurrent or sub-agent runs don't collide: bind dev servers and services to unique ports, scope tmux sessions (`tmux -L <name>`), give each browser session a unique name so cleanup can target only its own, and write screenshots and other scratch state to absolute paths under a uniquely named subdirectory of `$TMPDIR`. Derive each such identifier once and reuse that exact value in every later command, writing it as a literal or reading it back from a note in that subdirectory. A value recomputed per shell, such as `$$`, differs between the command that creates a resource and the command that releases it, so cleanup releases something it never created and reports success while the real resource leaks. A port picked as unique may already be held by a concurrent agent, so check it before binding and move to another when it is taken, leaving the incumbent running. When a unique port moves a service off its default address, find the settings elsewhere in the stack that name that default, such as allowed origins and sign-in callback URLs, and bring each in line through runtime overrides, leaving the working tree unchanged: add the new address beside the default in a list, and replace the default only where no process outside this run reads the setting.
- Reuse a running dev server only when this session started it, has not handed it to the user to try, and serves the current code: nothing it serves changed since it started, or, since the latest such change, its output reports having reloaded that code or a served response reflects the change. Stop each server this session started that fails only that last condition by the process group captured when it started, and confirm its port released and no process from that group still running before starting another; leave one whose group is unknown running. When no server qualifies for reuse, start one on a port this run selected and wait for it to be ready. Confirm it bound to that port before sending it traffic — a failed bind leaves another agent's service answering. Move to another port when the port is taken. When the start refuses to run because the project allows one dev server instance and one is already running, build and serve the project's production output with the configuration the dev server reads, on a port this run selected and confirmed bound the same way, and name that substitution in the result. A scenario that needs development-only behavior, or configuration the production output cannot be given, is then blocked: name what it needs. Report the error and stop when the server failed to start for any other reason.
- Treat state the app persists at a fixed path, such as a data file or a local database, as shared with every other live instance of the app. When another live instance is using it, give this run's instance a copy of its own under the run's scratch subdirectory: repoint the path there through runtime configuration, or, when the path cannot be repointed and the app resolves it relative to its own tree, run the instance from a copy of the working tree placed there, uncommitted changes included. Leave the working tree unchanged. When neither separates the state, every scenario that writes to it is blocked: name the path.
- Run each test runner this run starts in its own process group under a timeout enforced from outside the runner. Before reading an exit status as the runner's, confirm the runner ran: it printed its own output, or wrote a result artifact after this invocation started. An artifact left by an earlier invocation reads the same as a fresh one. When the runner did not run, fix the launch and rerun; the scenario is blocked when no fix starts it.
- Scope a screen capture or recording this run takes to the surface under test, addressed by its identifier, whenever the capture tool can. A capture of the whole display holds whatever else is open on it and can miss a surface that is covered or not in front.

## Unavailable Infrastructure