Choose the right level
A green fixture is evidence for its assertions. Registration, credential verification and a successful model answer are each narrower than a completed user task.
Deterministic commands
Prepare dependencies and an isolated test database using development setup. ExportTEST_DATABASE_URL before both gates:
verify refuses to proceed without the database URL, then runs check and test. Running npm test alone without that URL can omit database-backed coverage; do not report it as the full gate. Vitest creates a random schema, applies migrations, and drops the schema during normal teardown. Test files are serialized because the global scheduler can claim another file’s fixture Runs. Preserve explicit concurrency tests; do not turn file parallelism on casually.
verify:packages requires Python packaging dependencies, builds packages, installs them in temporary consumers, verifies imports, and runs the Harness API, SDK interoperability, local SDK and authorization-denial suites against the installed artifacts. It writes .local/evidence/sdk-packages.json, including artifact hashes and status. Temporary consumers must remain available for cloud runners that reference this record.
For a targeted iteration, use the actual test path:
release:check writes a private inventory report; its pattern scan is not comprehensive secret detection, and it does not publish anything.
Startup and backup have different prerequisites
Never rely on fallback database addresses when running destructive or stateful checks. Inspect the script before use and retain sensitive evidence privately. Startup/backup output identifies its evidence location. Review and remove disposable resources deliberately after investigation; do not remove a real installation’s identity or encryption key.
Real cloud checks are an explicit separate step
They require a running control plane, ready cloud/model connections, correct ingress, quota, pinned Harness definitions and an installed-package verification record where required. They may incur provider and model charges. Connection IDs belong in a private configuration; provider keys stay in control-plane credential storage.npm run verify:cloud -- /absolute/path/to/private-config.json pi uses its own configured Harness map and runs both clients; its configuration is not interchangeable with the compatibility runner’s schema. Read its implementation and the child journey inputs before preparing it.
npm run verify:serverless -- <provider> <harness-key> is a separate pause/resume test. It selects ready connections from the local installation and currently expects MiniMax-M3 model connections. Inspect the runner before use: its private report name is provider/Harness-specific, so preserve earlier reports before rerunning. It is not a generic benchmark or the complete Web journey.
Preserve the full journey
Each of Python and TypeScript must independently create a Session, upload and verify input, submit idempotently, observe/replay events, complete a real task, handle authorization and approval, retrieve artifacts, continue the conversation, cancel and close. Also continue each language’s Session from the other language. For the full Web task, use the fixed filter application. Do not edit its acceptance test to make the task pass. Confirm real file changes, test output, authenticated interactive preview, zero report writes before approval and exactly one afterward, final response/state agreement, retained exported artifacts after close, and revoked preview access. Keep separate branches for connection rotation/disable/delete, Tool refusal/expiration, worker and Runtime failure, cancellation races, uncertain create/delete outcomes, tenant isolation, and actual allowed/denied network probes. Exercise supported pause/resume or other capabilities explicitly. A conversation-only Harness cannot be credited with Tools or previews. A successful compatibility row does not silently cover these branches.Read test outcomes, not just exit codes
The current testing/tenant-boundaries.test.ts usesdescribe.skipIf(!process.env.TEST_DATABASE_URL). Without database configuration it can exit 0 while reporting one skipped test: no tenant-isolation assertion ran. Full gates therefore reject a missing TEST_DATABASE_URL before starting their tests.
The 2026-11-20 missing-database check observed:
This check deliberately removed the child process’s database variable, connected to no database and produced no cloud evidence. During normal acceptance, inspect executed/skipped counts, failures and actual assertions alongside exit status. Recording zero-execution output as passed silently removes required coverage.
Schema isolation has boundaries
Database setup creates a random hexagon_test_… schema in the configured database, migrates it and provides a connection with the matching search_path through test context. Environment setup supplies that connection to tests. Normal teardown deletes the corresponding schema. This does not automatically isolate every external service, extension or database-level action; use a dedicated disposable database. A forcibly terminated process may not run teardown. Identify the original process and specific schema ownership before removing leftovers. Prefix-based bulk deletion is not ownership verification. Retain private failure records so cleanup cannot accidentally affect a development workspace.Keep installed-artifact and source checks separate
verify:packages checks its database prerequisite before packaging. It then builds a TypeScript tarball and Python wheel, installs them in independent temporary directories, and uses HEXAGON_TEST_TS_MODULE and HEXAGON_TEST_PYTHON_PATH to run selected API/SDK suites against installed artifacts. The script names specific test files; it does not automatically cover every SDK method. A repository import or TypeScript compile cannot prove that a distributed package includes all required files. A successful package import cannot prove a real-model journey. Record source checks, installed-package checks and real-cloud checks separately, linking package SHA-256, version and source revision/snapshot. The script writes the fixed path .local/evidence/sdk-packages.json, so a rerun replaces the current record there. Preserve earlier records needed for investigation and keep temporary consumer directories required by subsequent cloud runners. Failure can occur before a record is written or during build, installation or tests. An absent file or an old verified record cannot substitute for the current command’s outcome.Evidence template
This is an unexecuted template, not a passing report. Replace each placeholder with an observation and retain failed attempts.cleanup.status independently from task success. Use not_checked, pending, failed or confirmed with the actual observations; these are report labels, not new API states. completion_basis names the Provider-specific condition: confirmed absence may suffice for one resource, while another requires a completed deletion receipt and associated-resource checks. null unresolved_operation_count means unknown, not zero. A failed model task can still have confirmed cleanup, and a successful reply can still leave cleanup pending. Keep each attempt’s execution and cleanup conclusions separate.
Public evidence in testing/evidence contains sanitized commands, timestamps, scope, assertions, failures and cleanup conclusions. Keep actual Session/Run/resource/operation IDs, provider receipts, endpoint identifiers and useful correlation data in access-controlled local evidence. Do not publish keys, raw prompts, request bodies or backups. Do not strip the private IDs needed to retry cleanup.
A lost response is an unknown outcome. Reconcile the recorded operation and exact resource before retrying; do not create another resource or replay a write to make the report green. Successful task cleanup and global account cleanup are separate claims. Report unresolved operations even if every later task succeeds. Historical evidence remains dated and does not automatically certify changed code.
Next: verified combinations, reconciliation, contribution review.