Skip to main content
Version: Latest (v2.0.0)

scripts/tests

Covers scripts/poll.py — the cron entrypoint — plus status.py, service.py, uninstall.py, the print_version.py / migrate_db.py / backup_db.py release CLIs, the release compose generator, the release tarball builder, the maintainer pre-flight checker (check_release.py), the release orchestrator (release.py), and — structurally, since neither runs here — the .github/workflows/ CI and release definitions.

python -m pytest scripts/tests

Must run separately from renderer/api/tests. Neither parent has an __init__.py, so pytest names both tests/ directories the same top-level package and the second fails to import.

FileGuards
test_poll_api_auth.pypoll.py's API token. One opener adds Authorization: Bearer to every request for RENDERER_URL and to no other URL — not one on a neighbouring port, not one that merely mentions the API in its query string. The token file is read per request, because the supervisor outlives a --rotate-tokens; and main() exits non-zero before doing anything when there is no token. The suite's conftest points TUBEMATIC_SECRETS_DIR at a temporary directory with a token in it, so no test reads or writes the host's own
test_poll_main.pymain() — what each of the supervisor's three lanes starts: the pause gate, the pipeline lane, the two independent research gates, and the script-profile lane, which runs beside the other two and still runs while paused. Also that the user preferences reach every prompt, and that an idle pass never reads the settings at all
test_poll_supervisor.pyThe looping supervisor's mechanics, several with real processes and threads because a mocked Popen cannot show a killpg reaching a grandchild. The pipeline lane never exceeds max_concurrent_agent_sessions, fills oldest first, and never dispatches a video already in flight; research runs beside a full lane and once per supervisor, and not at all when no strategy is active (the API leaves mining out while no Gemini key is stored, so a mining-only install reports none) — without a run recorded against the daily cap; a pause starts nothing new and lets running sessions finish. The verdict read from the database: any status change is progress whatever the exit code; no progress on exit 0 is a warning, on a non-zero exit an error; a stuck flag is an error naming Retry agent; a VOICING session that ran on into scene work is flagged. The scene-claim cooldown, and its absence outside the claim window. A session past its deadline is killed with its children, and a child that ignores SIGTERM is killed at the end of the grace even though its leader exited at once. A session stays registered — output files included until its usage is recorded — until it is scored, and a successor records a finished-but-unrecorded session under its original run id without re-reporting a run already recorded, and reports its outcome under the session's own key. Every outcome carries its session key, and one the API reports as a replay is logged, never scored again. An adopted session's end is noticed within seconds, and an overdue orphan's SIGTERM-ignoring child gets SIGKILL within the grace. The registry holds a running session and is empty after it; a live orphan is adopted, holds its video and is scored when it exits; an adopted research session keeps the research lane to itself; one past its deadline is killed; a reused pid is never mistaken for an orphan. A stale heartbeat is logged by the next tick and kills nothing. Every log line names its session. The cooldown and its statuses are checked against the MCP server's SCENE_CLAIM_TTL_SECS and _CLAIM_STATUSES as source text, since poll.py cannot import the server. The stub API turns itself unreachable after a few hundred passes, so a regression that loops forever fails instead of hanging the suite
test_invoke_agent.py(OpenCode: that its ladder retries without the variant while keeping the model, that dropping its model necessarily drops its effort, and that three runs remains the ceiling) invoke_agent() command construction — the exact argv for each agent binary, the model flag, the effort flag, and the fallback ladder. Pins the two flags' opposite failure shapes (an unrecognised model exits 0 and must be read from the output; an unknown --effort exits non-zero having run nothing), that effort is dropped before the model, that no invocation ever runs more than three subprocesses, and that the effort marker is never matched against the agent's own words — matching the session's text would let one that merely discusses the flag strip it from every later run. Also that the row's agent id, not the poller, picks the argv, that a Codex row DOES retry when the API refuses the effort its model cannot use, and recovers when the fallback model refuses it too, without ever exceeding three runs, and that the agent's stdin is closed rather than inherited (Codex reads piped stdin and appends it to the prompt, so an inherited one can hang an unattended tick). Usage reporting: every run posts its parsed usage to /usage/agent-session with the batch's videos and stages — including a retry, which is a second billed session — while a run that never started (a rejected model) posts nothing, a dry run posts nothing, and a failed POST never costs the tick but logs the full payload, so the units (the part that cannot be recovered later) survive in poll.log A 200 that recorded fewer events than were sent is treated as a failure, so the lost units still land in the log rather than a success line. OpenCode's session-usage read is retried and every failed attempt is logged with its reason — seen in a live run, the read failed right after a session that ended on a provider error and worked moments later, and a silent miss leaves only the stream fallback, which can lose the last step. Logging: a run's log entry is the adapter's summary and never the raw stream (a marker inside a tool result must not reach the log), stderr is logged by its tail, and a model rejection printed past the log cap still triggers the retry — the pin that trimming is for the log only
test_agent_adapters.pyscripts/agent_adapters.py — every argv shape pinned verbatim, because the Claude CLI treats an unrecognised subcommand as a prompt, so a malformed verification command silently becomes a billed model call. Also the two output matchers, including that a bad --effort value (exit 0, warning, work still done) is deliberately not a rejection: retrying it would be a second billed session for nothing. For Codex it additionally pins the -- separator before the prompt (codex exec has resume/fork/review subcommands, so a bare prompt of exactly one of those words would be parsed as the subcommand), --skip-git-repo-check (the agent workspace is not a git repository and Codex otherwise refuses to start), that "Logged in" is a prefix test so the logged-out "Not logged in" cannot match, that the model matcher does not catch the CLI's "metadata not found" warning — which is printed for models the API then accepts, so treating it as a rejection would drop the model flag on healthy runs every tick — that Codex's rejected-model matcher covers both auth modes (a ChatGPT account and an API key word it completely differently, and matching only one leaves the other with no fallback), that it still ignores the benign "metadata not found" warning even when that warning sits directly beside a real rejection in the same transcript, that an effort the chosen model cannot use is a rejection worth retrying while a zero exit never is, that internal catalog entries marked visibility: "hide" are never suggested; that every Codex argv carries approvals_reviewer="auto_review" — without it codex exec refuses every MCP tool call and still exits 0, so the agent does nothing while looking healthy — and that no argv ever contains a dangerously bypass flag; and that a malformed model catalog yields no suggestions rather than raising. For OpenCode it pins that the effort appears only as a #variant suffix on a model and never alone (the effort has nowhere else to live, which is also what makes the poller's third fallback round unreachable for this agent); that --auto is on every argv and no bypass flag ever is — an unmatched permission resolves to ask, and a non-interactive ask is auto-rejected and cancels the whole session, so without --auto a run that merely reads the poll log dies having done nothing; that logged_in cannot lean on the exit code, because opencode auth list exits 0 logged out and prints No authenticated integrations, with empty output also counting as logged out; that a missing variant is told apart from a missing model (Variant unavailable vs Model unavailable), which is what lets the retry keep the chosen model; that mcp_remove_argv returns None, because no opencode mcp remove command exists and an argv that does not exist would have an uninstall reporting success for a command it never ran; that the permission rules deny edit, write, shell and .env reads while allowing .env.example after its deny, since last match wins; and that Code Mode is turned off on the server entry while oauth is deliberately left unset. It also covers OpenCode's config dialect: that the document name is resolved between opencode.json and opencode.jsonc in the order the CLI itself picks (probed — with only a .jsonc present, mcp add --global appends to that one), that a model slug with a vendor segment (openrouter/qwen/qwen3.8-27b) is suggested — requiring exactly two /-separated segments dropped every OpenRouter model while opencode models listed them, and a dropped line is indistinguishable from a header line that was meant to be dropped — while a decorated or malformed line still is not; that the JSONC parser is string-aware so a // inside a URL or a , } inside a string value survives it, and that a trailing comma followed by a comment parses — all three forms of that raised, because doing both transformations in one pass leaves the comma's lookahead staring at a comment that has not been removed yet. The comment-preserving removal has its own group: that it drops exactly one member and keeps comments across seven document shapes (including a // inside a URL and braces inside a string), and that it refuses rather than guessing on five malformed ones — every result is verified against the parsed original, which is what makes text surgery on a human-authored file acceptable. A separate test pins that the permission rules assert what the pipeline needs, not only what it must not do: deny-only left webfetch, websearch and the tubematic_* MCP namespace to be decided by the operator's global config, and a global deny of any of them won — silently, since the pipeline would then do nothing while looking healthy. The uninstall sweep is likewise pinned to remove Tubematic's own config documents from the agent workspace. Two whole-registry tests come from bugs found in review: every adapter that names a config document must also say how to parse it, and every root-resolving call must accept respect_override=False — honouring an environment override on some of them and not others removed the default skills while leaving the default commands and MCP registration in place. A whole-registry test keeps every adapter's effort levels equal to the shared tuple, another keeps HAS_COMMANDS honest against what is actually on disk, a third keeps every agent's command set identical so a command added for one cannot silently not exist for the other, a fourth forbids any agent's commands from naming another agent's CLI, and a fifth forbids a command from telling an agent to run a tool its own install denies — OpenCode's shell: deny is absolute inside the agent workspace and cannot be approved interactively the way an unlisted Claude tool can. The JSON output modes each argv now carries are how usage is read, and the usage parsers are pinned against trimmed copies of real probed output: Claude yields one event per model with its own list-price costUSD, a rejected-model run yields nothing; Codex sums its turns and splits cached input out of input_tokens (OpenAI counts it inside), falling back to the model in its config.toml when none was passed; OpenCode reads the session API rather than its stream, whose last step_finish can be missing, and falls back to the stream marked estimated. The JSON modes move Codex's and OpenCode's own errors onto stdout, so their effort matchers read the stream's error events — and a test pins that the same phrase inside the agent's own message is still not a rejection. The log summaries (summarize_output) are pinned per agent: the final message is the last one, not the first; Codex counts a call once although it appears as both item.started and item.completed; OpenCode dedupes a call by its call id; failed calls are counted and the CLI's error events are surfaced. Bulky tool results, arguments and Codex reasoning never reach a summary. A long message is clipped with a count of what went, output in no known shape is clipped rather than dropped (it may be the only clue to a crash), and empty output summarises to nothing MCP authentication: no registration ever carries the token itself — Claude's add-json entry names a headersHelper, OpenCode's header is a {file:} reference to the token file, and Codex's mcp add gets neither (it has no header flag) — and each adapter declares whether its mcp add replaces an existing entry, since Claude's refuses one. The Codex http_headers_helper patch is a line edit inside [mcp_servers.tubematic] only: it keeps the operator's comments and other tables, replaces rather than duplicates, quotes a path with specials into valid TOML, and returns None when the table is absent.
test_bootstrap_args.pyThe argv bootstrap, and that every prompt names one video and points at the absolute poll.log. Cron strips the environment to HOME/LOGNAME/PATH/SHELL, so the workspace path and agent binary arrive on argv from the cron line
test_workable_videos.pyThe /videos/workable path, exercised through the real functions rather than a reimplementation. Every workable video gets its own session under its own agent, model and effort. A pipeline session is tagged with its one video and the status it started at (the stage its cost is shown under). An unreachable endpoint starts no session at all: there is no unscoped fallback, because a session told to work "every workable video" crosses agent boundaries
test_poll_prompt.pybuild_research_prompt — strategy blocks, per-run targets (the mining target only when mining is active, so a run without a Gemini key is never invited to mine), and the configured source lists, and that it deduplicates with list_project_topics (archived topics included) and skips an expired domain like a paused one. build_pipeline_prompt — one video and where its stage ends, for every workable status (a cross-copy check keeps the stage table in step with the API's WORKABLE_STATUSES, since a status with no stage would be skipped forever); the ban on looking for other work, and that it stays a ban on looking for work and never becomes a flat ban on get_pipeline_status, which the skills need for the pronunciation lexicon and the voice roster (a flat ban made agents skip those steps); that a revision is told not to stop at VOICE_READY, where the scene-work claim would refuse the next session; and that a pronunciation revision is told — in the prompt and in the operator skill's row — to end at VOICING without re-voicing, since a live session otherwise followed the skill straight through to a render. Also the USER PREFERENCES block on all three prompt builders: each names its own stage, scopes the text to that stage, and says it cannot override a checkpoint. Empty preferences must leave every prompt byte-identical, so an install that never sets them sends what it always did
test_voicing_authorization.pyThe wording that lets an unattended agent voice a video already in VOICING. An agent once refused generate_voice there, asking for approval of the spend and of sending the script to ElevenLabs — approval the human had given by moving the video out of SCRIPT_READY — and since VOICING is workable, every tick refused again: a wedge, not a delay. The cause was prose, so these pin prose, in two halves that are both needed. The operator skill's VOICING row and pronunciation re-voice row, the build_pipeline_prompt text and the generate_voice docstring each say the voicing is already authorized, naming both the spend and the disclosure (naming only one left the other to stop the agent), and the row says a refusal wedges the video. The SCRIPT_READY checkpoint stays, scoped to moving a video into VOICING rather than read as a ban on the tool. The generate_voice error row must not re-set the status, because that only rewrites updated_at, the clock the worker's stall alert measures
test_poll_json_logging.pyThe poller's structured poll.<date>.jsonl sink. Retention arrives over HTTP (this module has no database access, and a test pins that), poll.log is kept beside the JSONL file rather than replaced, and an unwritable shared log directory degrades to poll.log only. That degradation must be reported: setup_json_logging returns None quietly by design, and before this was checked a root-owned ~/.tubematic-data/logs dropped every poll entry from the dashboard with nothing anywhere saying so. The test asserts the poll.log warning names the directory and the chown fix, and that a writable directory produces no warning
test_poll_lock.pyWhere poll.py::LOCKFILE lives and that a run never deletes it. It sits beside the poller's logs under the agent home, not in /tmp, so every poller sharing a home directory shares one lock. A finished run leaves the file in place: unlinking a flock'd file lets two pollers each hold a lock on a different inode, so the file must outlive every run
test_update_price_snapshot.pyupdate_price_snapshot.py, the maintainer script that writes the built-in model prices a fresh install starts from. It must apply the same precedence as the runtime sync (OpenRouter, else LiteLLM), must refuse to write when a model Tubematic can select is in neither feed — a missing row means that model is unpriced on every new install — and the committed tubematic_core/prices_snapshot.json must cover every model in KNOWN_MODELS
test_status_spend.pystatus.py's spend summary, which now reads the usage ledger instead of three per-video cost columns nothing writes any more. It keeps the dashboard's honesty rules — unpriced calls are called out rather than counted as $0, ElevenLabs is shown in credits, the API-list-price caveat is printed — and a database that has not been migrated yet says so instead of crashing the whole status report
test_clear_db_wipe.pyclear_db.py's "Wipe all videos" also clears the spend ledger — including when there are no videos at all, which is an ordinary state (research that filed nothing, voice previews, videos deleted one by one). It used to stop at "no videos" before reaching the ledger, leaving the destructive full reset as the only way to clear recorded spend. A genuinely empty database still asks for nothing
test_status_format.pystatus.py's local-date formatting
test_mcp_url.pyconfig.py::_mcp_base, which turns MCP_URL into the server root. status.py probes {MCP_URL}/health, and that route is served beside /mcp, not under it, so an MCP_URL in the endpoint form would build /mcp/health, get a 404, and report a healthy MCP container as unreachable. That form is the one someone would actually paste: claude mcp add takes it and the docs quote it. A trailing /mcp is stripped as a whole path segment only, so /mcpx is left alone and a reverse-proxy prefix survives
test_service.pyservice.py — start goes through data directories and volume reconciliation before up, which is the whole reason to prefer it to a bare docker compose up -d; restart is stop then start. stop and restart refuse while a render is in flight and --force overrides, the same gate upgrade.py uses; restarting the dashboard alone is not refused, since it holds no work. The exception is pinned in both directions: when a Docker volume has to be re-pointed, the reconcile runs docker compose down on every container, so a plain start and a dashboard-only restart are gated too — and refused before anything stops, so a refusal never leaves the dashboard stopped and not restarted. status is the shared report; logs hands the terminal to docker compose logs. Bad commands, unknown services and unknown options are refused before the venv bootstrap runs Starting the stack creates any missing service token just before up.
test_uninstall_config.pyuninstall.py's config read must survive deletion of the database it reads from. Also its non-Claude agent sweep: the whole-project uninstaller only ever removed Claude's files, so a Codex install used to survive it — these pin that ~/.codex/skills/ and the global MCP registration are removed, that a skill nobody else installed is left alone, and that a machine which never had Codex prints nothing at all. They now drive that through the single adapter-keyed entry point rather than a function per agent, which is the thing that stops a newly supported agent shipping with an uninstaller that quietly leaves it behind. Four more came out of review, each pinning a way the sweep used to miss files that were really there: a project-scope install is swept from its own workspace rather than the user directories (otherwise it found nothing, printed no header, and left the whole install behind), every skill the installer deploys is looked for rather than a hardcoded list that had fallen one behind (simulated by dropping a shipped name from the list), a retired skill an older release deployed is removed too, the config-root override is ignored on every path and not just the skills, and a .jsonc config is found — and left untouched, with the edit reported, when it carries comments. A user-level Claude Code uninstall removes from ~/.claude/commands/ only the command names it lists, so a further test pins that list to cover every file in agent_files/claude/commands/: four commands had been added there and never to the list, and survived an uninstall. The same Claude path now sweeps every shipped and retired skill rather than its literal list, and the step numbers printed never exceed the total (agent sweeps all share step 6)
test_collect_diagnostics.pycollect_diagnostics.py, whose report is meant to be pasted into a public issue. One fixture plants every value the collector could be handed — each secret-typed setting, the three service tokens, the password hash, unstored OpenAI, ElevenLabs and Google keys, a Bearer and a Basic header, a session cookie, an email, a public IPv4 and IPv6, the home path, user and host name, a postforme account id, a voice id (and ones only a video or a domain's default still holds, after the voice left the roster), a handle, and a video's title, topic, domain name and id (an id is a slug of the domain's subject, so it is content too), a single sentence of its script, its review feedback, its publish description, one line of a human comment, the user's preferences and a pronunciation-lexicon entry — into all three log sources at once (structured logs, poll.log, container output), and the finished report is searched for each. Content stays out unless --include-content, except agent-authored comments: the worker files its errors as those, so the test asserts a worker error survives; ids become the same short hash wherever they appear, so lines about one account still match up; loopback addresses and timestamps survive. Content under 12 characters is replaced as a whole word only: a short comment once survived because short pieces were skipped outright, and plain substring matching would cut into longer words, so the test checks a short comment goes, a longer word starting with it and one ending with it stay, a placeholder that contains the same word is not rewritten, and anything under four characters is left. The residual scan is shown to fail closed by breaking the redactor on purpose: nothing is written and the surviving value is not echoed to the terminal either. A stored value with no recognisable shape is pinned separately, because only the literal check can see it. A secret straddling the 2,000-character line limit is pinned too: the cut once ran before redaction, leaving a prefix the stored value no longer matched, so lines are now cut only after redaction and the residual scan. --since is shown to bound poll.log both ways — a narrow window drops older lines (a continuation line takes its message's time) and a wide one reaches the dated rotations; --lines 0 once meant lines[-0:], the whole file, so non-positive windows are refused. It also refuses to write inside ~/.tubematic-data (an agent reading the report would then need the directory that holds every key), refuses an existing report.md — including a symlink planted while the report is being built, which only opening with O_EXCL catches, the up-front check being a courtesy — writes 0700/0600, shell-quotes the path in the printed gh command, stops when a token file exists but cannot be read, and opens the database read-only
test_troubleshooting_skill.pyThe troubleshooting skill is explicit-only on all three CLIs — disable-model-invocation and opencode/autoinvoke in the frontmatter, agents/openai.yaml for Codex — and no scheduled prompt names it. Losing any switch is silent: the skill still works when asked for, so nothing else would fail. It also pins that the skill never tells the agent to query the database, read a token or file the issue itself, and that it reads no raw log — not poll.log, not container output — only the redacted report: seen live, an agent told to read the poller's log tail read the whole unredacted file and sent stored keys to its model provider
test_release_clis.pyprint_version.py, migrate_db.py, backup_db.py — the CLI-level contract (exit codes, what lands on stdout vs stderr, and that --allow-no-backup prints a visible warning) over tubematic_core.db, which is covered separately in tubematic_core/tests
test_mcp_auth_header.pymcp_auth_header.py, the helper Claude Code and Codex run on every MCP connection, run for real as a subprocess. It prints one JSON object carrying Authorization: Bearer <mcp_token>; a missing token exits non-zero with nothing on stdout, so the CLI sends no header and the server's 401 is the visible failure; and it never reads the API token, which is a different credential
test_gen_release_compose.pygen_release_compose.py — drives generate() against the real docker-compose.yml, not a fixture, so a real topology change is what the test would catch: every build: section (and every service reusing the locally-built image under its tubematic-renderer-image tag) becomes a pinned ghcr.io/<owner>/tubematic-<role>:<version> image, every published port is forced onto a 127.0.0.1: binding, and everything else — volumes, environment, healthchecks, depends_on, shm_size, restart, container_name, the top-level name: — passes through byte-for-byte. Also pins check_version_is_tag_safe/invalid_version_chars: a plain or -prerelease version passes, a +build metadata version (legal SemVer, illegal Docker tag) is rejected with every offending character named, and generate() itself refuses an unusable version rather than emitting a broken compose file The top-level secrets: block and every service's secrets: list pass through unchanged, so a release install mounts the same token files a dev one does. The default --owner is the owner part of tubematic_core.version.GITHUB_REPO, so the repo is spelled in one place.
test_build_release_tarball.pybuild_release_tarball.py — builds the real dist/tubematic-v<version>.tar.gz from this actual checkout (not a fixture), and asserts: every allowlisted path is present, including upgrade.py, scripts/service.py and the onboard/stack.py they share (a release user cannot upgrade or restart without them), the LICENSE file (Apache-2.0 requires a copy in every redistribution, and the tarball is one), and the public docs — docs/README.md, docs/quickstart.md, docs/release-notes.md, and the docs/user-guide/ and docs/developer/ trees, each named explicitly and never a bare docs/ tree, which would sweep the gitignored docs/superpowers/ in behind it; every excluded pattern (renderer/, .env, HANDOFF.md, .onboard-state.json, docs/superpowers, both packages' own tests/, …) is explicitly absent; the bundled compose file is the generated release one, named plainly docker-compose.yml; tubematic_core/migrations/ survives; building the same commit twice is byte-identical; CHECKSUMS.txt covers every other file and no more; a missing allowlisted path raises rather than shipping a hole; no non-.py file it ships mentions a TMDEV-/VQA-/EXQA- ticket ID (the archive is a public download; .py files are exempt since internal # comments keep ticket IDs deliberately); no non-.py file it ships carries a relative markdown link to another .md file, resolved against the linking file's own directory inside the archive, whose target isn't also in the archive — catches both an explicit docs/ prefix (a root guide linking into docs/) and a relative path between pages (a user-guide page linking ../developer/…), since only part of the docs/ tree ships; both shapes shipped broken once, before this guard covered it; and — the test of the allowlist itself — the extracted tree can import onboard.actions/onboard.steps/onboard.config/tubematic_core.db in a fresh subprocess. A dedicated fixture repo also drives the forbidden-name guard's fail-loud path itself (a "mounted" directory planted inside onboard/, whose own ignore set excludes only tests/__pycache__, still gets copied and must be caught by the whole-tree sweep) — previously only its absence on a clean build was ever exercised scripts/mcp_auth_header.py and tubematic_core/auth.py ship in the tarball: every agent registration points at the helper, so a release install without it would leave every agent refused by MCP. scripts/collect_diagnostics.py ships too, since the troubleshooting skill asks a release user to run it. The compose-file check matches compose files by name, not every .yml: a skill may carry agent config such as agents/openai.yaml.
test_build_release_tarball.py (cont.)One case is about the developer's machine rather than the archive: pytest writes a .pytest_cache inside each suite directory, and .pytest_cache is also a forbidden name — so before the per-tree prune sets excluded it, the build failed with "excluded pattern found in staged tree" for anyone who had run the tests, which is everyone about to cut a release. Git hides the problem entirely (pytest drops a .gitignore containing * into each cache, so git status is clean and the directories are invisible until a tool walks the filesystem directly). The test plants one in every shipped tree and asserts the build succeeds and the archive contains none
test_kasm_workspaces.pyThe two Kasm Workspace images, kasm_workspace/ (published) and kasm_workspace_testing/ (QA), which CI never builds. In both, the dind /tmp patch's RUN body is executed with sh against a fixture custom_startup.sh: with the anchor line it inserts exactly one line before supervisord, and with the anchor changed it fails — a skipped patch is a blank-screen session nobody notices until they launch one. Every hook and build script is 100755 in the git index (COPY keeps the mode, and an inert hook looks exactly like Kasm never calling it), parses with bash -n, and the published ones pass shellcheck where it is installed. The published Dockerfile must not COPY the repository — its whole point is that a release never needs a rebuild — and every file it copies exists in its context. It also runs on UTC (ENV TZ=Etc/UTC, linked into /etc/localtime): the session is Tubematic's host, so its zone is every time Tubematic shows, and a re-copy from the testing Dockerfile would otherwise bring that image's fixed zone along. The registry entry (registry/Tubematic/workspace.json) has the schema 1.1 keys, an icon that exists, privileged, gpu_count: 0 (an attached GPU crash-loops XFCE), the answers mapped under /dockerstartup/ rather than /tmp, a version list newest first whose first entry is the default and the only (latest), every value matching the field's own validator_regex, and allow_saving: false on the version so a remembered pick never pins an old one. The form's keys, mapping path, methods and regex must equal the bootstrap's own constants
test_kasm_workspace_bootstrap.pykasm_workspace/image/tubematic_bootstrap.py, what the published workspace runs at launch. No test touches the network: downloads go through a fake urlopen and clones through a fake run. Every refusal downloads nothing — bad JSON, a missing key, a version that is not a plain X.Y.Z (a v prefix, a prerelease, shell metacharacters) or an unknown method, checked end to end through main() with the request log asserted empty. The relaunch cases: an empty profile installs, the same version is left alone, a different version is refused with the upgrade pointer (and, through main(), makes no request), and Tubematic data with no install marker — or an unreadable marker — is refused rather than installed over. A release is checksum-verified before extraction: a mismatch discards the download and extracts nothing, a CHECKSUMS.txt without the tarball's line is refused, an unpublished version (404) is refused, an archive with any path outside tubematic-v<version>/ is refused, and an existing target directory is never extracted over. Source clones the version's tag, never the default branch. A fresh install writes the marker and runs onboard.py in the install directory. GITHUB_REPO must equal tubematic_core.version.GITHUB_REPO, since the image cannot import it. Trap: the fakes are installed by patching the module's urllib.request.urlopen/subprocess.run, so the functions must look those up at call time — a default argument bound at definition time made the "downloads nothing" tests pass while never seeing the fake
test_kasm_registry_sync.pykasm_workspace/sync_registry.py, which release.yml runs to write the Tubematic entry into the workspace registry repo. The launch form has no free-text fallback, so the list it writes is exactly what can be installed. Pinned: versions sort numerically (2.0.10 after 2.1.0, before 2.0.9 — a string sort gets this wrong); tags and duplicates collapse; a prerelease or anything not X.Y.Z is never offered; an empty list is refused and writes nothing; the written entry passes the same checks test_kasm_workspaces.py applies to the in-repo one (newest first and the default, (latest) on the first only, the form keys the bootstrap reads); and nothing but the version list changes, so the in-repo entry stays the source of truth for every other key
test_check_release.pycheck_release.py — every pre-flight check exercised independently against a real git repository built in tmp_path (real git init and real commits, never a mocked git): a dirty tree, a version mismatch in both the plain and v-prefixed forms, a version that is not tag-safe (with every offending character named), a duplicate migration version, a modified vs. a merely-new-and-uncommitted migration file, a compose file that fails to generate, a missing allowlisted path, a forbidden name riding along inside an allowlisted directory tree, and the missing-workflow-files warning. The tag check is three real cases, not two: no such tag (pass), the tag already pointing at HEAD (pass — this is the release workflow running in CI after actions/checkout has already fetched the tag that triggered it, not a conflict), and the tag pointing at a different commit (fail, naming both SHAs — the actual danger, reusing a version for different code). Each tag case creates the tag for real with git tag, never a mocked lookup. A separate "all clear" fixture drives run_checks()/main() end to end and pins that the warning alone does not fail the exit code, while a real failure does
test_check_release.py (cont.)Two checks are warnings, not failures, and that is the behaviour under test. The release-line check reports when HEAD is not contained in release/X.Y.x — the mistake the line model makes easier, since the line and the development branch both look taggable — but warns rather than fails, because the first release of a minor version legitimately runs before the line exists and CI checks out a detached tag with no branches at all. A check that is permanently red in the two situations it is most often run is a check that gets ignored. The all-clear fixture creates the line so the pass path is exercised too, not only the warning. The development-branch marker develop is pinned on both sides of that line: the version-format check fails on it (every other check accepts it, so this one is all that stops a vdevelop tag), while the release-line check only warns and is skipped, because develop has no release line and one crashing check would take the whole report down with it
test_release.pyrelease.py — the thin orchestrator: a failing pre-flight check aborts before any build is attempted (run_checks is proven to be consulted, not skipped); the all-clear path builds the tarball, prints the release-notes skeleton and the git tag/push/branch commands — and, checked against a real git repository, never actually runs them (no new tag, no new branch); a tarball build failure aborts after checks pass; and the v-prefixed version form is accepted
test_release_notes.pyrelease_notes.py — extracting one version's section from the release-notes page: the matching section without its own heading, both the plain and v-prefixed argument forms, and the two things that end a section (the next version heading, a horizontal rule). Four failure cases are pinned because each is a different mistake: a version with no section at all, a section present but still an unfilled skeleton (the one that matters — a pasted skeleton looks written and says nothing), a prerelease whose base version is also absent (the error must name both), and an unreadable file. The prerelease fallback is pinned in both directions: 1.2.3-rc.1 with no section of its own publishes 1.2.3's notes, but a prerelease that does have its own section wins over the base. A final test runs the extractor against this repository's real page at the current VERSION, so a page that would fail its own pre-flight check fails here instead of at release time. On the development branch, where VERSION is develop, it checks the page's newest section instead: that is the release the branch is heading towards. release_line() is pinned here too, including that it refuses develop
test_workflows.pyThe .github/workflows/*.yml files — the only automated check they get, since GitHub Actions runs only on GitHub and actionlint is not installed here. Parses each workflow with PyYAML and asserts structure, not behavior: ci.yml's test matrix is derived from the suite directories actually on disk (not restated), the e2e job's if: never has a path to pull_request, every third-party action across both files is pinned to a released version tag or a full SHA (never a floating branch), and every needs: reference resolves to a real job. For release.yml specifically: it triggers on v*.*.* tags only; it reuses ci.yml via workflow_call rather than re-declaring its jobs; --owner is always github.repository_owner, never a script's hardcoded default; permissions: is exactly contents: write + packages: write; its one Release-creating step sets prerelease from a - in the version, because the action's default (false) would publish a release candidate as an ordinary release GitHub can list as Latest; the smoke-test job extracts the published tarball (never actions/checkout) into its own temp directory; it creates every secret file docker-compose.yml mounts before up — checked by running its token step against a scratch home and looking for each file the compose secrets: block names, because the smoke test skips onboarding and Compose refuses to start a service whose token file is missing (the first 2.0 tag run failed exactly there); its ordering — walked transitively through needs:, not just one hop — places the smoke test after the image push and before the Release job, since reversing that lets a broken artifact become a published, un-un-publishable Release; and its curl checks against the running stack carry --retry/--retry-connrefused/a time bound, so a later "simplification" can't quietly reintroduce a race against a container that reports healthy before its process has actually bound its port. The docs-site sync is pinned too: publish-docs runs after release and nothing depends on it, so a docs failure can never block a release; every job that syncs (it and docs-site.yml) declares the TubeMatic environment and passes TUBEMATIC_DOCS_TOKEN itself, because an environment secret reads as an empty string in a job that does not name its environment; both share one concurrency group; docs-site.yml fires only on a push to develop or a release/*.*.x line touching docs/, or by hand, and checks out its own commit before the target ref so an older tag without the action can still be re-synced; its input resolution and release-line labelling steps are run for real in bash (the labelling against a scratch git repo), pinning that a manual run accepts only develop, a release version or a release/X.Y.x line, that a line is checked out with its tags and labelled with the newest stable release it contains (never a prerelease on it, never a newer line's tag), that a push syncs the pushed branch at the pushed commit, and that a line with no release yet is never synced, since as the highest line it would become Latest on the site (a push to one passes and skips the sync; asking for one by hand fails); and the shared action refuses an empty token and builds the site before it pushes. The Kasm workspace registry sync is pinned the same way: publish-kasm-registry runs after release, gates nothing, skips a prerelease (a release candidate must never be offered on the launch form), names the TubeMatic environment and passes TUBEMATIC_KASM_REGISTRY_TOKEN, reads this repository only, and has its own concurrency group; its action refuses an empty token, lists only published non-draft, non-prerelease Releases (and fails on a gh error rather than reporting no releases), takes the entry from the newest release's tree — fetched without writing a local tag, checked by running that step against a clone holding the lightweight tag actions/checkout leaves, because fetching the annotated tag onto it is refused and failed the first 2.0 registry sync — and checks the workspace image anonymously with docker manifest inspect before it checks out, writes or pushes anything. No workflow builds that image, and none may call kasm_workspace/build.sh: it carries no Tubematic version, so a per-release rebuild is exactly what it exists to avoid, and it is pushed by hand

What these encode​

poll.py has no database access, by design. Everything it knows arrives over HTTP from the Flask API, or on argv from the cron line. Tests that need pipeline state stub the HTTP layer, never a DB.

Research runs are attempt-counted, not completion-counted. poll.py records the run before invoking the agent, and _record_research_run fails closed — if the API is unreachable, research is skipped this tick rather than invoking an uncapped agent session with no record. Completion counting fails open exactly when the cap matters: a crashed run leaves the counter at zero forever and re-fires every tick.

Two independent research gates. research_pending_cap bounds queue breadth; max_research_runs_per_day bounds attempt frequency. A day where research keeps coming up dry cannot burn an unbounded number of agent sessions. Both are covered separately because they fail for different reasons.

One session per video, one stage per session. The poller used to batch every video sharing the oldest one's (agent, model, effort) into one session a tick. It now runs a session per video, up to max_concurrent_agent_sessions at once, each told to stop where its stage ends. That makes cost attribution exact and removes the batch key: no video can ride along at another's model or effort. test_workable_videos.py covers dispatch through the real function — an earlier version reimplemented the rule in the test and therefore proved nothing.

Success is read from the database, never the exit code. The agent CLIs exit 0 on failures that matter (an unknown model; every MCP call refused), so the supervisor asks the API whether the video's status moved. Three sessions in a row that leave it where they found it flag the video stuck. That replaces the cron tick as the throttle on a video the agent cannot advance: the loop would otherwise re-run it every few minutes.

A dry run has zero side effects. --dry-run logs the intent to record a research run without making the request, matching invoke_agent's own dry-run behaviour.

--allow-no-backup must never be silent. It relaxes a safety guarantee (migrate_db.py skips the pre-migration backup), so its use prints a warning banner — the same rule onboarding already follows for --disable-verification. test_allow_no_backup_prints_a_visible_warning pins that the banner actually appears, not just that the flag works.

The release compose generator's transformation list is closed by design. It exists so the release compose cannot become a second source of truth for the topology — every field it does not explicitly rewrite (volumes, environment, healthchecks, depends_on, shm_size, restart, container_name, the top-level name:) is asserted to survive byte-for-byte from the development file. Two services (tubematic-mcp, tubematic-renderer) carry no build: key at all — they reuse the image tubematic-api builds — so they are repointed at the same pinned image by matching on that shared local tag, not by name, so a future service adopting the same pattern is covered for free.

The release tarball is built from an explicit allowlist, never a copy of the repo. The checkout also holds source trees (renderer/, dashboard/, mcp/, vqa/), local state (.onboard-state.json, mounted/), and .env/.env.example — none of which belong on a public download page. The allowlist test proves presence; a separate test proves absence of every excluded pattern explicitly, because the failure mode of assuming the allowlist implies exclusion is a future allowlist entry (a bare docs/, say) sweeping something forbidden in behind it. The forbidden-name guard itself (_assert_no_forbidden_names) has its own fail-loud path exercised directly, against a minimal fixture repo with a name planted where the per-tree ignore set does not reach it — proving the guard actually fires, not just that a clean build has nothing for it to catch.

Reproducibility is what makes the shipped CHECKSUMS.txt mean anything. Every archive entry gets a fixed mtime/uid/gid/uname/gname and entries are written in sorted-path order, so the same commit produces byte-identical output on every build. The gzip wrapper itself is also normalized (empty filename, fixed mtime) — gzip otherwise embeds the output path and the current time in its header.

A release version must double as a Docker image tag, and that is one rule, defined once. gen_release_compose.py interpolates the version straight into image: ...:<version> with no escaping, so a legal SemVer version carrying a +build metadata suffix (legal SemVer, illegal in a tag — + is not a tag character) would otherwise reach docker compose pull/up as a broken reference on every service, discovered by whoever runs the install rather than whoever cut the release. check_version_is_tag_safe is the single definition; generate() calls it itself (so a maintainer invoking gen_release_compose.py/build_release_tarball.py directly, not only through check_release.py, still gets a hard failure instead of a broken file) and check_release.py's own check_version_format check imports it rather than restating the rule.

check_release.py reports every failure in one pass, and treats "new" and "modified" migration files differently. A maintainer fixing up a release wants the whole list, not five separate runs, so no check short-circuits the rest. The migration-immutability check specifically compares the working tree against HEAD for tubematic_core/migrations/: a file git already has committed history for that has since changed (git diff status M) is a violation — a shipped migration must never be edited, since editing one silently diverges every database that already applied it from every one that has not — but a file with no committed history yet (untracked, or staged but never committed) is ordinary in-progress work on a migration that has not shipped, and is not flagged. The missing-workflow-files check is a WARNING, not a failure — the CI/release workflows now exist in this repository and the check passes silently, but it warns rather than fails for a checkout that lacks them (an older fork, or .github/ stripped some other way), since a maintainer's release should not be permanently blocked by a file this script does not itself need.

The tarball allowlist check only ever proved satisfiability, not exclusion. Every allowlisted path existing says nothing about whether a forbidden one also rides along inside one of those directory trees — that enforcement lives entirely in build_release_tarball.py's own ignore filters and its whole-tree sweep. check_tarball_exclusions closes that gap by running the real builder against a scratch directory as part of pre-flight, rather than restating _FORBIDDEN_NAMES or any per-tree exclusion set a second time.

release.py prints the tag/push/branch commands and must never run them — publishing is a later phase's CI/CD, and a script that tags and pushes on its own can ship a release nobody approved. test_release.py checks this against a real git repository rather than mocking subprocess: after a full run, no new tag and no new branch exist. It also pins that a failing pre-flight check stops the script before build_release_tarball.build() is ever called, and that a build failure aborts after checks pass rather than being silently swallowed.

test_uninstall_config.py exists because of an ordering bug. The install config used to come from .env, which uninstall removes partway through — after which every later step read empty values. The test pins that the config survives removal of its own source.

Known gap​

invoke_agent treats a non-zero exit code as the failure signal. The agent CLI exits 0 when given an unrecognised --model, printing unrecognized_model and doing nothing, so a video carrying a bad model produces a tick that logs success and advances nothing. _usable_model's docstring asserts the opposite. No test covers this because the code's own assumption is wrong — see the tracking ticket before adding one.