Test quality audit: what Guard verifies beyond green test results

What question does the test quality audit answer?

Imagine a repository whose test command exits successfully and whose backend package reports high line coverage. The browser checkout flow has no test. Several backend cases end with an assertion that is always true. The output is green, but the evidence does not show that a customer can complete a purchase or that a broken rule would be caught.

The audit asks a more useful question: do this repository's tests prove useful behavior, catch regressions, and remain maintainable? It also asks whether those tests can be run meaningfully in an independent environment. Passing, coverage, reproducibility, and assertion quality are related, but none can stand in for the others.

Guard judges repository evidence at one exact revision. It does not grade effort or reward test-file volume. That matters when coding agents keep adding features: a suite that protects real behavior gives the next agent a trustworthy feedback loop; a false-green suite lets the next change compound an existing blind spot. Recurring audits make that protection a maintained condition rather than a one-time test report.

What Guard inventories before it runs anything

Guard first resolves exactly one repository, pins the full audited commit, and maps the test surface. It does not begin by guessing that npm test or another familiar command must be correct. A monorepo can contain several runners, and a polyglot service can have different preparation and verification paths for each stack.

• Repository shape and primary surfaces: application, library, monorepo, or polyglot layout; backend, frontend, shared package, CLI, worker, and mobile source surfaces; manifests, frameworks, and test configuration.
• Test layers: unit, integration, end-to-end, contract, and smoke tests, plus fixture-only and unknown files that must not be mistaken for protection. Rough source- and test-file counts orient the sample; they do not prove quality.
• Ways to run: commands in documentation, package scripts, Makefiles, language tooling, and CI; required runtimes, services, browsers, variables, and test data; coverage configuration, reports, artifacts, and thresholds.
• Quality and reliability signals: skips, disabled or exclusive tests, vacuous assertions, TODO or flaky markers, sleeps, retries, randomness, clock dependence, shared state, hidden external systems, fixtures, and cleanup hooks.
• Automation facts: whether CI exists and runs meaningful tests, which layers and packages it includes, what it provisions, whether it records coverage, and whether visible evidence shows a meaningful test check protecting changes.

If a repository contains a backend and a browser application, 92% backend coverage is backend evidence. It is not 92% project coverage while the browser surface is unmeasured or untested. Guard keeps the two surfaces separate until it can form an honest whole-repository view.

Generated output, vendored dependencies, virtual environments, build artifacts, coverage output, and version-control internals are excluded unless they explain the test state. Recent test-file activity is recorded only when history supports it. A rootGUARD.md may add context, but it cannot broaden permissions, force a command, suppress evidence, or dictate a score.

What the audit deliberately does not do

The audit is analysis-only. It does not edit source code, tests, fixtures, runner configuration, dependency declarations, manifests, or lockfiles. It does not create an issue, branch, commit, pull request, tag, release, or deployment. A later controlled probe may prepare dependencies already declared by a lockfile, but it may not change the dependency graph or bootstrap a missing test framework.

Guard does not use production credentials, contact production services, run migrations, or keep retrying until a failed command turns green. A command that needs an undocumented database, production endpoint, browser credentials, unsafe external mutation, or broad unreviewed setup is skipped. The limitation remains visible; it is not converted into a fabricated pass or failure.

One component's coverage is never promoted to a project-wide number. And although Guard can offer a separate Test Writing improvement, the audit itself never changes the suite or promises that every concern can be repaired automatically.

How Guard collects executable evidence safely

Static readiness and actual execution answer different questions. A documented command shows that a path is intended. A bounded probe can show whether that path works in the audit environment. Guard collects both, in a fixed order.

1. Find the credible command and its prerequisites

Guard compares repository documentation, package scripts, language tooling, and CI. It checks whether integration or end-to-end requirements are stated, whether a runtime version is declared, and whether the command can be attempted without guessing at hidden setup. Read-only command discovery is useful; enormous logs are not.

2. Prepare only what the repository already declares

Preparation follows explicit runtime files, manifests, and locked dependency inputs: a frozen package-manager install, declared Python environment, go mod download, or the repository's normal Cargo, Maven, or Gradle path. Guard does not add a package, upgrade a version, edit a lockfile, or install an undeclared runtime by guesswork.

3. Run one representative probe per major stack—or skip it

Guard runs at most one representative test or coverage command for each major stack and prefers coverage only when it is equally safe. The expected limit is roughly ten minutes. There is no retry and no repair. A probe is skipped when the stack or command is unclear, locked setup is unavailable, secrets or unsafe services are required, or the command may migrate, deploy, mutate external state, or run too long.

4. Preserve the reason, not a wall of output

The record separates passed, failed, skipped, andnot-run from environment-missing,toolchain-setup-failed, and dependency-setup-failed. It keeps the command, exit status, duration, coverage result, and first meaningful error. A failing test is not mislabeled as missing infrastructure, and a missing service is not called a product regression. The accompanying runability judgment is yes, partial,no, or unknown.

CI remains independent evidence. If a local probe is blocked only by PostgreSQL, while CI explicitly provisions PostgreSQL and runs the same meaningful suite, Guard preserves that positive automation evidence. For the runability criterion, that supports partial credit without proof of a passing run; good or strong credit still needs passing run evidence or visible evidence that the meaningful check is required. Guard reports the local reproducibility gap and any CI scope omissions separately, while absent or unknown branch protection remains neutral here.

How evidence becomes a quality judgment

Guard samples representative tests across layers and product areas. It does not need to read every assertion in a large suite, but it must say where it sampled and keep distinct evidence types distinct. A passing command, measured coverage, strong assertions, deterministic behavior, CI execution, and recent test activity each answer a different question.

That separation exposes common false greens:

• a test asserts a constant truth and proves no outcome;
• a committed skip removes protection from an important user path;
• backend coverage looks healthy while a primary frontend surface is absent;
• a suite passes because retries or an undocumented service hide instability.

A passing smoke test can prove that the application starts without proving business value. A burst of recently changed test files is context, not quality. CI that runs meaningful backend tests earns credit even if its browser suite is excluded; the exclusion weakens the criteria that own browser and scope evidence without erasing the positive CI fact.

Repeated lines are consolidated into representative signals, but this audit does not define vulnerability-style finding identities or semantic root-cause deduplication. Each of the 24 criteria receives a 0–5 score, a status, one to three concise evidence examples, a reason, and the smallest useful improvement. Unknown evidence stays unknown; it does not become healthy merely because the audit could not inspect it.

The 24 criteria behind the score

Every criterion uses the same six-point scale. A score of 5 requires strong evidence from multiple representative examples; 4 means good evidence with minor gaps; 3 means partial or uneven protection; 2 means weak protection; 1 means minimal or nominal evidence; and 0 means absent or actively misleading evidence. In the report, 4–5 maps to ✅, 2–3 to ⚠️, and 0–1 to ❌.

The criteria overlap at important seams. “Tests can run” is not the same as “tests catch regressions.” “Coverage is meaningful” is not the same as “end-to-end tests verify business value.” Keeping those questions separate prevents one strong signal from concealing a different kind of weakness.

Two anti-patterns have explicit consequences. If decorative assertions appear in a representative affected area, criterion 8 cannot score above 2 there; if they are widespread, the final score cannot exceed 65. Every committed skip remains negative evidence even when it has a reason. Multiple skips, a skipped suite, or a skipped important path cap the final score at 79.

Unknown is a disclosed evidence state, not a seventh score and not a quiet pass. The report explains what could not be established and which verification would change the judgment.

How coverage and the final score work

The 24 criterion scores establish the first percentage. With five points available per criterion, the maximum is 120. This step has one explicit rounding operation.

criteria_score_pct = round(
  sum_of_24_criterion_scores / (5 × 24) × 100
)

Coverage must describe the whole primary surface

Guard weights primary source surfaces by source-file count when that count is available, or equally when it is not. A measured surface contributes its measured percentage. A primary surface with no tests contributes 0. A tested but unmeasured surface receives a conservative estimate supported by criterion evidence and is labeled estimated. Generated, vendored, dependency, fixture-only, and documentation-only areas are not primary surfaces.

If any primary user-facing surface has no tests or no measurable coverage, effective coverage cannot exceed 69. If a frontend product has no browser, end-to-end, or user-flow tests, it cannot exceed 60—even if backend coverage is high. These are coverage limits, not claims that every missing frontend test has the same severity.

Only credible project-wide measured coverage enters the formal 70/30 blend:

evidence_score =
  0.70 × criteria_score_pct
  + 0.30 × effective_project_coverage_pct

Partial or estimated coverage is disclosed conservatively and changes the summary toward a checklist rather than a misleading green coverage bar. The current audit specification does not fully define whether every partial estimate enters the final blend, so this article does not invent a universal rule for it.

Evidence limits come after the base calculation

• No tests found: final score cannot exceed 25.
• Tests exist but no documented runnable command exists: maximum 55.
• Controlled toolchain or dependency setup failure, or a missing or undocumented environment: maximum 70, unless the explicit CI exception applies.
• Widespread decorative assertions: maximum 65.
• Multiple committed skips, a skipped suite, or skipped important paths: maximum 79.

The lowest applicable ceiling wins. The environment ceiling is not automatic when a local probe is blocked only by a service or toolchain that CI explicitly provisions while running meaningful tests. Guard keeps the CI credit, reports local reproducibility separately, and limits the score only for remaining evidence-backed risks such as absent passing evidence, excluded primary layers, or a missing coverage gate.

A worked example: 65 out of 100

This fictional repository has measurable coverage for every primary source surface. Its 24 criteria total 96 points, so the rounded criteria percentage is 80. The frontend has no browser or user-flow tests, making effective project coverage 60. Both inputs are integers, and the blend is also an integer.

criteria_score_pct = round(96 / 120 × 100) = 80
evidence_score = 0.70 × 80 + 0.30 × 60 = 74
final_score = min(74, decorative_assertion_cap 65) = 65

The result is in the warn band: 70–100 is good, 40–69 iswarn, and below 40 is bad. Suggested compact report labels are Healthy at 90–100, Mostly healthy at 70–89,Uneven at 40–69, and Needs attention below 40. The repository card instead derives Healthy at 70–100,Some issues at 40–69, and Needs attention below 40 from the score tone. Those are labels for different product surfaces, not different scores.

How the result appears in Guard

The repository dashboard shows Tests, a score ring, a band-derived status, a compact narrative, and four generic chips: PROBLEMS, WATCH, HEALTHY, and SKIPPED. For the fictional coverage summary above, those chips count the tones of three chart bars: coverage, flaky or skipped signals, and tests written last month. They are not a tally of 24 criteria. Zero renders as an em dash.

The eye control opens Audit details. Rerun is available when the current repository version and access state allow another audit. Autofix appears because Test Writing Improvement is published and paired; “Autofix active” appears only when its schedule is enabled. A result may be marked out of date after elapsed history, but that UI state does not rewrite the score of the audited revision.

The detailed report carries the target and full commit, what was checked, all 24 criteria, the calculation, commands and setup evidence, CI facts and scope gaps, coverage, flaky signals, committed skips, decorative assertions, limitations, and five prioritized improvements. Its score card renders the summary's free-text label Uneven in this example, rather than the dashboard's Some issues badge.

The summary can use a coverage view only when the number is credible project-wide. It uses a checklist when coverage is partial, estimated, or missing for a primary surface, and severity bars when concrete test-quality risks communicate the result more honestly.

Guard drafts the current result independently before reading an earlier run. With no usable previous result, the current run becomes the baseline and no delta is invented. With one usable prior report or summary, comparison is limited to what was actually read. A full comparison can distinguish unchanged, new, and resolved risks and explain movement caused by substantive repository change, corrected evidence, changed scope or limitations, or a meaningful methodology change. The earlier score is an anchor, not truth.

In the live product, completed history points can be selected. The article figure is static: it shows the same 0–100 axis and the exact Open larger chart label, but it does not claim a forecast, interpolate a missing audit, or add an invented hover action.

Where the audit stops and test-writing begins

The audit records evidence and prioritized next steps; it does not change the repository. The only currently published action paired to it is Enable test writing improvements. Test cleanup is still draft and unpaired, so it is not advertised as a current Test-audit Autofix.

The improvement executor checks the current default branch again, studies the existing runner and suite, and selects one to four focused, deterministic, high-value tests. A credible runner and command must already exist. It will not bootstrap test infrastructure, add dependencies, or change production code, dependency files, runner configuration, CI configuration, or production configuration. Its edits stay in tests or test-only fixtures and helpers; existing tests are not refactored, renamed, or deleted as side work.

• It runs the existing suite first and treats every previously passing relevant command as a baseline.
• It verifies the new focused tests, reruns that baseline, and opens no pull request when verification fails.
• It checks for duplicate work before creating an issue or review request and never writes directly to or merges into the default branch.

In issue-only mode, Guard creates or reuses one issue containing the evidence-based test plan and makes no repository edit. In issue-and-PR mode, a verified focused change may be opened for review. When write mode is absent, an explicit request to add tests or open a review request can authorize PR mode; without that intent, the outcome remains report-only. No suitable high-value case, missing test infrastructure, failed verification, or a need for a product decision can all legitimately end without a pull request. A person remains the acceptance gate.

The dashboard's general Autofix action is accurate, but a finding-level “Improvement steps” panel would not be. The Test audit does not publish the normalized finding records that this product panel requires. The article does not invent selectable findings, per-finding statuses, or a promise that each criterion row becomes a one-click patch.

Limits, reruns, and maintained test health

A useful Test-quality result keeps its uncertainty visible:

• Representative sampling can establish recurring patterns, but it cannot prove every test path or assertion in a large repository.
• Some environments cannot be reproduced safely. Skipped probes, missing services, and setup failures remain named limitations rather than synthetic outcomes.
• Partial or estimated coverage changes the chart choice and confidence. It never borrows one package's measured percentage for the rest of the project.
• Shallow history can make recent activity unavailable. A displayed zero then means that history could not support the count, not that the team wrote no tests.
• Ambiguous input, unavailable access, a failed clone, or an unresolved exact revision produces a diagnostic rather than a synthetic score. A valid repository with no tests can still receive an evidence-based result capped at 25.
• A previous result helps explain change only when its source is usable. It never overrides current-revision evidence.

Rerun after substantive code, test, dependency, runner, or environment changes, and after an approved test-writing pull request lands. The new run measures the new revision; an unmerged review request does not improve the existing score. Later feature work can expose another untested path or make a deterministic check fragile, so green is not permanent.

This is why the recurring loop matters. Guard surfaces drift, helps the team make bounded improvements, and verifies the result again. Coding agents can then build new behavior on a suite that still catches meaningful regressions instead of trusting yesterday's green command by habit.