Test quality audit

What this audit answers

The test quality audit asks how well the repository’s tests prove useful behavior, catch regressions, and remain maintainable. It does not reward file count alone. A large suite with decorative assertions, hidden environment requirements, or no coverage of primary user flows can be less protective than a smaller, focused suite with clear execution evidence.

What Guard inspects

Guard inventories every primary source surface and the test layers that protect it: unit, integration, contract, end-to-end, browser, and other repository-specific checks. It reads test configuration, scripts, fixtures, CI jobs, coverage reports, skip markers, and documented environment requirements. Bounded test commands may be run when setup is safe and practical; unavailable services or toolchains are recorded explicitly.

Coverage is treated project-wide. A backend coverage number is not presented as repository coverage when a separate frontend or worker surface is unmeasured. Generated, vendored, fixture-only, and docs-only areas are excluded from the primary-surface calculation.

The 24 criteria

The scorecard covers whether tests solve real problems and catch regressions; unit, integration, and real user-flow coverage; state transitions, evidential assertions, maintainable test code, meaningful coverage, realistic and negative scenarios; reproducibility, passing execution, documented commands, isolation, environment requirements, fixtures and cleanup; separation of mocks from real integration, reasonable execution time, and testability signals.

Each criterion receives 0–5. Scores of 4–5 show strong evidence, 2–3 show partial protection, and 0–1 show absent or misleading protection. CI that runs meaningful tests is positive evidence, but branch-protection quality belongs to the CI/CD audit.

How the score works

The baseline is the sum of all 24 criterion scores divided by 120 and converted to a percentage. When credible project-wide measured coverage exists, the final calculation uses 70% criteria score and 30% effective project coverage. Partial or estimated coverage is kept conservative and clearly labeled.

Evidence-backed caps prevent a nominal suite from appearing healthy:

• no tests caps the final score at 25;
• tests with no documented runnable command cap it at 55;
• setup-only execution failure can cap it at 70, unless CI explicitly provisions the missing dependency and supplies meaningful independent evidence;
• widespread decorative assertions cap it at 65;
• committed skips that disable multiple tests, suites, or important paths cap it at 79.

If a primary user-facing surface has no tests or measurable coverage, effective coverage cannot enter the green zone. A frontend product without browser, end-to-end, or user-flow tests caps effective coverage at 60.

Limitations and boundaries

A controlled audit environment may not reproduce private services, production data, hardware, or provider-only gates. Guard separates that coverage limit from evidence that tests are broken. It also avoids turning CI policy, application architecture, or broad code style into duplicate test findings, although those facts can explain why verification is hard.

Improvements and reruns

Prioritize the missing evidence that protects real user value: repair a runnable command, replace decorative assertions, restore an important skipped suite, add a boundary or failure scenario, or cover an unprotected primary surface. A raw coverage increase is not enough if it exercises irrelevant lines.

After merging the change, rerun the audit and compare the same primary surfaces, execution evidence, skips, and checked revision. The new score should trace back to stronger proof, not simply more test files.