A CLAUDE CODE PLUGIN · ONBOARDING
qa-pilot
Skills for Claude Code: a QA plan per task, coverage analysis, requirement and test audits, test writing.
3 commands
1100+ QA rules
script checks
RULES · TESTS
Four rules for tests
A test must prove it can fail
New test: the agent breaks the code under test or the assertion, sees it fail, reverts; the main session spot-checks. Existing tests: the test audit breaks the code under a sample of tests with a script that restores the file itself.
Removes tests that pass whatever the code does.
Every expected result has a source
A source is a spec or AC line, a project decision, an invariant from the rule base, or a convention elsewhere in the code. The code under test is never a source. No source means a question for the owner, not a test.
Removes tests that lock in a bug as expected behaviour.
A UI test checks what the user sees
Deleted: the row is gone from the list. Saved: the value is there after a reload. A response status is checked only in addition. Exceptions (no trace in the UI, an API test) are marked in the test with the reason.
Removes the green test over a 200 response and an unchanged screen.
“Covered” means three references
A test by file and line, an assertion on this exact condition, a test layer that CI runs. “Not covered” is proven too: the search and its match count are attached.
Removes “it’s all covered” on trust.
RULES · WORKING IN A PROJECT
How the plugin works in your repo
Requirements are checked before code
When test runs before development: AC testability, consistency with the code and sibling specs, undefined entity states.
Asks instead of guessing
What is missing comes as one list of questions. Unanswered items go to “Open gaps”, and dependent work is marked blocked.
Project rules win over plugin rules
The repo’s CLAUDE.md, AGENTS.md and configs override the plugin. A conflict is noted in the report.
Limits on writing tests automatically
Without separate approval: up to 5 tests and 1 e2e per task. Product code is not changed. A bug test stays red. Deleting or weakening a test is the owner’s call.
Review before hand-off
Reviewers with narrow lenses: false greens, flakiness, simplicity, conventions. Up to 3 rounds; every finding is checked against the code.
Safety
The browser runs only on environments marked safe in the profile. Secret values are never read.
WHAT IT IS
A Claude Code plugin: a set of QA skills and a base of testing rules
Install. Two commands in Claude Code. Nothing in the repo needs to change.
Project context. The stack, test layers, commands and CI are written to a profile once.
Results. Plans, reports and tests are saved as files in the repo. Progress is written as work goes on, so an interrupted session can resume.
Contents. Three user commands, eight internal skills, a base of 1137 rules from the book plus 176 of its own, Python check scripts.
HOW IT WORKS
Three commands
| Command | What it does | When to run it |
|---|---|---|
| /qa-pilot:onboard | Creates the project profile: stack, test layers, commands, CI, critical flows, environments | Once, when you start. Again after a change of framework or test structure |
| /qa-pilot:test [context] | QA for one task: a plan before code, verification after code, browser testing, tests from a ready plan | Every task: before development starts and after it is done |
| /qa-pilot:audit [context] | Audits the whole project, one feature’s requirements, or the test code. The result is a prioritised work list | When joining a project, after a big feature, before a release, when tests are flaky |
Context is a ticket number, a feature name, a URL or “in the browser”. Without context, the mode is chosen from the branch and the diff.
THE OVERALL LOOP
Which command to run when
JOINING A PROJECT · ONCE
You join a project
onboard
audit
→ a risk map and a plan: what to fix, what to cover first
EVERY TASK
Ticket
test
QA plan
You write code
test
verification + tests
Merge
FROM TIME TO TIME
audit — after a big feature or a series of small changes, before a release
test “in the browser” — walk a flow by hand, reproduce a bug, check a fix
STEP 1 · HOW IT WORKS
/qa-pilot:onboard
1 · Scan, no questions
Reads manifests, configs, CI workflows, CLAUDE.md and AGENTS.md, a sample of tests. Records the test, lint and typecheck commands; which test layers CI runs; how API errors reach the UI; test framework maturity in 9 areas. Secret values are not read.
2 · Questions, in one message
Only what the code can’t tell: safe and forbidden environments, where test accounts come from, the tracker, 3–7 critical flows (the plugin proposes, you confirm), product traits, contradictions from step 1. Unanswered items go to “Open gaps”.
3 · The profile and where it is used
- commands and CI layers → running tests, the “covered” check
- critical flows → a P2 on them becomes Must
- product traits → which rules are picked from the base
- environments → where browser testing is allowed
4 · Checks
Runs the cheapest command from the profile. The profile-vs-repo check script must pass with no drift. The same check runs before every test and audit; on drift you get the list and an update of just those lines.
Output: docs/qa/project-profile.md. The other commands read it instead of scanning the repo.
STEP 2 · CHOOSING THE MODE AND THE PLAN BEFORE CODE
/qa-pilot:test
MODE FROM SIGNALS: WORDS AFTER THE COMMAND, TICKET, BRANCH DIFF, PLAN FILE
ticket, no code → plan
a diff exists → verify
“in the browser”, URL → exploratory testing
path to a plan → write tests
“cover” → tests for existing code
“PLAN” MODE — BEFORE DEVELOPMENT
1 · Scope
The change in user terms, the touched modules and entities, whether a critical flow is involved.
2 · AC check
Each criterion: testable? consistent with the code? complete against the must-check web patterns? A gap → an AC edit or a question.
3 · Impact map
Who uses the touched code: imports, queries, service calls. For each edge — file, line and expected effect.
4 · Test delta
Which tests to change, new tests by bucket Must / Should (up to 5) / Highlight, the regression set.
Output: docs/qa/tasks/<ticket>.md. Budget: about 80k tokens. Before it starts, the mode and the reason are shown in one line.
STEP 2 · VERIFICATION AFTER CODE
/qa-pilot:test
1 · Reconcile with the plan
Each diff hunk: planned, an explained refactor, or an unplanned behaviour change. Plan items not done are listed.
2 · Recompute impact
The impact map is rebuilt from the real diff. A change where “unchanged” was expected → an “intended?” question with evidence.
3 · Coverage
For each Must and Should: a test, an assertion on this exact condition, a layer in CI. Regression and new tests run with the profile’s commands.
4 · Missing tests
Written in the project’s style and reviewed. A test for a found bug stays red and goes to the owner with the bug.
LIMITS FOR AUTOMATIC TESTS
The expected result comes from the AC, the spec or a decision — not from the changed code
At most 5 new tests and 1 e2e per task; the rest goes to the owner as a list
Product code is not changed
Output: a “Verify log” section in the task file and a ready block for the PR description. Budget: about 100k tokens. If there was no plan, a compact one is built first.
STEP 3 · HOW THE PROJECT AUDIT WORKS
/qa-pilot:audit
No context: the whole project, steps 1–5. With a feature, spec or ticket: an audit of that feature’s requirements (step 3 at full depth). With the word “tests”: a test-code audit (step 4 over the whole suite).
1 · Foundation
Test framework maturity in the profile’s 9 areas. Blockers (a layer counted as coverage that CI doesn’t run; false-green tests) go first in Must.
2 · Critical flows
3–7 flows from the profile. You confirm the list and see the cost before it runs: about 150–250k tokens per flow.
3 · Flow audit
A script picks the rules that fit the product traits. Each entity gets an “entity × must-check pattern” card: every cell has a verdict with evidence, and an empty cell blocks the report.
4 · Test code
Tests around the critical flows: false greens, flakiness, wrong test level.
5 · Roadmap
A report and a JSON task list for an executing agent. The JSON goes through a validator: errors must be 0.
WHAT IT RELIES ON · 1
Priority by risk, not by coverage percentage
A finding’s priority is set by its consequences for the product, not by the wording of a rule.
P1 — data loss, broken security, no check before prod. Assigned only with the place in the code shown.
P2 — a serious gap. P3 — cosmetic.
Must
P1, any confirmed bug, and P2 on a critical flow. Always done
Should
Other P2s — at most 8, so the list can actually be done
Highlight
One line about the risk, no test — until you decide to raise it
Risks excluded by project filters are listed in a separate section of the report.
WHAT IT RELIES ON · 2
The testing knowledge base
1300+
rules in the testing knowledge base
Test design, requirements, APIs, forms, accessibility (WCAG 2.2), security, localisation, performance, mobile platforms, UX, AI features, test-code quality
Every rule is checked against standards
RFC, W3C, ISTQB, OWASP, NIST. Each rule has an authority level — standard, industry practice, vendor guideline or author’s opinion. An author’s-opinion rule cannot be mandatory.
Extended for product teams
qa-pilot’s own domains: web failure patterns, security (OWASP ASVS), HTTP APIs, authentication (NIST), test strategy, AI features, test-code quality — and rules learned on real audits.
Special thanks to Roman Surkov: the core rule set comes from his book “Quality Decides Everything”.
CONTROL · 1
What scripts check
The model
Judgment: what can break here, which scenarios matter, how to word a finding.
Scripts, no LLM
- pick rules by product traits
- find “TBD”, “fast”, “easy” in requirements
- validate the audit report: fields, source, verdicts
- break the code under a test and restore the file
- compare the profile with the repo
- check the plugin itself: skills and links
You
Product decisions: what is critical, which depth to choose, what to do with a bug.
Scripts check data and the shape of the result, not the model’s judgment. A report with a validation error is not done.
CONTROL · 2
Evidence requirements
“COVERED” ONLY WITH THREE PROOFS
1
A test exists — with the exact file and line
2
It catches this exact case, not a neighbouring one
3
This layer actually runs in CI
“No test” — only with the search attached: command and match count
Can fail: a new test is checked against broken code; existing ones by a script probe on a sample
Test code is reviewed by separate agents, up to 3 rounds; every finding is checked against the code
Product bugs are not fixed automatically: they go to the owner with steps and the expected behaviour
CONTROL · 3
Limits
| Limit | What it means in practice |
|---|---|
| Strongest on frontend and e2e | Works on a pure backend, but relies on the project’s conventions rather than a ready pattern library |
| Audits cost tokens | About 150–250k per flow. The plugin names the cost up front; you pick the flows |
| One repo, one profile | The plugin does not analyse links between repos |
| Needs answers to questions | In the first week, answering takes time: the profile’s accuracy depends on it |
| Product decisions stay with the team | Product code is not changed, business priorities are not set |
WHEN IT HELPS
Typical situations and commands
A new project with existing tests
onboard → audit
Test framework assessment, critical-flow risks, order of work
A task before development
test TICKET-123
AC check, impact map, list of tests to add
A task after implementation
test
Diff reconciled with the plan, missing tests
Checking a spec
audit <feature>
Contradictions, undefined states, product questions
Flaky tests
audit tests
False-green and flaky tests, duplicates, a refactoring plan
Manual check in the browser
test in the browser
Walking a flow, bugs with repro steps, re-checking a fix
HOW TO START
Install
1 /plugin marketplace add grooownow/qa-pilot
2 /plugin install qa-pilot@qa-pilot
3 /qa-pilot:onboard
Needs: Claude Code and Node.js. Works on a Claude Pro subscription.
Next: audit for the big picture, then test for every task.