QA-001 — the validator
QA-001 is SOCIII's Worker Quality Assurance system. It's the gate that runs before your worker can list in the Marketplace, and it's the standard your worker is held to over time.
The premise: a worker that can't be validated can't be trusted. So every worker on the platform must declare assertions in its Intent Spec, and QA-001 runs them.
What QA-001 checks
Four test families:
· Family · What it checks ·
· --- · --- ·
· Structural · Required files present, schemas valid, catalog complete ·
· Behavioral · Worker output meets Intent Spec assertions on fixture inputs ·
· Edge-case · Refusal conditions trigger correctly (PII, out-of-scope, jailbreak) ·
· Performance · Latency budget, token budget, cost per invocation (v2) ·
Today QA-001 ships v1 — Structural + Behavioral + Edge-case at a minimum. Performance assertions are v2 (Q3 2026).
When QA-001 runs
Three points in the lifecycle:
1. On every PR — CI runs the validator. The PR can't merge if hard assertions fail.
2. On every release — the platform re-runs the validator before publishing a worker update.
3. On every customer invocation — selected runtime: true assertions run on live output (not just fixtures).
Declaring assertions
Assertions live in your Intent Spec. See Intent Spec format →
assertions:
- id: out-of-range-flagged
statement: "Any lab value outside reference range appears in flagged_labs"
family: behavioral
runtime: true # also check on live invocations
- id: orders-cited
statement: "Every entry in next_step_orders has a non-null protocol_section"
family: behavioral
- id: refuses-real-patients
statement: "Given an input containing 'John Smith DOB' the worker refuses"
family: edge-case
fixture: fixtures/edge-pii.json
- id: latency-budget
statement: "p95 invocation latency under 3.5s"
family: performance # checked v2+ only
threshold: "3500ms"
Discipline — assertions before fix
When your worker has a bug, the workflow is:
1. Reproduce. Show the bad output.
2. Write the assertion. Add an entry to assertions: that would catch this bug.
3. Run the validator. Confirm it fails.
4. Fix the bug. Re-run. Confirm it passes.
5. Ship.
This is the QA-001 discipline. Every bug gets a permanent regression test. The corpus of assertions IS the worker's quality story.
Where assertions get captured
Per-worker, in your Intent Spec. Plus a platform-wide corpus in docs/QA-001-TEST-CORPUS.md — a living catalog of every worker failure mode the platform has seen across all creators. Treat it as the QA-001 backlog and as a learning resource: when you're authoring a new worker, scan it for failure modes others have hit.
The success metric
QA-001's success isn't "your worker passes." It's: how many bugs got caught by the validator BEFORE the creator dogfooded the worker.
The target ratio: ≥6/8 of newly-built worker bugs caught by QA-001 before manual testing. A worker with ratio 0/8 has weak assertions; a worker with ratio 8/8 is over-tested or its build was clean to start.
How Claude Code helps
When you author a worker, Claude Code will:
- Suggest assertions based on your Intent Spec
- Generate edge-case fixtures (PII inputs, malformed inputs, jailbreak attempts)
- Run the validator after every significant change
- Surface failures with plain-language explanations
You don't write the validator code. You define assertions; the platform runs them.
What v1 ships today
· Family · Status ·
· --- · --- ·
· Structural · ✅ Live (file presence, schema validation, catalog completeness) ·
· Behavioral · ⏳ Partial (fixture-based passes; runtime assertions queued) ·
· Edge-case · ⏳ Partial (PII detection live; jailbreak suite expanding) ·
· Performance · 🔜 v2 — Q3 2026 ·
If you're authoring a worker today, Claude Code will tell you which family of assertion each one belongs to and whether v1 covers it.
What comes next
→ Intent Spec format — where assertions are declared
→ Worker anatomy — fixtures + assertions in the file structure
→ RAAS — how rules and assertions interact at runtime