A grounded retrieval agent that drafts citation-checked NIST 800-171 assessment findings for every one of the standard’s 320 objectives — and an evaluation harness that maps exactly where a language model can be trusted on the task, and where it can’t.
Built on my own CMMC L2 / NIST 800-171 AWS landing zone. Python stdlib-only (boto3 the one optional dep). SQLite control graph · two-layer grounding gate · MCP server · a stub → local → Bedrock provider ladder behind one interface.
CMMC Level 2 makes a contractor prove, per assessment objective, that a control is implemented — and keep the SSP honest about what is actually evidenced versus merely claimed. In practice the implementation matrix over-claims: a row says “implemented” while the defining artifact is a TODO, or the mapped check tests something adjacent to what the objective asks. Reconciling claim against evidence, at objective granularity, across a whole standard, is exactly the work a grounded LLM should help with — if it can be trusted not to invent the evidence, and if you can measure where its judgment holds.
Same harness, three generators, all 14 families. The stub is the floor a model must beat — “has evidence → met” — and it cannot emit customer or inherited. Sorting families by how far the local qwen2.5:14b beats that stub traces a clean line from +100 to −18.
Read top-to-bottom, this is the deliverable. Where ground truth is mostly human judgment the heuristic can’t express (3.2 is customer 9/9 → stub 0, model 100), the model wins by the maximum margin. Where it aligns with an evidence-presence heuristic (3.4 is met-heavy), the model loses — by refusing to rubber-stamp borderline met calls the stub gets for free.
Why this is the point, not a footnote. A single accuracy number, averaged across families, would rank a reasoning model below a one-line heuristic — because half the standard is exactly where the heuristic looks good. Aggregate rank is a property of the family’s shape, not of the models. That is why the harness scores the per-objective diff, and the curve above is the proof it has to.
Could a self-hosted model keep CUI on infrastructure you control? I ran the identical harness — same prompts, same parser, same gate → judge flow — differing only in the transport. A difference in the numbers is a difference in the model, not the scaffolding. (Family 3.3, temperature 0.)
| Role · metric | stub $0 | llama 8B | qwen 14B | Haiku (Bedrock) |
|---|---|---|---|---|
| judge · fabrications caught | 40% | 70% | 80% | 90% |
| generator · status-correct | 66% | 38% | 69% | 76% |
The 8B is a usable judge but an unusable generator — below the trivial stub, failing to emit parseable output for a third of objectives (a multi-part structured-output ceiling that recurred, near-identically, on all 14 families — evidence it’s a model property). Constrained decoding fixed the formatting and left correctness pinned; a bigger local model crossed to usable (38%→69%, zero unassessed). Recommendation, with a number behind it: Bedrock/GovCloud is the default, a 14B is a defensible self-hosted alternative, an 8B is a $0 judge pre-filter and never the sole gate.
You can’t catch a fluent fabrication by scoring it — a confident hallucination scores well. You catch it by refusing any sentence that doesn’t resolve to a hand-checked source. The gate is two layers: a deterministic citation gate (no model) rejects uncited or unresolvable sentences outright; an LLM entailment judge then answers the question the gate can’t — does this specific source actually support this sentence? When the agent met a real AWS account, a rule’s citable text said “CloudTrail is enabled” while its recorded verdict was NON_COMPLIANT: a grounded, cited, false finding that a lexical judge waves through. The reasoning judge reads the verdict and catches it — while still supporting the accurate reading.
An eval earns its keep by catching your own errors. Across the build it surfaced:
The eval catches my errors while I build. But once a family’s answer key is ratified, what audits it? I built an adversarial panel: an independent frontier model that challenges every verdict — argues the opposite — across four lenses (ownership, sufficiency, reproduction, promotion), on a strict-majority vote, defaulting to concede. It never sees the answer key; it disputes the reasoning, and correctness is scored afterward — so it audits the ground truth rather than mirroring it. No class auto-applies — every lens flags for a human, a discipline earned twice: first by a measured false-positive (a grounded challenge that a not_met should become met because a control is “enforced,” when the objective asked whether it was approved — a governance sign-off, not a config), and later by real deployment evidence that reversed the one lens still auto-applying (see Proven on a real deployment, below).
Run across the whole standard, three findings — the per-objective diff, not a headline:
Everything above was scored against a curated evidence manifest — faithful, but hand-assembled and (like all test data) implicitly well-formed. So I deployed a real financial application onto the compliant landing zone, turned on live AWS Config, and re-ran the identical harness on recorded evaluations from running infrastructure — six control families, 224 objectives.
The headline isn’t the score (unchanged — itself a finding: it measures rule-to-objective mapping, which the evidence source doesn’t move). It’s that real evidence caught a defect the synthetic fixtures had hidden — in my own adversarial layer. The one lens that still auto-applied reversed sign on a bare account: it looked like a correction on tidy fixtures, but on real, sparse evidence it broke more calls than it fixed (−17), because a fresh deployment has no customer-side policy or process evidence to anchor the customer vs. not_met line. The one-line fix — make that lens flag like the others — lifted status-correct +17 (to 67%), reproduced end-to-end; adjudicating the panel’s remaining signals reached 76%, with zero new corrections to the ratified key. Six contested calls even landed on the exact objectives a human had already tagged “ratify” — an independent second opinion converging on the human’s own soft calls.
Then the inverse: a deliberately non-compliant deployment, to prove the guardrails catch violations, not just validate good ones. It met four independent gates and beat none:
apply denied (unencrypted database, wrong region, public-access-block disable); nothing created.0.0.0.0/0:22 security group the SCP can’t express, flagged NON_COMPLIANT the moment it existed, and only that resource.met and support the honest gap. This is the gate this system adds to an otherwise-standard defense-in-depth stack, and it closes the exact hole the project exists for.▶ Watch the four-gate catch — a self-contained terminal recording of the negative test, reproduced live; every command and verdict is real output, ending on the grounding gate refusing the false pass.
If one model has blind spots, does a second, independent one catch them? It’s the obvious tempting answer — run two models, trust what they agree on — so I measured it rather than assumed it. A much smaller, cheaper model went through the identical harness on the same real evidence, everything else held constant, and the two runs were compared objective by objective. Three results, none of them the tidy one:
Built a retrieval-augmented compliance agent that drafts citation-grounded NIST 800-171 findings for all 320 objectives, with a two-layer grounding gate (deterministic citation check + LLM entailment judge), an agentic revise loop, an MCP server, and a hand-labelled eval harness. Ran a stub → local (8B/14B) → Bedrock comparison over the complete standard and produced an eval-validated per-objective map of where a self-hosted model is trustworthy — establishing a monotone relationship between a model’s advantage and how much of a family needs human judgment (+100 to −18), and proving a naive cross-family accuracy number ranks the models backwards. The eval also caught and corrected a systematic defect in its own upstream catalog. A first-party adversarial verification layer then audited the ratified answer key across the full standard, surfacing exactly two real over-credits (both the same evidence-scope error signature) while validating the rest. Finally, proved the system against a real deployed application: assessed the agent on live AWS Config evidence from a running app on the landing zone — which exposed and let me fix a sign-reversed defect in the adversarial layer itself (+17 points) — and caught a deliberate misconfiguration at four independent gates (CI, preventive SCP, detective Config, and the agent’s own grounding gate rejecting a false pass).
“An eval that catches your own ground-truth errors is doing its job — and real evidence that audits the auditor is worth more still.”