The Control Evidence Agent — Grounded GenAI for NIST 800-171 / CMMC

This is the summary. The full case study — with the interactive per-family chart — lives here: View the full visual case study →. There is also an AI-engineering deep-dive for the systems detail. Or watch the whole loop in 90 seconds → — a self-contained recording of the four beats: the grounding floor, the cited POA&M, the refusal of a false pass, and the trust-map.

A retrieval-augmented agent that drafts NIST SP 800-171 assessment findings — a status (met / not_met / customer / inherited) plus a cited justification for each of the standard’s 320 assessment objectives — and self-checks every sentence against the source it cites before the finding is allowed to stand. Built entirely as internal R&D on my own CMMC Level 2 / NIST 800-171 AWS landing zone, with a stub → local-model → Amazon Bedrock provider ladder behind a single interface.

The project was never “an LLM writes compliance text.” It was to build the machinery that makes an LLM’s compliance text trustworthy and measurable — and, ultimately, an eval-validated map of exactly where a language model can be trusted on this task and where it can’t. That map is the deliverable.

The Challenge

CMMC Level 2 requires a contractor to show, for each of 110 requirements (320 assessment objectives), that a control is implemented — and to keep the SSP narrative honest about what is actually evidenced versus merely claimed. In practice the implementation matrix over-claims: a row says “implemented” while the defining artifact is a TODO, or the mapped Config rule checks something adjacent to what the objective actually asks.

Reconciling claim against evidence, at objective granularity, across a whole standard, is exactly the tedious, high-stakes work a grounded LLM should help with — if it can be trusted not to invent the evidence, and if you can measure where its judgment is sound.

What I Built

Four layers, deterministic-first, each testable at $0 with no model, no network, and no AWS:

• A deterministic control graph (SQLite) — the standard’s requirements and objectives (vendored, pinned, hashed OSCAL), the implementation matrix (the claims), typed evidence (Config rules plus IaC / SCP / AWS-inherited artifacts with file-and-line provenance), and a hand-labeled answer key. Every importer is a gate that refuses a graph it can’t vouch for.
• A grounded generator — retrieve a typed context pack per control, draft a per-objective status plus one cited sentence, then pass it through a two-layer grounding gate: a deterministic citation check, then an LLM entailment judge. An agentic revise loop re-drafts ungrounded claims — with the status frozen, so it can only fix wording and citations, never change what it asserts.
• An MCP server — the reconciliation tools exposed over JSON-RPC (standard library only) so any MCP client can drive the harness.
• An evaluation harness — scores every objective against the answer key on two orthogonal axes (grounding vs. correctness), across a ladder of models, with every accuracy change attributable to a specific diagnosed cause.

Python standard library only (boto3 the single optional dependency, lazily imported for the Bedrock path). temperature=0 and a fixed seed, so runs reproduce exactly. 225 tests, none touching a model, a network, or AWS.

The Headline Result

I brought all 14 families / 320 objectives to hand-verified ground truth, then ran the identical harness with three generators: a $0 lexical stub, a local qwen2.5:14b (Ollama, on a 16 GB laptop), and Claude Haiku 4.5 on Bedrock. The stub is the floor a real model must beat — “has evidence → met” — and it structurally cannot emit customer or inherited.

Sorting the 14 families by the 14B’s status-correctness margin over the stub produces a clean, monotone curve in how much each family’s ground truth requires human judgment:

Where a family’s ground truth is dominated by human judgment the heuristic can’t express — Awareness & Training is customer 9/9, so the stub scores 0 and the reasoning model scores 100 — the model wins by the maximum possible margin. Where the ground truth aligns with an evidence-presence heuristic (Configuration Management is met-heavy because the IaC codebase is the config baseline), the model loses by refusing to rubber-stamp the borderline met calls the stub gets for free.

The takeaway is methodological and load-bearing: aggregate correctness rank is a property of the standard’s shape, not of the models. A single accuracy number, averaged across families, would rank a reasoning model below a one-line heuristic — because half the standard is exactly where the heuristic looks good. That is why the harness scores the per-objective diff, and the curve above is the proof it has to.

Self-Hosted vs. Bedrock, for CUI

For CUI you’d love to keep data on infrastructure you control — so could a self-hosted model do the generator or judge job, or is a managed frontier model required? I ran the identical harness, differing only in the transport (a standard-library HTTP call to Ollama vs. the AWS SDK), so a difference in the numbers is a difference in the model, not the scaffolding. On family 3.3, temperature 0:

The 8B is a usable judge but an unusable generator — below the trivial stub, failing to emit parseable output for a third of objectives (a structured-output ceiling that recurred, near-identically, on all 14 families). A bigger local model crossed to usable (38% → 69%, zero unassessed), within 7 points of the frontier — so the bottleneck was reasoning, not formatting. The recommendation, with a number behind it: Bedrock in GovCloud (FedRAMP-High) is the default for generation, a 14B-class model is a defensible self-hosted alternative, and an 8B is a $0 judge pre-filter — never the sole gate.

What the Eval Caught (That I Didn’t)

An eval earns its keep by catching your own errors. Across the build it surfaced:

• A systematic 23-objective defect in the vendored ground-truth catalog — every single-determination objective mis-split upstream. Unrepaired, the importer would have silently scored those requirements against blanks. Fixed at import, verified against the authoritative NIST PDF, locked by tests.
• An inconsistency in my own answer key — two near-identical objectives labeled differently, which the key itself had flagged. Provably not score-gaming: the fix didn’t move the deterministic stub.
• False matrix claims, family after family — an MFA-requiring SCP, an “unused-credentials” Config rule, pipeline approvals, GuardDuty Malware Protection, a wired incident-notification path, an “immutable” Object-Lock archive, an SSM patch module aimed at hosts that don’t exist. Each claimed in docs and absent in code — exactly the “claimed but not implemented” gap a C3PAO exists to find.

A Second Referee — Auditing the Ground Truth Itself

An eval catches my errors while I build; but once a family’s answer key is ratified, what audits it? I built an adversarial panel: an independent frontier model that challenges every verdict — argues the opposite — across four lenses (ownership, sufficiency, reproduction, promotion), on a strict-majority vote, defaulting to concede. It never sees the answer key; it disputes the reasoning, and correctness is scored afterward — so it audits the ground truth rather than mirroring it. No class auto-applies — every lens flags for a human, a discipline I earned twice: first from a measured false-positive (a grounded challenge that a not_met should become met because a control is “enforced,” when the objective asked whether it was approved — a governance sign-off, not a config), and later from real deployment evidence that reversed the one lens still auto-applying (see Proven on a real deployment, below).

Run across the whole standard, it produced three findings — each the per-objective diff, not a headline:

• The error signature is singular. It found exactly two genuine over-credits in my ratified key — both the same shape: a real control cited for a slice of a mostly-human-owned objective. Everything else it flagged held on review. A tool that finds real errors and knows when to leave ground truth alone is worth more than one that only accuses.
• The fault line has a physical address. All the contestation localizes to the handful of infrastructure-native families where infrastructure-as-code gets credited for a “defined/identified” objective; the rest came back near-silent (one flag across ~136 objectives, and it caught the generator, not the key).
• One model doesn’t exhaust a key’s weak spots. Two different generators surfaced largely disjoint flags on the same families — same seam, rotating targets — so auditing a control key reliably takes multiple models, not one.

An eval that can audit its own ground truth is worth more than one that only scores it.

Proven on a real deployment

Everything above was scored against a curated evidence manifest — faithful, but hand-assembled and (like all test data) implicitly well-formed. So I deployed a real financial application onto the compliant landing zone, turned on live AWS Config, and re-ran the identical harness on recorded evaluations from running infrastructure instead of the manifest — six control families, 224 objectives.

The headline isn’t the score (unchanged, which is itself a finding: it measures rule-to-objective mapping, which the evidence source doesn’t move). It’s that real evidence caught a defect the synthetic fixtures had hidden — in my own adversarial layer. The one lens that still auto-applied reversed sign on a bare account: it had looked like a correction on tidy fixtures, but on real, sparse evidence it broke more calls than it fixed (−17), because a fresh deployment has no customer-side policy or process evidence to anchor the customer vs. not_met line. The one-line fix — make that lens flag like the others — lifted status-correct +17 (to 67%), reproduced end-to-end; adjudicating the panel’s remaining signals reached 76%, with zero new corrections to the ratified key. Six of the panel’s contested calls even landed on the exact objectives a human had already tagged “ratify” — an independent second opinion converging on the human’s own soft calls.

Then the inverse test: a deliberately non-compliant deployment, to prove the guardrails catch violations rather than only validate good ones. It met four independent gates and beat none — CI static analysis (build red), a preventive org policy (apply denied, nothing created), runtime detection (an over-permissive rule flagged the moment it existed), and the agent itself (the flipped evidence made the grounding gate reject a confidently-false met and support the honest gap). That last gate is the one this system adds to an otherwise-standard defense-in-depth stack — and it closes the exact hole the project exists for.

▶ Watch the four-gate catch — a self-contained terminal recording of the negative test, reproduced live: every command and every verdict is real output, ending on the grounding gate refusing the confidently-false pass.

A Second Opinion — Breadth Net, Not a Consensus Oracle

If one model has blind spots, does a second, independent one catch them? It’s the tempting answer — run two models, trust what they agree on — so I measured it rather than assumed it: a much smaller, cheaper model through the identical harness on the same real evidence, everything else held constant, then the two runs diffed objective by objective.

Agreement turned out to be a weak confidence signal, because the blind spots are shared. The two models agree on 78% of objectives — but on that agreed subset they’re right only 72% of the time, barely better than either alone, and 28% of their agreements are agree-and-both-wrong. The shared errors sit in exactly the two places already identified as frontier blind spots (under-crediting real controls; missing the ownership boundary), now shown to persist across a large gap in model size. Two models that fail in the same direction make you more confident, not more correct — which is precisely the failure mode an assessment can’t afford.

What a second model is good for is narrower and more useful: the cheaper one uniquely recovers 16 objectives the larger one missed, lifting the two-model ceiling to 74%. So the defensible use is a breadth net whose disagreements feed the review queue — not a consensus vote that launders correlated error. The human still adjudicates; the second model only says where to look. (It also found a latent bug before it scored anything: both the grounding judge and the challenger panel extracted verdicts with a pattern that broke on the second model’s output style — something the first model never produced, so no amount of testing against it would have surfaced the fault. The same lesson as the ownership lens, one layer down.)

Honest Limitations

• One run per family at temperature 0 — reproducible, but these numbers carry real run-to-run variance; the direction and the per-objective diff are the measurement, not the third significant figure.
• One local hardware tier — a 16 GB laptop caps at ~14B; whether a 32B/70B closes the last 7 points to the frontier is inferred, not measured.
• The residual generator errors are a genuine judgment boundary, not fabrications — chiefly the customer vs. not_met line (is a missing process control the contractor’s duty, or a gap the platform should close?). That is precisely where a human assessor still owns the decision, and the tool’s job is to surface the evidence, not pretend to settle it. The real-deployment run put a number on it: 63% of that run’s errors live on the customer boundary — the honest edge of what an automated grounder should be deciding at all.
• The real-evidence run is one deployment and six of fourteen families — the families where technical evidence actually bites. The customer/inherited-heavy families weren’t re-run against live infrastructure, because a bare account has nothing new to show them. Broader real-evidence coverage, across more deployments and more postures, is the next measurement, not a settled one.

Results

An eval that catches your own ground-truth errors is doing its job — and real evidence that audits the auditor is worth more still.

The project produced an eval-validated, per-objective map of where a language model can be trusted on NIST 800-171 assessment — the delta between models, measured on one harness, rather than asserted — and then proved it against a real deployed application: the agent assessed live infrastructure, real evidence found and fixed a flaw in the agent’s own verification layer, and a deliberate violation was caught at four independent gates. Deterministic first, model last. For the full write-up with the interactive per-family chart, read the complete visual case study →.