A grounded retrieval agent, a two-layer verification gate, and an evaluation harness that answers the question most applied-LLM projects skip — not “does it work?” but “on exactly which cases can you trust it, and on which can’t you?” Validated across a real, adversarial 320-case ground-truth set.
Grounded RAG · LLM-as-judge · agentic self-correction · a stub → local (8B/14B) → Bedrock provider ladder behind one interface. Python stdlib-only, deterministic-first, 225 tests at $0. The test domain: NIST 800-171, the DoD contractor-security standard — chosen because its ground truth is genuinely hard, not because compliance is the point.
An LLM will fluently assert anything. The hard part of an applied system isn’t generation — it’s making the output trustworthy, and then knowing where to trust it. I built the machinery for both: retrieval that only exposes hand-checked facts, a gate that refuses any claim it can’t trace to a source, and an eval that scores every case against a human answer key across a ladder of models.
The test corpus is NIST 800-171 — a real compliance standard with 320 assessment objectives — and it was chosen precisely because the ground truth is adversarial: “the system provides evidence” and “the objective is satisfied” are frequently different things, a confident plausible answer is often wrong, and roughly half the objectives turn on human judgment no amount of evidence-matching can settle. That’s exactly the setting where a naive accuracy number lies and a careful eval earns its keep.
Same harness, three generators, all 14 families. The stub is the floor a model must beat — “has evidence → met” — and it cannot emit customer or inherited. Sorting families by how far the local qwen2.5:14b beats that stub traces a clean line from +100 to −18.
Read top-to-bottom, this is the deliverable. Where ground truth is mostly human judgment the heuristic can’t express (3.2 is customer 9/9 → stub 0, model 100), the model wins by the maximum margin. Where it aligns with an evidence-presence heuristic (3.4 is met-heavy), the model loses — by refusing to rubber-stamp borderline met calls the stub gets for free.
Why this is the whole point, not a footnote. A single accuracy number, averaged across the corpus, would rank a reasoning model below a one-line heuristic — because half the cases are exactly where the heuristic looks good. Aggregate rank is a property of the data’s shape, not of the models. So the harness scores the per-item diff, not a headline number, and the curve above is the proof it has to. That is the transferable lesson: an eval’s job is to tell you where capability lives, not to emit a leaderboard figure that averages the answer away.
Which model tier can actually do each job — and could a self-hosted model do it at all? I ran the identical harness — same prompts, same strict parser, same gate → judge flow — changing only the transport (a stdlib HTTP call to a local Ollama model vs. the AWS SDK to Bedrock). So a difference in the numbers is a difference in the model, not the scaffolding: a controlled ablation across four capability tiers. (Temperature 0, reproducible.)
| Role · metric | stub $0 | llama 8B | qwen 14B | Haiku (Bedrock) |
|---|---|---|---|---|
| judge · fabrications caught | 40% | 70% | 80% | 90% |
| generator · status-correct | 66% | 38% | 69% | 76% |
The 8B is a usable judge but an unusable generator — below the trivial stub, failing to emit parseable output for a third of objectives (a multi-part structured-output ceiling that recurred, near-identically, on all 14 families — evidence it’s a model property). Constrained decoding fixed the formatting and left correctness pinned; a bigger local model crossed to usable (38%→69%, zero unassessed). Recommendation, with a number behind it: Bedrock/GovCloud is the default, a 14B is a defensible self-hosted alternative, an 8B is a $0 judge pre-filter and never the sole gate.
You can’t catch a fluent fabrication by scoring it — a confident hallucination scores well. You catch it by refusing any sentence that doesn’t resolve to a hand-checked source. The gate is two layers: a deterministic citation gate (no model) rejects uncited or unresolvable sentences outright; an LLM entailment judge then answers the question the gate can’t — does this specific source actually support this sentence? When the agent met a real AWS account, a rule’s citable text said “CloudTrail is enabled” while its recorded verdict was NON_COMPLIANT: a grounded, cited, false finding that a lexical judge waves through. The reasoning judge reads the verdict and catches it — while still supporting the accurate reading.
An eval earns its keep by catching your own errors. Across the build it surfaced:
The eval catches my errors while I build. Auditing the ratified answer key is a different job, so I built an adversarial panel: an independent frontier model that challenges every verdict across four lenses (ownership, sufficiency, reproduction, promotion), on a strict-majority vote, defaulting to concede. It is answer-key-blind — it disputes the reasoning, and correctness is measured afterward, so it audits the ground truth rather than mirroring it. No class auto-applies — every lens flags for a human, a design decision with a number behind it, earned twice: first from a measured false-positive (a grounded challenge that a not_met should be met because a control is “enforced,” when the objective asked whether it was approved), and later from real deployment evidence that reversed the one lens still auto-applying — see below. An auto-flip would have been wrong; the flag caught it.
Run across the whole standard, three findings — the per-objective diff, not a headline:
The eval needed real infrastructure to reconcile claims against, so the agent runs on a production-grade AWS platform I designed and built: a dual-partition Landing Zone Accelerator (commercial + GovCloud), seven compliant-by-default Terraform modules (KMS, VPC, RDS, ECS/Fargate, endpoints, Inspector, patch management), Service Control Policy guardrails, an AWS Config conformance pack for continuous evidence, and a CI gate (fmt / validate / Checkov / policy-lint) on every change. The GenAI system is the differentiated work; this is the substrate that made its evidence real — and the platform / DevOps / IaC engineering it took to ship.
Every number above was scored against a curated evidence manifest — faithful, but hand-assembled and implicitly well-formed. So I deployed a real application onto that platform, turned on live AWS Config, and re-ran the identical harness on recorded evaluations from running infrastructure instead of the manifest — six families, 224 objectives. The status-correct number barely moved, which is itself the finding: it measures rule-to-objective mapping, and the evidence source doesn’t change that.
What moved was a defect the synthetic fixtures had hidden in my own adversarial layer. On tidy fixtures the one auto-applying lens (ownership) looked like a correction; on real, sparse evidence it reversed sign — it broke 23 already-correct calls to fix 6 (net −17). The cause is exactly what well-formed test data can’t produce: a freshly-deployed account has no customer-side policy, personnel, or process evidence, so the customer / not_met boundary is un-anchored and the model’s vote on it is unreliable in both directions — 63% of all errors landed on that one boundary. The fix is one line (demote the lens to flag-only, like the other three): status-correct 59.4% → 67.0%, reproduced end-to-end, and adjudicating the panel’s remaining flags reached 76% with zero new corrections to the ratified key — six of the contested calls converging on rows a human had already flagged to ratify.
Then a negative test, because validating good infrastructure isn’t the same as catching bad: a deliberately non-compliant deployment met four independent gates and beat none — Checkov (build red), a preventive Organization SCP (apply denied, nothing created), an AWS Config rule (an over-permissive 0.0.0.0/0:22 SG the SCP can’t express, flagged NON_COMPLIANT), and the agent itself: the flipped evidence made the grounding gate reject a confidently-false met and support the honest gap. Gate 4 is the layer this system adds to an otherwise-standard defense-in-depth stack — the one that refuses a fluent, cited, wrong answer.
▶ Watch the four-gate catch — a self-contained terminal recording of the negative test, reproduced live 2026-07-27: Checkov exit 1, three verbatim SCP AccessDenied errors, a fresh Config recorder flagging only the open SG, and the Haiku 4.5 grounding judge refusing the false pass. Every line is real output.
If one frontier model has these blind spots, does a second, independent one catch them? The tempting answer is self-consistency — sample two models, trust the agreement — so I measured it instead of assuming it. A much smaller, cheaper generator (Haiku 4.5) ran the identical harness on the same real evidence, with only the gen+judge model changed and the panel and evidence graph held constant, then the runs were diffed objective-by-objective. Three results, none of them the tidy one:
met → not_met, under-crediting real controls; customer → not_met, missing the ownership boundary), now shown to persist across a model-size gap. This is the empirical case against naive self-consistency as a correctness proxy: ensembling models whose errors share a direction buys confidence, not accuracy.{.*} regex. When a model emits a second brace-object after its verdict — which Haiku does deterministically and Opus never did — the regex swallowed both and json.loads aborted the entire family. Fixed to take the first valid object via raw_decode (the tolerance the generator’s array path already had), with four regression tests. Only a second model’s output distribution exposed it — the same lesson as the ownership lens, one layer down.Built a grounded RAG + LLM-as-judge system with a two-layer verification gate (deterministic citation check + LLM entailment judge), agentic self-correction, and an MCP tool server, then designed the evaluation harness that measures where it can be trusted — hand-labelling a 320-case ground-truth set and running a controlled stub → local (8B/14B) → Bedrock ablation across four capability tiers. Produced an eval-validated per-item map of model trustworthiness, established a monotone relationship between a reasoning model’s advantage and how much of a task needs human judgment (+100 to −18), and proved that a naive aggregate accuracy number ranks the models backwards. The harness also caught and corrected a systematic defect in its own upstream ground truth. Added an answer-key-blind adversarial verification layer that audited the ratified key across the full standard, surfacing exactly two real over-credits while validating the rest. Then proved the whole system against a real deployed application: ran it on live AWS Config evidence from a running app, where sparse real evidence exposed and let me fix a sign-reversed defect in the verification layer itself (+17 points), and a deliberate misconfiguration was caught at four independent gates. Runs on a production AWS platform (Terraform / Landing Zone Accelerator / SCPs / CI) I built.
“An eval that catches your own ground-truth errors is doing its job — and real evidence that audits the auditor is worth more still.”