When a local 31B model matched the cloud frontier on grounded RAG drafting

Grounded RAG drafting eval

local 31B on the clean frontier

Gemma 4 31B AWQ did not merely look good for a local model. On this corpus and prompt set, it stayed near GPT-5.1 on verified substance while avoiding the highest-risk failure: real citations attached to the wrong project.

GPT-5.1 was not the original comparison target. It became the practical OpenAI baseline after GPT-5.5-Pro's uncapped trace ran about $15 and 577 seconds.

6 prompts N=10 reliability 8-model panel Sonnet 4.6 + GLM 5.1 rep=0 substance grading

90.8%Gemma 4 31B clean score; 59/65 strict-graded claims

93.1%GPT-5.1 clean score; 54/58 strict-graded claims

0Gemma fabrications or ambiguous project binds

~172xGPT-5.5-Pro cost gap vs GPT-5.1 in the capped N=3 rerun

TL;DR

• I expected the cloud frontier models to dominate a citation-heavy RAG drafting workload. They did not dominate.
• GPT-5.1 and self-hosted Gemma 4 31B AWQ formed the clean frontier: both had 0 fabrications and 0 ambiguous project bindings in the checked claims.
• The expensive failures were not simple hallucinations. They were wrong-project citation binding, SQL scope leakage, lifecycle-state mistakes, hidden reasoning cost, and evaluator false positives.
• The practical takeaway: for high-stakes RAG, "has citations" is not enough. The cited evidence must support the exact claim, project, value type, and lifecycle state.

I have been building a corpus-grounded drafting and analysis system for government-contracting documents. The workload is deliberately unforgiving: proposal sections, past-performance narratives, contract values, agencies, lifecycle state, SQL aggregates, and inline citations that a reviewer can click.

That makes it a good testbed for grounded generation. A fluent model can sound right while being useless if it attaches a real dollar value to the wrong project, cites an overview page for a claim that actually lives in a different document, or turns an in-flight proposal into completed past performance.

The surprise was not that a local model could write fluent prose. The surprise was that Gemma 4 31B stayed disciplined under citation and substance checks.

The setup

The main run was a 6-prompt x N=10 bench across cloud and local models. The prompts spanned the product surface: short and long past-performance drafting, SQL-only rollups, a single-project SOW prompt, an honest-abstention prompt, and a cross-tool portfolio overview.

The panel included GPT-5.1, Sonnet 4.6, Gemini 2.5 Pro, GLM 5.1, and several local models on a dual-3090 Xeon rig: Gemma 4 31B AWQ, Qwen 3.6 27B AutoRound, Granite 4.1 30B FP8, and Gemma 4 26B MoE.

Reliability and wall time were measured across the N=10 runs. Substance was graded on rep=0 for each model x prompt cell using a six-verdict rubric:

• CORPUS_GROUNDED + CITATION_CORRECT: the claim is in the corpus and the cited evidence supports it.
• CITATION_WEAK: the claim is real, but the cited chunk is an overview, adjacent document, or weaker source.
• AMBIGUOUS_GROUNDED: the value is real, but the model attached it to the wrong project context.
• VALUE_TYPE_UNSPECIFIED: the value is real, but the model blurred award, ceiling, estimate, or final-cost cadence.
• PARTIALLY_GROUNDED: the base fact is real, but the model added an unsupported qualifier.
• FABRICATED: the name or value was not found after exhaustive search.

The clean frontier

The headline result was simple: GPT-5.1 and Gemma 4 31B were the two cleanest models in the panel.

Clean correct means the strictest verdict: the claim is in the corpus, the citation supports it, and the project context matches. The gap between "claims checked" and "clean correct" does not automatically mean "wrong." For GPT-5.1, for example, the four non-clean rows were minor flags: one value-type/cadence issue and three partially grounded editorial overlays, not fabrications or wrong-project bindings.

Gemma's strongest property was not output beauty. It was discipline. Across 65 checked claims, it had zero fabrications, zero wrong-project bindings, exact SQL aggregates, and a clean honest-abstention result on an in-flight proposal prompt.

Its weaknesses were real but much less severe: four citation-weak rows where it leaned on overview chunks, one award-vs-final-cost cadence ambiguity, and one "low-risk partner" editorial qualifier. Those are reviewer-labor issues. They are not the same as inventing a contract value or binding a school project to a clinic narrative.

The GPT-5.5-Pro cost trap

The result that first forced me to slow down was GPT-5.5-Pro. In an initial six-way bake-off, one GPT-5.5-Pro run cost about $15 and took 577 seconds. That looked suspicious, so I dug into the trace.

The cost was real. It came from three compounding factors:

1. GPT-5.5-Pro chose a more exhaustive tool-use strategy: 16 retrieves across 4 turns.
2. Every agent turn resent the accumulated conversation and prior tool results, producing 418k cumulative input tokens in the N=1 trace.
3. The model appeared to spend heavily on hidden reasoning: 15.5k billed output tokens versus far fewer visible prose tokens.

The later N=3 rerun gave the cleaner comparison. That rerun used a per-call max_tokens=3000 cap, so GPT-5.5-Pro dropped to a $9.78 median run cost instead of the uncapped initial ~$15 trace. Even under that capped protocol, GPT-5.5-Pro stayed high-quality but GPT-5.1 delivered substantively similar quality at a radically lower cost.

The lesson is not "never use a reasoning model." The lesson is that reasoning-model economics are workload-dependent. In a multi-turn RAG agent, more thinking and more retrieval can create a compounding bill without creating a proportional quality gain.

The interesting failures were not just hallucinations

In high-stakes RAG, hallucination is too blunt a word. The panel surfaced several distinct failure modes with different operational meanings.

One example: a model can cite a real chunk that mentions a real 153-container logistics fact, but place that fact inside the wrong project's paragraph. A simple corpus regex says "grounded." A procurement reviewer sees a cross-project attribution slip.

Another example: SQL scope. A model can compute a real total from real rows and still be wrong because it failed to filter to document-scope awarded values. That is not a prose hallucination. It is a query semantics failure.

The evaluator can be wrong too

The most uncomfortable part of this work was that the judging process failed before the models did.

In earlier checks, Opus repeatedly accused Gemma 4 31B of fabricating values that were actually present in the corpus. The failure was mine: narrow substring searches, fixed-width chunk previews, missing page-level OCR stores, and stopping at the cited chunks instead of searching the full corpus before using the word "fabricated."

Do not collapse citation weakness into fabrication

The corrected rubric separates corpus truth, citation support, and project binding. A missing value is fabrication. A real value with a weak cite is a citation problem. A real value attached to the wrong project is an attribution problem.

This changed the methodology. I now treat "fabricated" as a high bar. Before writing that word, the verifier must search across the relevant stores: chunk content, contextualized content, multimodal page text, and tool-call traces, with value variants and OCR-tolerant patterns.

The corrected distinction matters:

• Fabricated: the value or name is not in the corpus after exhaustive search.
• Citation weak: the claim is corpus-real, but the cited chunk is not the best or sufficient evidence.
• Ambiguous grounded: the claim is real, but attached to the wrong project context.

That distinction saved the Gemma result from being misread. Gemma's recurring issue was mostly citation pattern weakness, not fabrication. That is still important, but it is a different product problem.

A note on the fast MoE sibling

The Gemma 4 26B MoE result deserves its own note because it is tempting to overread it. It ran the full 6-prompt x N=10 sweep in about 4 minutes, 5-14x faster than the other local models on the same rig. It also had 0 fabrications.

But the substance profile was weaker. On a tight single-project prompt, it was excellent: 12/12 citation-correct. On broad multi-project drafting, it under-retrieved and bound too much narrative to too few chunks. On SQL aggregation, it deterministically failed the rollup prompt.

My conclusion: route the MoE to fast, tight-scope single-project drafting and abstention checks. Do not use it as a drop-in replacement for dense Gemma 31B on broad past-performance or aggregate-heavy prompts.

What I take from this

If I had to deploy today for this specific workload, I would use GPT-5.1 as the cloud anchor and Gemma 4 31B AWQ as the local anchor. Qwen 3.6 27B is a credible second local option, especially where exact SQL behavior matters, but it is slower and more citation-weak. I would not use GPT-5.5-Pro by default for this agent loop. I would reserve it for cases where its extra reasoning visibly buys down risk.

Caveats

The caveats are not footnotes. They are part of the result.

• Single corpus. The evaluation uses one private domain corpus, with its own document artifacts and retrieval structure.
• Substance variance is under-measured. Reliability is N=10 per prompt. Substance grading is rep=0 per cell. More rep-level substance grading would make the claim stronger.
• LLM-assisted grading. The rubric is stricter than a naive citation check, but a publishable audit should include blinded human review on a stratified sample.
• Cache control matters. GPT-5.1 costs were affected by OpenAI prompt caching. The cost ordering is robust, but exact ratios should be treated carefully.
• Local runtime configuration matters. vLLM concurrency, parser choice, quantization, and speculative decoding all affected outcomes.

The result I am comfortable sharing is narrower and stronger than a leaderboard: in this citation-heavy procurement RAG workload, a self-hosted Gemma 4 31B model was much more trustworthy than I expected, GPT-5.1 was the clean cloud default, and the most useful eval signal came from analyzing failure modes rather than ranking models by one number.

Source notes

This post summarizes internal bench journals and reports from May 2026: the initial six-way drafting bake-off, the capped N=3 publication rerun with cost forensics, the Gemma substance-check methodology failure postmortem, and the N=10 multi-prompt rebench. The public-facing claim is intentionally narrower than the internal notes: one corpus, one tool-using RAG system, one prompt family.