Same model, same GPU, 4× the context: a weekend of inference-stack dogfooding

I have an RTX 3090 sitting in a Xeon Silver 4314 box at home. I wanted to:

1. Stand up a local inference stack (vLLM nightly with all the bells: speculative decoding, FlashInfer, prefix caching).
2. Use the same model for actual coding work via OpenCode and Claude Code, pointed at the local endpoint.

What I expected: a long evening of YAML editing. What I got: three days of fighting context-length math, discovering that two inference engines on the same model can give you a 4× difference in usable context, and learning the hard way how hybrid Mamba-attention architectures are (and aren't) handled by today's inference stacks.

The tl;dr is at the bottom.

Contents

1. The rig
2. The model
3. Standing up vLLM nightly
4. Plugging in OpenCode
5. Same model on llama.cpp
6. Hybrid Mamba-attention layer accounting
7. Quantization comparison
8. Picking an engine
9. The prompt cache
10. Hybrid models and inference frameworks
11. TL;DR

1. The rig

A single 24 GB consumer GPU is a useful constraint. It's the most common "serious local LLM" setup, and most of the friction you hit is friction everyone with a 3090/4090 is going to hit.

2. The model

The model under test for the bulk of this work was Lorbus/Qwen3.6-27B-int4-AutoRound:

• 27 billion parameters
• 4-bit weights via AutoRound (Intel's gradient-based post-training quantization, weights 4-bit / activations 16-bit)
• Hybrid Mamba-attention architecture (more on this below — it ends up being the whole story)
• Native 32K context window, RoPE-extensible to 262K
• Multi-modal (image-text-to-text) at the architecture level, even though I never use vision

Weights on disk: ~17.69 GiB. On a 24 GB card, that leaves ~6 GB for everything else: CUDA graphs, activations, KV cache, multimodal warmup buffers, and (if you enable it) speculative decoding state. The KV cache is what's left over after the rest grabs its share — and "what's left over" turns out to be the entire game.

3. Standing up vLLM nightly

vLLM has been moving fast on Qwen3 / Qwen3.5 / Qwen3.6 support. Most production-feature work (multi-token prediction, FlashInfer, hybrid attention page sizing) lands in nightly first. So nightly it is:

python3.11 -m venv ~/venvs/vllm-nightly
~/venvs/vllm-nightly/bin/pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly

Got vllm 0.20.1rc1.dev51+ge48cb8518 with torch 2.11.0+cu130 and CUDA 13.0 support bundled.

The serve command I landed on, with everything cranked:

vllm serve Lorbus/Qwen3.6-27B-int4-AutoRound \
  --tensor-parallel-size 1 --quantization auto_round --dtype float16 \
  --max-model-len 32768 --gpu-memory-utilization 0.95 --max-num-seqs 1 \
  --kv-cache-dtype fp8_e5m2 --trust-remote-code \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --enable-prefix-caching --enable-chunked-prefill \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Key knobs: - fp8_e5m2 KV cache: cuts KV memory in half versus fp16 with a small accuracy hit. - MTP speculative decoding (num_speculative_tokens=3): uses the model's native multi-token-prediction head to draft 3 tokens per forward pass; the main model verifies. Acceptance rates landed at ~75% — meaning real wall-clock throughput nearly tripled what you'd expect from raw decode latency. - --reasoning-parser qwen3 + --tool-call-parser qwen3_coder: parses Qwen's <think>...</think> blocks into separate reasoning_content and structures tool calls. - --gpu-memory-utilization 0.95: vLLM's "max fraction of total VRAM I'm allowed to use." Default is 0.9. We'll come back to this.

Startup throws ~12 of these in the first three minutes, and I read every one of them eventually:

INFO  [weight_utils.py:904] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.69 GiB.
INFO  [interface.py:606]    Setting attention block size to 1600 tokens to ensure that
                            attention page size is >= mamba page size.
INFO  [interface.py:630]    Padding mamba page size by 0.25% to ensure that mamba page size
                            and attention page size are exactly equal.
WARN  [kv_cache_utils.py:1152] Add 3 padding layers, may waste at most 6.25% KV cache memory
WARN  [compilation.py:1390] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode
                            for attention backend FlashInferBackend ...; setting cudagraph_mode=PIECEWISE
INFO  [gpu_worker.py:440]   Available KV cache memory: 2.47 GiB
INFO  [kv_cache_utils.py:1710] GPU KV cache size: 42,780 tokens
INFO  [kv_cache_utils.py:1711] Maximum concurrency for 32,768 tokens per request: 1.31x

Two numbers matter here, and the relationship between them is what makes vLLM's hybrid-model handling tight on a 24 GB card.

The first is available KV cache memory: 2.47 GiB — what's left after weights, MTP draft, vision encoder, CUDA graphs, and Mamba page alignment grab their share.

The second is the GPU KV cache size of 42,780 tokens, with Maximum concurrency 1.31x for 32,768-token requests. That's the entire KV pool. At --max-model-len 32768 and --max-num-seqs 1, one request of up to 32K fits with about 30% headroom. But you can't crank --max-model-len much past ~40K without the pool refusing to allocate — that's where 1.0× concurrency runs out, which is the point at which a single request would fill the pool exactly.

So the practical envelope on this card with this config: single-request prompt + output ≤ 32,768 tokens, with no easy way to push higher than ~40K without giving something else up.

I'd come back to that envelope. Multiple times.

4. Plugging in OpenCode

I wanted to use Qwen3.6-27B as the backend for OpenCode (a Cursor-like CLI) and Claude Code (with the env-var redirect trick to point it at a local OpenAI-compatible endpoint).

OpenCode is configured by writing a custom OpenAI-compatible provider in ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "model": "vllm-xeon/Lorbus/Qwen3.6-27B-int4-AutoRound",
  "provider": {
    "vllm-xeon": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Xeon vLLM",
      "options": {
        "baseURL": "http://10.10.10.10:8000/v1",
        "apiKey": "EMPTY"
      },
      "models": {
        "Lorbus/Qwen3.6-27B-int4-AutoRound": {
          "name": "Qwen3.6 27B (AutoRound int4)",
          "limit": { "context": 17000, "output": 4096 }
        }
      }
    }
  }
}

Smoke test:

opencode run 'Reply with PONG.'

vLLM's response:

VLLMValidationError: This model's maximum context length is 32768 tokens. However, you
requested 4096 output tokens and your prompt contains at least 28673 input tokens, for
a total of at least 32769 tokens.

OpenCode's smallest valid prompt is 28,673 tokens. That's the system prompt plus tool definitions plus minimal context — before any user message. With my 4096-token output budget, that totals 32,769 tokens. The vLLM cap is 32,768. Off by exactly one token.

That's not a coincidence on the cap — --max-model-len 32768 is doing its job. But the off-by-one is suspicious enough that I tested it across other output budgets. Same vLLM, same model, same 32,768 ceiling, just different output settings in the OpenCode model config:

OpenCode sizes its prompt as max_model_len − output_budget + 1 — it lands on exactly one token over the cap regardless of how aggressive you make the output budget. There's a fence-post error somewhere in its prompt-packing logic. Lowering the output budget doesn't help because OpenCode just packs more file context into the prompt to compensate.

Even setting that aside, the deeper problem is the pool. The whole KV pool is 42,800 tokens; if I could relax the per-request validation, OpenCode's prompt would technically fit. But raising --max-model-len to 40K or 50K hits a different wall: the pool isn't big enough to support concurrent requests at that size with the alignment overhead. You can squeeze a few thousand more tokens out of the per-request cap, but you can't get to "comfortable headroom for an agentic CLI" on this card with this engine.

Claude Code, pointed at vLLM via ANTHROPIC_BASE_URL=http://10.10.10.10:8000:

prompt contains at least 30,721 input tokens

30K. Same shape of problem — total exceeds 32,768 by ~2K.

So I needed more context. Time to claw it back from somewhere.

What's actually in my 24 GB

To grow the KV pool, I had to free VRAM from somewhere else. Over the next few hours I tried:

Disabling multi-modal with --limit-mm-per-prompt '{"image":0,"video":0}'. Result: model weights dropped 17.83 → 16.96 GiB, KV cache grew 2.46 → 2.73 GiB. Pool grows from ~43K to roughly ~47K tokens — enough to bump --max-model-len to ~40K with margin.

Bumping --gpu-memory-utilization from 0.95 to 0.97/0.98. Each 0.01 increment ≈ +0.24 GiB → roughly +4K tokens of pool. The risk: less headroom for transient memory spikes during long-context attention. A 0.98 setting could put us in the low 50K-token pool range. 0.99 is reckless.

Dropping MTP speculative decoding (--speculative-config removed). Frees ~3K tokens of pool (no MTP padding, no spec buffers). Throughput drops from 89 to ~55–65 tok/s — that's the price.

Best plausible combination of all of these: a pool large enough to allow --max-model-len ~50K, single request capped accordingly. OpenCode's first prompt would fit comfortably; longer sessions might still grow past 50K.

But every tweak costs something: throughput, multimodal capability, headroom for transient spikes. And none of them get you near the kind of context budget llama.cpp delivers on the same hardware out of the box.

I started thinking about a smaller model.

5. Same model on llama.cpp

I'd been using a different quantization of the same base model with llama.cpp earlier — unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-UD-Q4_K_XL.gguf — and remembered it had felt comfortable for context. So I switched backends:

./llama.cpp/llama-server \
  --model unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-UD-Q4_K_XL.gguf \
  --alias "unsloth/Qwen3.6-27B" \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 \
  --host 0.0.0.0 --port 8001 --kv-unified \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on --fit on \
  --ctx-size 131072

Startup log:

print_info: file size   = 16.39 GiB (5.24 BPW)
load_tensors:        CUDA0 model buffer size = 16104.14 MiB
llama_kv_cache: size = 4352.00 MiB (131072 cells, 16 layers, 4/1 seqs),
                K (q8_0): 2176.00 MiB, V (q8_0): 2176.00 MiB
llama_memory_recurrent: size = 598.50 MiB (4 cells, 64 layers, 4 seqs),
                       R (f32): 22.50 MiB, S (f32): 576.00 MiB

131,072 tokens of context. On the same RTX 3090. With the same ~16 GB of model weights. 4× more usable context than vLLM gave me on the same model (3× the pool size, 4× the per-request cap I was actually able to configure).

Same model. Same hardware. Different inference engine. Four times the context, with no flag-juggling.

That's not a quantization difference. The quantizations are similar in size (16.96 vs 16.10 GiB resident). It's not a KV cache dtype difference (vLLM uses fp8_e5m2, llama.cpp uses q8_0 — both are 1 byte per element). It's something more fundamental about how the engine handles this specific architecture.

The clue is in two log lines.

6. Hybrid Mamba-attention layer accounting

Look carefully at llama.cpp's KV cache breakdown:

llama_kv_cache:           size = 4352.00 MiB (131072 cells, 16 layers, 4/1 seqs)
                                                            ^^^^^^^^^
llama_memory_recurrent:   size = 598.50 MiB (4 cells, 64 layers, 4 seqs)
                                                      ^^^^^^^^^

Two separate memory pools: - KV cache (the big one): 16 layers, 131K cells per layer - Recurrent state (the small one): 64 layers, but only 4 cells (one fixed buffer per sequence)

That's because Qwen3.6-27B is a hybrid Mamba-attention model. From the GGUF metadata:

qwen35.block_count = 64
qwen35.full_attention_interval = 4
qwen35.ssm.state_size = 128
qwen35.ssm.inner_size = 6144
qwen35.attention.head_count_kv = 4

64 total transformer blocks, but only every 4th layer is a real attention layer (16 attention layers total). The other 48 layers are State Space Models — Mamba layers.

Mamba layers don't have per-token KV cache. They maintain a fixed-size recurrent state that's the same size whether your context is 100 tokens or 100,000 tokens. That's the entire point of SSMs: linear-time, constant-memory inference.

So real per-token KV memory only grows for 16 layers out of 64. At q8_0 (1 byte per element) with 4 KV heads × 256 head dim × 16 layers × 2 (K + V) × 131,072 cells, that comes out to ~4.35 GiB. Which is exactly what llama.cpp allocated.

Plus 598 MiB of fixed-size SSM state for the other 48 layers. That number doesn't change if you bump context to 262K.

llama.cpp's allocator is hybrid-aware. It treats the two layer types differently and only allocates per-token cache where the architecture actually needs it.

vLLM is aware of the hybrid architecture too — but its handling on this specific model is less efficient. From the vLLM startup logs:

WARNING [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration
                        by default when prefix caching is enabled
INFO    [interface.py:606] Setting attention block size to 1600 tokens to ensure that
                           attention page size is >= mamba page size.
INFO    [interface.py:630] Padding mamba page size by 0.25% to ensure that mamba page size
                           and attention page size are exactly equal.
WARNING [kv_cache_utils.py:1152] Add 3 padding layers, may waste at most 6.25% KV cache memory

vLLM unifies the page sizes between attention and Mamba layers — probably to keep its block manager simple. Attention pages get padded up to match Mamba page size (1600 tokens of attention granularity). Three padding layers get added. Up to 6.25% of KV memory is wasted to alignment. Then MTP speculative decoding adds its own draft state and verification buffers on top.

The cumulative effect is dramatic. vLLM's KV pool fits ~43K tokens out of 2.47 GiB (with single-request capped at the 32K --max-model-len); llama.cpp uses 4.35 GiB for 131K tokens — over 3× more tokens per gigabyte of KV memory, because it's only allocating per-token cache for 16 of the 64 layers.

It's not that vLLM is broken. It's that vLLM's hybrid implementation is optimized for throughput (the page-aligned blocks let MTP and FlashInfer do their thing efficiently), while llama.cpp's is optimized for context length (separate pools, no alignment waste, no spec buffers).

7. Quantization comparison

While the engines were the dominant variable, the quantizations are also worth comparing — the choices each format makes about how to compress weights are different in interesting ways.

The "UD-Q4_K_XL" naming: Unsloth Dynamic Q4_K_XL — same base 4-bit K-quant family as Q4_K_M, but Unsloth runs layer-by-layer sensitivity analysis with an importance matrix (the quantize.imatrix.* keys at the bottom of the GGUF metadata). It bumps individual tensors up to q5/q6/q8 where 4-bit hurts perplexity. Result: ~0.7 BPW more than pure Q4 but noticeably better quality for almost the same size.

AutoRound takes a different approach: a single uniform 4-bit recipe with a calibration-based rounding strategy that minimizes per-layer error. Marlin kernels then provide very fast int4 GEMM, which is part of why vLLM's throughput on this format is so good.

For quality, UD-Q4_K_XL probably edges out at ~5% larger size. For throughput per watt, AutoRound wins. For context window on a 24 GB card, the GGUF format paired with llama.cpp wins by a mile, but mostly because of the engine, not the quantization.

8. Picking an engine

For interactive code assistants (OpenCode, Claude Code, anything with a 25K+ baseline prompt): llama.cpp + UD-Q4_K_XL on Qwen3.6-27B is the right combination today. You get 130K context, the model is plenty smart for editing tasks, and 50–60 tok/s feels responsive enough.

For benchmark numbers / leaderboard glory or a high-throughput chat backend with short prompts: vLLM + AutoRound INT4 + MTP is hard to beat. 89 tok/s wall-clock decode at greedy temperature, very low TTFT, FlashInfer doing efficient batched prefill.

For "I just want to use it": pick the engine based on whether your bottleneck is context or speed. They genuinely diverge on this hardware for this model class.

The benchmarked side-by-side

To compare the two engines honestly, I ran both under identical methodology — median of 5 cold-prompt runs after 2 warmup runs, greedy decode, ignore_eos: true, single active slot — and submitted both to LocalMaxxing, a community leaderboard for local-LLM throughput, so anyone with comparable hardware can stack their numbers against these.

* vLLM's KV pool fits ~43K tokens; --max-model-len 32768 is what's achievable with comfortable concurrency headroom on this card. With more aggressive flags (no MM, no MTP, higher gpu-memory-utilization) you can get the per-request cap to ~50K — still well short of llama.cpp.

Three things stand out:

• vLLM ≈ 2.3× faster on decode, ≈ 1.9× faster on prefill (TTFT). MTP speculative decoding does most of the heavy lifting — the model's native multi-token-prediction head drafts 3 tokens per forward pass at ~75% acceptance, so wall-clock throughput is roughly tripled vs the raw decode rate. llama.cpp doesn't have spec decoding for this model class, so it's just running base decode.
• llama.cpp ≈ 4× more usable context (131K vs 32K configured) and ~3× more tokens per gigabyte of KV memory — the gap we covered in Section 6.
• Both engines are remarkably consistent across short vs long shapes — vLLM only loses ~4% decode tok/s going from 512 to 4096 prompt tokens; llama.cpp loses ~2%. KV cache pressure during decode is real but small at these context sizes.

The lesson I keep relearning is that backend choice matters as much as model choice. Two engines, one model, one GPU, four times the usable context, half the throughput. Most "compare quantizations" discussions on Reddit miss this entirely; they hold the engine constant and treat the result as a property of the format. It's a property of the format × engine × architecture combination, and the third term is doing a lot of the work.

9. The prompt cache

There's one more piece that makes llama.cpp + UD-Q4_K_XL actually usable as a code-assistant backend, beyond just "fits the prompt at all."

The first turn in any opencode session against a real codebase took 60 seconds (39s prefill + 21s decode for a ~42K-token prompt). At first that felt expensive enough to be a deal-breaker — if every conversational turn took a minute, opencode would be too slow to use. But subsequent turns in the same session feel near-instant. A 50K-token prompt on turn 5 takes ~20 seconds total; the prefill drops from 39 seconds to under 2. The reason is llama.cpp's prompt cache + checkpoint system.

The mechanic is worth understanding because it explains both why this works and what to expect when it doesn't.

The math, with measured numbers

A real example from my session, captured from the llama-server logs. The user asked an architecture question that turned into a 49,883-token chat-completion request:

slot get_availabl: id 0 | task -1 | selected slot by LCP similarity,
                                    sim_best = 0.968 (> 0.100 thold), f_keep = 1.000
slot update_slots: id 0 | task 3769 | new prompt, n_ctx_slot = 131072,
                                       task.n_tokens = 49883
slot update_slots: id 0 | task 3769 | n_tokens = 48272, memory_seq_rm [48272, end)
...
prompt eval time =    1934.25 ms /  1611 tokens (1.20 ms/tok, 832.88 tok/s)
       eval time =   18562.14 ms /   574 tokens (32.34 ms/tok, 30.92 tok/s)
      total time =   20496.39 ms /  2185 tokens

Out of 49,883 prompt tokens, only 1,611 were actually prefilled. The other 48,272 were reused from cache — llama-server detected that this new prompt's prefix matched 96.8% of an existing slot's KV state, kept that state in place, and only computed K/V for the 1,611 new tokens at the tail.

68% wall-clock saving. And it's not a clever trick — it's a direct consequence of how chat completion APIs work.

Why a code-assistant prompt is 96.8% identical to the previous turn

Chat completion APIs are stateless. Opencode (and Claude Code, and Cursor, and every other agentic CLI built on this pattern) re-sends the entire conversation context with every request. So your prompt on turn N looks like:

[system prompt: ~15K tok of opencode's hardcoded instructions]   ← unchanged across turns
[tool definitions: ~5K tok of JSON schemas]                       ← unchanged across turns
[file context: ~25K tok of source code opencode injected]         ← unchanged across turns
[user msg 1] [assistant 1] [user msg 2] [assistant 2] ...         ← previous turns are immutable
[user msg N]                                                       ← only this is new

Almost all of it is byte-identical to turn N−1. Only the trailing assistant response from the previous turn (now part of history) plus the new user message changes. For a 50K-token conversation, that's typically 500–2000 new tokens out of 50K — a 96–99% prefix match.

What's actually IN the cache

This is the part that confused me until we worked through it. The prompt cache does not contain "the conversation" as text, "the files" as content, or "the system prompt." It contains the K (key) and V (value) tensors at every attention layer for every position from 0 to N.

For Qwen3.6-27B at q8_0 KV dtype: 16 attention layers × 4 KV heads × 256 head_dim × 1 byte × 2 (K + V) ≈ 16 KiB per token. For our 48,272-token cached prefix: ~775 MiB of attention KV in VRAM (plus a Mamba snapshot — see below).

Why is this reusable? Causal attention. Token N's K and V vectors depend only on tokens 1..N — never on what comes after. So if positions 1–48,272 are byte-identical between two requests, the K and V at those positions will be byte-identical too. Identical input + identical model weights = identical output. The cache from request 1 is a 100% valid drop-in for the matching prefix of request 2.

The two-tier architecture

There are actually two separate caches:

When a slot's request finishes and the slot stays idle, llama-server serializes its KV state to host RAM so the GPU compartment can be reused for unrelated work. Your 42K-token AstroTrak conversation, once idle, is a ~2.5 GiB blob sitting in host RAM (you have 251 GB, so this isn't a real constraint).

When a new request arrives, the matcher checks both pools: 1. Every live slot's current KV state's prefix similarity to the new prompt 2. Every saved prompt in the host RAM cache

It picks the highest-similarity candidate. If it's a live slot, just continue with prefill of the delta tokens (the case shown above — instant resume). If it's a host-cached prompt, deserialize it back into a free GPU slot first, then continue. The slot index is administrative — the same logical conversation can travel from slot 2 (live) → host RAM (saved) → slot 0 (re-loaded) over its lifetime.

Checkpoints: why they exist and why they're 149.626 MiB

This is the genuinely novel piece for hybrid Mamba-attention models. Look at this log line during prefill:

slot create_check: id 2 | task 75 | created context checkpoint 3 of 32
   (pos_min = 30704, pos_max = 30704, n_tokens = 30705, size = 149.626 MiB)

Every ~8K tokens during prefill, llama-server takes a snapshot. They serve two purposes:

Resilience. A 42K-token prefill takes 39 seconds. Without checkpoints, a server restart mid-prefill loses everything; with them, the worst case is ~7.6 seconds of lost work.

Partial cache hits. This is the bigger one. If your new prompt shares the first 30,000 tokens with a cached one but diverges after, the matcher can resume from the deepest matching checkpoint (say position 30,705) and only prefill the remaining ~10K tokens. Without checkpoints, you'd either need an exact full-prompt match or a fresh prefill.

The size — exactly 149.626 MiB — wasn't obvious to me at first. It turns out to be one slot's worth of Mamba recurrent state:

llama_memory_recurrent: size = 598.50 MiB (4 cells, 64 layers, 4 seqs)
                        598.50 / 4 seqs = 149.625 MiB per slot ← exactly the checkpoint size

The reason this is a separate buffer (and not just part of the attention KV cache) gets to the heart of why hybrid models are trickier to checkpoint than pure transformers. Mamba state is recurrent — it evolves through time, with each token's state depending on all previous tokens via state-space recurrence. You can't reconstruct the Mamba state at position N by just re-prefilling from a nearby checkpoint; you'd have to re-run from token 0. So the checkpoint has to physically save the Mamba state at that position. The attention K/V is recoverable by re-prefilling the delta tokens (since attention is causal); the Mamba state isn't.

That's why each checkpoint is exactly the size of one slot's full Mamba state buffer (149.626 MiB). The attention K/V "snapshot" is implicit in the in-VRAM state up to that point; the explicit serialized checkpoint is the Mamba state, the thing that can't be recomputed.

How the matcher picks a slot

Three different log signatures show three slot-picking outcomes from my session:

selected slot by LCP similarity, sim_best = 0.968    ← cache hit on a live in-VRAM slot
selected slot by LRU, t_last = -1                    ← no useful match, fresh prefill
(implicit: deserialize from host RAM cache, then    ← cache hit but state was paged out
 LCP-match check passes after restoration)

The LCP-match threshold is > 0.100 — just 10%. That's a very low bar, so multi-turn conversations almost always trigger reuse. The matcher isn't trying to be clever; it's just looking for any shared prefix worth keeping.

What this means for the user experience

Three practical implications I noticed:

1. The first turn in any new opencode session is the slowest. You're paying for cold-start prefill of the entire system prompt + tools + file context (~30K tokens at 1080 tok/s ≈ 30 s). After that, every subsequent turn touches mostly cached state.

2. Switching projects mid-session is expensive. If opencode re-injects file context for a new project, the file-context block early in the prompt changes. That invalidates the cache from there onward — and since the file block sits before the conversation history, you get a near-total cache miss.

3. Compaction doesn't bust the cache. Compaction runs in a separate parallel slot with a different prompt structure ("summarize this conversation"). Your main conversation's slot sits idle and untouched during compaction, then resumes normally on the next turn. (I confirmed this from the slot IDs in the logs: compaction landed in slot 1 while my main conversation lived in slot 0.)

Tuning the cache

If you ever bounce between many projects and want more conversations cached simultaneously, the lever is --cache-ram (host RAM, not VRAM). My defaults are 8 GiB, which holds about 2 long-context (~50K-token) conversations before LRU eviction. With 251 GB of host RAM available, bumping to --cache-ram 32768 would let me hold a dozen+ active sessions across projects with no VRAM cost. The cache lives in CPU RAM and only pages back to GPU on resume.

The prompt cache landed in llama.cpp PR #16391 — worth reading for the implementation details.

10. Hybrid models and inference frameworks

Mamba and other SSM-style layers have been creeping into production model architectures for two years now. Mistral's Codestral Mamba, IBM's Granite-3.x with hybrid blocks, Qwen3.5/3.6's hybrid arrangement, Falcon Mamba — they're not exotic anymore.

But inference frameworks are still catching up to handling them well. vLLM clearly recognizes Qwen3.5/3.6's hybrid nature (the warnings say so), but its block manager is built around uniform attention pages. Padding overhead and the interaction with speculative decoding eats your KV cache. llama.cpp has had separate llama_memory_recurrent plumbing for a while (it goes back to their original Mamba support) and so this just works.

Expect this to even out as vLLM's hybrid support matures. Until then, the rule is: if you're running a hybrid Mamba-attention model on consumer-class VRAM and you care about context length, llama.cpp will give you more headroom.

TL;DR

• vLLM nightly on Qwen3.6-27B int4 gives a ~43K-token KV pool out of 24 GB; with --max-model-len 32768, each request caps at 32K. Not enough for OpenCode (28K + 4K = 32,769) or Claude Code (~35K). Pushing to ~50K costs MTP, multimodal, and headroom.
• Same model on llama.cpp gives 131K usable context out of the box. ~4× more.
• Side-by-side on LocalMaxxing (cold-prompt, cache-busted): vLLM ~89 tok/s decode vs llama.cpp ~38 tok/s. vLLM is ~2.3× faster on decode and ~1.9× faster on prefill; llama.cpp gets ~4× more usable context. Two genuinely different operating points on the same hardware.
• The cause isn't quantization — llama.cpp's allocator is hybrid-aware. Qwen3.6 has 64 layers but only 16 are real attention; the other 48 are Mamba (fixed-size recurrent state). vLLM unifies page sizes for block-manager simplicity, pays a ~6.25% alignment tax, and compounds it with MTP draft state.
• llama.cpp's prompt cache + checkpoint system turns long-context conversations from "expensive" into "fast after turn 1." Measured 96.8% prefix match → 44 seconds saved on a single real opencode turn. Checkpoints exist because Mamba state can't be reconstructed by re-prefilling.
• Backend choice matters as much as model choice. Pick based on whether your bottleneck is context or speed.