Viscer: compressing coding-agent sessions with a small local model

1. Introduction

A coding agent that has worked for an hour carries an enormous, mostly stale history. Every subsequent turn re-reads it. This is slow, expensive, and eventually impossible: context windows end, and long before they do, the useful state is buried in tool output.

Compression is the obvious response, but the requirements are unusual. The output is not for a human to read; it is for a machine to act on. A summary that reads well but loses a file path, an error signature, or the distinction between resolved and unresolved work is worse than useless — it is actively harmful, because the agent will act on it confidently.

This report describes what happened when we tried to build that compressor with a small local model on a single consumer GPU. We describe the selection benchmark, the training pipeline, the resulting model, and — at length — the ways our own measurement apparatus misled us first.

Contributions

1. A selection benchmark for context compression (330 runs) showing the task is beyond zero-shot capability at 2B and 4B, with a measured compression ceiling.
2. A synthetic supervision pipeline built on programmatically verified fact ledgers, including eight families of negative/fallback training cases.
3. A working artifact: Viscer-4B v0.1, 2.1× the base model's literal recall at parity speed, verified at 131K-token context, released with weights, corpus and tooling.
4. Five negative results with mechanisms (§5), offered because each cost us time that better-specified method notes would have saved.

3. Phase 1: which model, at what precision

3.1  Method

Candidates were scored on semantic fidelity (exact-literal recall against ground-truth fact ledgers), speed (prefill, time-to-first-token, decode), and memory, at four input lengths up to 131K tokens. Hard gates were frozen before results were seen: schema validity ≥99.5%, critical-literal recall ≥99%, zero critical hallucinations, ≥95% downstream retention, ≤0.5% false completion, no OOM at 131K.

3.2  Results

Zero-shot compression is infeasible. Neither size produced schema-valid output from instructions alone: 0% at 2B and 4B, across every runtime and quantization tested. The failure taxonomy: degenerate renumbered-key loops (the model emits path6, path7, path9… until the budget ends), schema invention, and budget overruns. With two worked examples, validity rose to 45–81% depending on size and quantization — enough to compare candidates, not enough to ship.

3.3  The compression ceiling

Ground-truth targets verified for complete literal retention land at median 5.2× compression (range 2.2–40.7×, n=272). Every attempt at 16× failed: 40–79 mandatory literals cannot fit in 748–1,088 tokens. The base models' apparent “efficiency” (26–130×) was lossy compression, not skill.

Methodological consequence: a single “compression budget” conflates output length with fact retention. Both must be reported together, which is why every result in §4 carries a recall number beside its compression ratio.

5. Findings: five ways our own measurement misled us

We report these in detail because each was expensive, each was settled by a specific measurement, and each generalizes beyond this project.

5.1  The recommended sampler for structured output did not fix our failure mode

Our model's degenerate output is a verbatim repetition of a JSON entry ({"command": "ctest …", "outcome": "100% tests passed"}) repeated until the cap. Structured-output practice says: never use a flat repetition penalty on JSON (it penalizes the syntax tokens a schema must repeat) — use DRY instead, which penalizes suffix repetitions while honoring sequence breakers.

DRY cannot see our loop because the loop is made of JSON punctuation, which its sequence breakers deliberately exclude. The general guidance is correct in principle and inapplicable here — and only measurement revealed that.

5.2  An evaluation metric that scored degenerate output as clean

Our repetition detector counted duplicate lines. The failure above lives on one enormous line, so it scored the degenerate output 0.00 — spotless — while the model emitted 16K tokens of the same JSON entry. We replaced it with distinct_n_ratio (unique 12-grams: degenerate 0.006–0.237, healthy 0.887–0.964) and longest_repeat_span (degenerate 240–480 chars, healthy 0–60). Every earlier degeneracy rate we had computed was an underestimate.

5.3  A data bug that masqueraded as a model defect

Our first fine-tune emitted a field named reason_for_handling where the schema requires handling + handling_reason, and we attributed it to the model — and briefly to penalty-induced syntax drift. The actual cause: the positive training targets had been generated against schema v1 while the negative examples used v2, so the model saw the required field in only ~25% of examples. Deterministic backfill of 272 targets moved v2 conformance from 0/6 to 5/5 on held-out runs.

5.4  A preference signal that was already saturated

We assumed the remaining failure was behavioural and built preference pairs (verified target vs measured runaway) — then measured the margin before training:

loss 0.039 · margin +32.3 log-odds · preference accuracy 1.0

The model already prefers the correct output by 32 log-odds. There was nothing for preference optimization to learn, which explains why three training-side attempts (DPO, ORPO, and a direct implementation) produced nothing. The loop is a locally optimal decoding path, not a preference gap — consistent with §5.1, where a decoding-time constraint fixed what training could not. Measuring the margin before building the preference infrastructure would have saved this sub-project entirely.

5.5  Retraining a runtime-coupled component in isolation damaged it

Fine-tuning dropped speculative-decoding acceptance from 0.865 to 0.658, because the draft head consumes hidden states the fine-tune shifted while its own weights were never trained. Transformers ignores those tensors on load, so we assembled the head from the checkpoint's 15 tensors and trained it (120.6M params, loss 14.25 → 0.78).

Both attempts made it dramatically worse, and the “corrected” one was worse than the buggy one — so the objective alignment was not the error. When a component is consumed by a runtime you do not control, the runtime is the specification: read its source before training, not the tensor shapes.

7. Conclusion

A 4B model with 0.81% of its parameters trained can compress coding-agent sessions to 2.1× the literal recall of its few-shot baseline, at parity speed, in 2.7 GB, locally, at contexts up to 167K tokens. Getting there required a verification-first data pipeline and a benchmark that measured the right thing five separate times after measuring the wrong thing first. The compressor's remaining limitations are specific and enumerated; the negative results are, we suspect, the part most useful to others doing similar work.

9. References

1. Raschka, S. Repetition loops in generation — debugging order for degenerate fine-tuned output. sebastianraschka.com/faq/docs/repetition-loops-generation.html
2. DRY: Don't Repeat Yourself — suffix-repetition penalty with sequence breakers. arXiv 2608.22761
3. Why Repetition Penalty Breaks JSON and Code Generation — penalty effects on structured output. dev.to/ji_ai
4. Hugging Face transformers PR #40842 — Gated DeltaNet normalization epsilon fix.
5. Unsloth PR #6850 — Qwen3.5 gated-delta fast path requiring flash-linear-attention and causal_conv1d.
6. llama.cpp b10829 — Gated DeltaNet l2_norm epsilon fix (our pinned build b11100 includes it).
7. Qwen3.5-4B / -4B-Base model cards — architecture, context length, licensing.