890 bytes per token — and not one needle test

DeepSeek’s V4.1-Flash tech report documents the memory system in full and quietly shows where the evidence stops; Tencent’s Apache-2.0 WeVisDoc and Sarvam’s voice-agent regression tests round out the desks.

AI Engineering
Architecture
Open Weights
China
India
US

DeepSeek’s 340-author V4.1-Flash report makes the KV-compression architecture precise — constant-cost indexing, MXFP4 caches, a 196B n-gram memory — while shipping no retrieval-at-distance eval; Tencent enters a document-parsing race with Apache-2.0 WeVisDoc. Issue 6, coverage window 17–20 September 2026.

Author

the dimsum editorial

Published

September 20, 2026

Week of September 17–20, 2026 · Issue 6.1 (corrected 2026-09-27 — see Corrections)

The Week in One Sentence

DeepSeek’s 340-author report documented its KV compression to the byte while shipping no retrieval-at-distance evaluation at all, Tencent’s WeVisDoc made Chinese document parsing a two-horse open-weights race, the voice-agent stack gained version-pinned regression tests in India and per-hour transcription economics in the US, and Google put managed-agent builders on an 18-day migration clock.

The Big Story: the V4.1-Flash report is exact about memory and silent about retrieval

What happened. (China.) On September 17, DeepSeek published “DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression” (arXiv:2609.19969, ~340 authors) — the tech report behind Issue 4’s Big Story model, one week after the weights. (Labelled update on Issue 4 coverage; weights Sep 10, report Sep 17, both dates kept distinct.) We read it in full. The headline architecture claims from the model card are now paper-documented — and the paper adds the precision the card lacked.

Under the hood — now with numbers. CSA2’s static assignment is concrete: encoder layers 3–20 run in three groups of six (one Full layer computes fresh KV and indices, five Reuse layers take them as-is); the decoder’s five groups of four mix one Full group with Reindex groups that rescore a shared candidate pool. That pool is the genuinely new systems idea — a Hierarchical Sparse Indexer: the first Full decoder layer scores the entire context, picks Top-512, and expands to a blockwise pool of 16,384 candidates; every later indexer pass searches only the pool, making deeper indexing cost constant in context length. KV storage is MXFP4 (E2M1 values, one E4M3 scale per 16 channels, no second-level scale — justified by an RMSNorm bound; quantized after RoPE, QAT during post-training) with SWA KV kept in FP8, for a stated 890 bytes per token in HBM — roughly a quarter of V4-Flash and ~437× smaller per token than V1. The paper states 890 but does not show the arithmetic (open item). SWA Bounded Replay (window 128) moves sliding-window KV out of the persistent cache into a host-DRAM pool with minutes-scale TTL while global KV keeps ≥72-hour lifetime — persistent footprint ≈ 1/8 of baseline (2× eviction × 4× compression), and decoder-side replay nearly halves prefill compute; prefill complexity drops from O(NL) to ≈O(NL/2). And the Engram — the 196B “conditional memory” — is now specified: n-gram {2,3,4} hash tables (~16M entries, FP8, momentum updates plus Sinkhorn balancing), deterministic addressing that enables background RDMA prefetch from host memory. Readers of Issue 1 will recognize the family: Qwen’s Flash-Next shipped a 51B n-gram embedding table in August. Decoupling memorization from computation is now a convergent design at two vendors, at ~4× the parameter budget.

What the evidence says — including what isn’t there. Against V4-Pro-Base in DeepSeek’s internal framework, V4.1-Flash matches or wins on MMLU-Pro (74.1 vs 73.5), HumanEval, BigCodeBench, GSM8K with a third of the total and a quarter of the activated parameters — and the paper concedes MATH (61.1 vs 64.5), MultiLoKo (45.5 vs 50.9), and SimpleQA-Verified (42.3 vs 55.2). Then the load-bearing absence: the report’s evaluation sections contain exactly one distance-sensitive benchmark — LongBench-V2 at 45.2, versus V4-Pro’s 51.5. No RULER, no needle-in-haystack, no MRCR appear in the sections we read. A 1M-context model whose flagship feature is cache compression ships a report where the sole long-context table has it losing to its own sibling by 6.3 points. The agent figures claim frontier parity on Terminal-Bench 2.1 and DeepSWE while conceding the Terminal-Bench 4.0 science gap, and a “95% of real-world tasks” completion line that is marketing-grade until independently tested.

Why this matters. Two issues ago we wrote that the memory-systems thesis had “shipped as a rate card.” The report completes that sentence: the memory system is real, reproducible on paper, and extreme — constant-cost indexing plus 890-byte KV is the most aggressive deployed design we cover. But the evidence structure is now also clear: the compression axis is documented to the byte while the capability-at-distance axis is documented not at all. For agent fleets that re-read long prefixes, that trade may be exactly right — cache hits at $0.003/M (Issue 4) plus constant-cost indexing is an economics argument, not a retrieval argument. Buyers should know which one they’re being sold.

What I would test. The paper left the test on the table: RULER and needle suites at 128K/512K/1M against V4-Pro and V4-Flash, plus a cache-hit-rate trace on a repeating agent workload to check the ≥72-hour global-KV lifetime claim against real eviction behavior. If retrieval holds to 512K, this is the best cost-per-context-mile in the market; if it falls off a cliff past 128K, the 1M window is a prefill story.

Elsewhere in AI Engineering

Document parsing became a two-horse race in China (China) — Worth Testing

What happened. (China.) Tencent released WeVisDoc (4B, created September 16; paper arXiv:2609.20423) — an end-to-end page-image-to-Markdown parser fine-tuned from Qwen3-VL-4B-Instruct, with LaTeX formulas and HTML tables, served via vLLM ≥ 0.11.1. Apache-2.0 (license field read verbatim), with a 2B sibling, code, project page, and demo. Vendor benchmarks: OmniDocBench v1.6 overall 95.38, “best among compared end-to-end parsers,” and first in all four PureDocBench settings including the real-degraded track (69.08). Z.ai’s GLM-OCR (Issue 5’s ledger: MIT, 94.62) is the incumbent — but on OmniDocBench V1.5, so the 95.38-vs-94.62 comparison is not apples-to-apples and we won’t make it.

Why it matters. Boring, load-bearing, and now contested: enterprise document ingestion is the workload where small open models actually deploy, and within one week China’s two largest open-weight ecosystems each shipped a permissively-licensed specialist (MIT and Apache-2.0 — the licensing rule from Issue 5 holding again). WeVisDoc’s degraded-document track is the differentiator to watch: 69.08 on real-world scans is where parsers actually fail in production.

Sarvam shipped regression tests for voice agents (India) — Worth Reading

What happened. (India.) September 17, per the docs changelog: voice agents on the platform can now be regression-tested. You write test cases describing key conversations; a simulated user runs against a committed agent version; and “an AI judge grades each expected behavior” Pass or Fail. Suites support shared variables and global guardrails, “runs execute each case up to 20 times to expose flakiness,” every execution returns a full transcript with per-behavior reasoning, and the whole suite surface is manageable over REST via the Tests API.

Why this matters. The two engineering choices here are the signal: version-pinned evaluation (tests run against a committed agent version, so the artifact you deploy and the artifact you test are the same thing) and flakiness exposure via 20× repeats — treating agent behavior as a stochastic system rather than a deterministic function, which matches how these systems actually fail in production. The AI-judge layer is self-reported for now: there is no independent view into its grading, and we label it accordingly. “Agents need CI” is the same thesis the China desk’s items keep circling; this window it shipped as a billed platform feature rather than an open harness.

Go deeper: changelog entry

GLM-5.3 gets an Indian rate card (India) — Worth Reading

What happened. (Labelled update on Issue 5’s Sarvam-platform coverage. Dating note: the changelog entry is dated September 16 — the day Issue 5 shipped — and postdated that morning’s desk read; caught and covered here.) Sarvam’s platform batch added glm5.3 as an open-weight chat model on /v2/chat/completions — 1,048,576-token context, tool calling, visible reasoning, beta access on request — alongside a hosted MCP server for voice agents at mcp.sarvam.ai/voice-agents (OAuth 2.1 via Sarvam SSO, nothing to install; Claude, Cursor, and VS Code clients can build, deploy, test, and analyze agents) and a REST Boards API. The pricing page, re-read today, lists glm5.3 at ₹126 / ₹23.4 / ₹396 per 1M tokens (input / cached input / output; Beta), with reasoning tokens billed as output.

Why this matters. The cross-region story is the pricing row: an Indian platform is now a distribution surface for the Chinese open-weight flagships this publication covers — the same page’s billing note covers DeepSeek V4 Flash as well — with rupee rate cards aimed at Indian enterprise buyers. The license angle is consistent with Issue 4’s artifact read: GLM-5.3’s commercial gate applies to MaaS operators above $10B aggregate revenue, far above this deployment’s scale — our reading of the published terms, not a legal opinion. For engineers, the most reusable piece is the MCP server: agent lifecycle operations over a standard protocol instead of a vendor dashboard.

Go deeper: changelog · pricing

Grok Voice Transcribe 2.0 doubled accuracy at the same price (US) — Worth Testing

What happened. September 18, xAI shipped Grok Voice Transcribe 2.0 — speech-to-text built on the audio foundation model behind Grok Voice, claiming “twice as accurate as Grok Voice Transcribe 1.0, at the same price”: $0.10/hour batch, $0.20/hour streaming, with speaker diarization, word-level timestamps, and up to 100 key-term biases per request included. The prices are confirmed on xAI’s rate card, read today (“Speech to Text — $0.10/hr (REST), $0.20/hr (Streaming)”). xAI says it already powers Atlassian Loom transcription and voice features in Tesla vehicles — vendor-claimed, as is everything in the performance section. Grok Voice Transcribe 1.0 “will be deprecated soon” — no date given, with version pinning available during the transition.

Technical signal. The claimed gains concentrate where production ASR actually fails: short-phrase multilingual word error rate from 20.6% to 6.8%, “on telephony it leads every model we tested” (internal set), dozens of languages with automatic detection and mid-recording language switching in a single pass — plus smart turn detection for voice agents, up to 8 multichannel inputs, filler-word removal, and text formatting. The Artificial Analysis “first for accuracy among 32 streaming models” rank is vendor-cited, not our read of the instrument. Existing Speech-to-Text integrations get the upgrade “with no code changes.”

Why it matters. Per-hour audio pricing with diarization included is the billing shape voice-agent builders actually want, and holding it while doubling claimed accuracy resets the cheap end of the transcription market — worth a bake-off against your current STT vendor on your own telephony audio. Operationally, two flags from the same day-of rate-card read: the undated “deprecated soon” on 1.0 means anyone standardizing on 2.0 should pin deliberately and watch for a dated shutdown; and X Search pricing changes September 21, 12:00 PM PT — $5 per 1k posts fetched plus $10 per 1k user profiles fetched, replacing $5 per 1k calls — a dated, breaking cost change for X-data pipelines.

Go deeper: announcement · pricing

Google swapped its managed-agent preview: schema breaks, October 5 shutdown (US) — Worth Watching

What happened. September 17, per the Gemini API changelog: antigravity-preview-09-2026 — the date-stamped managed-agent preview in the Gemini API — launched and replaced antigravity-preview-05-2026, which stops on October 5. The swap is not drop-in: tool schemas for local execution change shape — parameters move from snake_case to PascalCase, and file edits switch to line-range replacements instead of full rewrites.

Technical signal. Two breaks in one version bump: the naming-convention change breaks schema-validated tool code, and line-range edits change what diff-based harnesses see from the agent. The version names encode the discipline — date-stamped previews with scheduled sunsets.

Why it matters. If you run Google’s managed agent via the API, this is a forced migration with code changes on an 18-day clock (launch to old-version shutdown) — the sharpest example yet of the preview-cadence risk this desk keeps flagging. Budget migration time into any build on a preview identifier, and pin deliberately. Same vendor, same page family, same window: the deprecations page (updated September 17) re-confirms gemini-omni-flash-preview’s shutdown for September 30, replacement gemini-omni-1.1-flash — ten days out at publication, and the Live-API migration reminder stands.

Go deeper: changelog · deprecations

No Japan items this issue: the window’s Japan-lab output was organizational news (Sakana’s Frontier Intelligence Group launch), product refreshes without engineering detail, and non-LLM tooling patches — none cleared the editorial bar. This is the second consecutive issue without Japan coverage (Issue 5’s gap was a missed desk run; this one is a genuinely thin window). The desk’s continuity checks re-verified unchanged: Fugu’s model pool stays unenumerated, its benchmark numbers stay unreproduced by any third party, and its harness repo still ships no license. Japan coverage returns when the labs ship artifacts.

Research Radar

The window’s China research story leads the issue as the Big Story — the V4.1-Flash report, read in full (paper-notes in the research packet). No second China paper met the read-beyond-abstract bar inside the window; SAS follow-ups (license, reproduction) remain open items rather than items.

Repo Radar

tencent/WeVisDoc-4B (China)

  • What’s actually in it: 4.83B-param BF16 checkpoint (~9.65 GB, sharded), 2B sibling, demo Space, and links out to code and project page. Inspection depth: model card only — repo tree not opened this cycle.
  • What you can run: vLLM ≥ 0.11.1 or local Transformers; PNG/JPEG/WebP page images in, structured Markdown out. Three community quantizations already exist.
  • What’s missing: benchmark versions are the trap — the card’s v1.6 numbers invite a false comparison with GLM-OCR’s V1.5 figures; and no degradation-suite numbers beyond the vendor’s own PureDocBench.
  • Why open it: a permissively-licensed, vLLM-native on-ramp to document-parsing pipelines — with GLM-OCR remaining the smaller MIT option if footprint matters more than degraded-document robustness.

My Read on the Week

(Synthesis and interpretation — not reported fact. Written after a cross-item pass spanning all desks’ items; the Japan desk published none this window — see the note closing Elsewhere.)

1. Documented absence became a first-class result. Three vendors in two issues have now shipped their most honest artifact as a gap: Tencent’s SAS with its own RULER failure tables (Issue 5), DeepSeek’s report with exactly one distance-sensitive benchmark — on which it loses to its own sibling by 6.3 points — and Sarvam’s test harness running each case up to twenty times precisely because flaky failure is the expected agent behavior. When the compression axis is documented to the byte and the retrieval axis is documented not at all (the Big Story), “what’s missing” stops being a critique applied from outside; it is a section vendors themselves are writing. The reader’s job shifts to noticing which absences are chosen and which are avoided.

2. Voice agents got an engineering layer on two continents. Sarvam’s regression tests pin a committed agent version and expose flakiness with repeats (India); Grok Voice Transcribe 2.0 prices the audio channel per hour — $0.10 batch, $0.20 streaming, diarization included — with smart turn detection aimed at agents (US); and Gemini’s 3.8 Live models, GA since September 15 (reproduced from the changelog this cycle), sit underneath as the low-latency component layer — with the audio channel itself now priced two different ways, per hour at xAI and per token at Google, a spread voice-stack architects get to arbitrage. Testing infrastructure, per-hour economics, and standard components landing in one window is what a stack looks like just before it stops being novel. The unreproduced part is exactly where you would guess: the “first among 32 streaming models” rank is the vendor citing an instrument we have not read.

3. Deprecation discipline is now a differentiator you can plan around — or can’t. Google date-stamps its previews and gives visible runway: 18 days for the antigravity swap, ten days to the gemini-omni-flash-preview shutdown on September 30. DeepSeek rerouted retired API names to their successors with billing unchanged (Issue 4). xAI says Grok Voice Transcribe 1.0 “will be deprecated soon” — no date. The dated shutdowns are engineerable: calendar, migrate, pin. The undated one is a liability you can only watch. And beneath the lifecycle story, distribution keeps widening — GLM-5.3 now has a rupee rate card on an Indian platform, a Chinese open-weight flagship carrying Indian retail pricing a week after its license gate was read down to a $10B threshold.

What I’d Watch Next

  1. Independent long-context tests of V4.1-Flash — the report’s missing table is now the single most valuable community contribution available; watch for the first RULER-at-1M run.
  2. Two dated Google shutdowns inside fifteen days — gemini-omni-flash-preview stops September 30 and antigravity-preview-05-2026 stops October 5; migrate on the calendar, not on hope, and pin deliberately.
  3. The V4.1 family’s larger siblings — Flash is “the smallest”; with the report public, the siblings’ cache economics are the next disclosures to watch for.

One Thing I’d Experiment With This Week

Cost the prefix, not the promise. V4.1-Flash’s design bets that agent workloads re-read long prefixes — global KV cached ≥72 hours at $0.003–0.006/M cache-hit with constant-cost indexing. Your workload either fits that shape or doesn’t, and the experiment is cheap because the bet is API-shaped.

  • Setup: one agent task family with a large stable context (a repo, a document set, a tool manual) and incremental queries over it; three models — V4.1-Flash, GLM-5.3-Flash, V4-Pro — same harness, same turn budget.
  • Conditions: two arms per model — queries engineered to reuse the exact cached prefix (cache-friendly) vs. small prefix perturbations each turn (cache-hostile, the realistic worst case).
  • Measure: cache-hit rate, billed split per turn, cost per completed task; quality delta between arms (does prefix perturbation degrade outputs, or only price?); and, for anyone self-hosting, realized KV bytes/token against the paper’s 890. Decision output: the break-even perturbation rate at which V4.1-Flash stops being the cheapest option — a number your CI can warn on, because cache-hostile agent code silently pays 50× on the miss path.

Corrections

Issue 6.0 → 6.1, 2026-09-27. One correction, filed from the next issue’s China-desk continuity pass.

  1. [China] The V4.1-Flash long-context “evidence vacuum” was true of the paper, not of the card. The Big Story stated the report’s evaluation sections contained no distance-sensitive test and framed the model’s distance behavior as unmeasured (“the report is exact about memory and silent about retrieval”). But the model card itself — unchanged since September 10 — carried community-submitted ExtractBench results (HF card): 94.44 short / 81.49 medium / 22.23 long, plus ParseBench rows (layout 28.61, charts 20.46), submitted via eval PRs #44/#45 around mid-September. The distance-weakness datapoint therefore existed before this issue published; we described a vacuum while half the evidence sat on the card. The paper claim was accurate as written; the framing of the model as unmeasured at distance was not. Updated in Issue 8. Root cause: the continuity sweep checked the paper, the discussions, and the org page, but did not re-sweep the card’s eval tables, which had been read once at launch (September 13) and treated as stable.

All vendor-reported figures are labeled as such; if independent evidence doesn’t exist yet, that’s stated rather than papered over. This issue’s desk ledgers live under research/2026-09-20/.

Citation

BibTeX citation:
@online{dimsum_editorial2026,
  author = {dimsum editorial, the},
  title = {890 Bytes Per Token — and Not One Needle Test},
  date = {2026-09-20},
  url = {https://news.llm101.in/issues/2026-09-20/},
  langid = {en}
}
For attribution, please cite this work as:
dimsum editorial, the. 2026. “890 Bytes Per Token — and Not One Needle Test.” September 20. https://news.llm101.in/issues/2026-09-20/.