The collapse graduated — and one benchmark evaporated

ExtractBench turns out to be real, enterprise-grade, and its own authors found the short-good/long-bad shape our ledger traced across two vendors — while the other instrument we were watching, MarathonMemBench, does not exist.

AI Engineering
Evaluation
China
India
US

Provenance work closes both open instruments: ExtractBench is a LlamaIndex benchmark (4,869 pages, 370 documents) whose authors independently corroborate the long-document extraction collapse — with the caveat that the maintainer’s product ranks first — and MarathonMemBench, cited in a V4.1-Flash community claim, is verified non-existent. Plus: Sarvam consolidates its chat line to one hosted model, and Gemini 4 Argon’s wider rollout lands with a dated price clock and a below-Opus independent debut. Issue 13, coverage window 12–14 October 2026.

Author

the dimsum editorial

Published

October 14, 2026

Week of October 12–14, 2026 · Issue 13

The Week in One Sentence

A quiet window for launches was a loud one for instruments: the benchmark behind the extraction collapse graduated to real-and-enterprise-grade — its own authors corroborating the short-good/long-bad shape this ledger traced across two vendors — while the other instrument we were watching, MarathonMemBench, dissolved on inspection into a phantom; Sarvam consolidated its chat line to a single hosted model, leaving its open weights self-host-only; and Gemini 4 Argon reached wider rollout claiming SOTA on vendor benchmarks with a price clock ticking under its intro rate — and a below-Opus debut on the only independent index.

The Big Story: two instruments, two verdicts

What happened. (China; labelled update on Issues 1, 6, and 8.) The Oct 11 ledger left two instrument questions open. Both closed this cycle, in opposite directions. First: ExtractBench is real. arXiv:2607.29677 (Zhang, Lyjak, Stewart, Li, Suo; July 31, revised August 5) specifies it fully: 4,869 pages across 370 enterprise documents, 8 business domains, 67 document types, scored on order-insensitive value F1 plus word- and page-level grounding F1 plus measured cost — claimed as the first benchmark to combine all three — with dataset and evaluation code maintained by LlamaIndex (run-llama/ExtractBench). Second: MarathonMemBench does not exist. Searched across papers, repositories, and blogs: no benchmark by that name appears anywhere. The community claim that cited it — V4.1-Flash “manages agent memory 10× less efficiently than V4 Flash (Hermes, MarathonMemBench)” — quotes a phantom instrument and is unattributable.

What the evidence says. The ExtractBench paper’s own finding, from its abstract: commercial VLMs do well on short documents but truncate long record lists; coding agents hold accuracy at higher cost. That is precisely the shape our ledger recorded on two vendors’ cards — GLM-5.3-Flash at 96.3 → 27.83 short-to-long (Issue 1) and V4.1-Flash at 94.44 → 22.23 (Issue 8) — measured by the community through eval pull requests on both cards (the GLM rows now formalized via PRs #29/#49 in this window’s card edits). Three independent-ish sources, one failure signature. Two caveats keep it honest: every score is self-reported or community-submitted rather than audited, and the benchmark’s top ranker is LlamaExtract Agentic Plus — the maintainer’s own product. That adjacency weakens the leaderboard column while, notably, strengthening the collapse finding: a maintainer has little incentive to publish a failure mode its own product avoids.

Why this matters. For buyers, the practical rule from Issue 8 now stands on an instrument: test schema-guided extraction at your real document lengths before trusting any long-context claim, because the one enterprise-grade benchmark we’ve located for exactly that task reports the truncation failure in its authors’ own abstract. For the field, the MarathonMemBench verdict is the sharper lesson: the verification layer we have celebrated — community PRs, card-level peer review — has an underbelly, and it is fabricated instrument names. A claim citing a benchmark nobody can locate is not a weak claim; it is very likely a made-up one. Our ledger’s new standing rule: agent-memory claims citing unidentifiable benchmarks are treated as fabricated until the instrument is named and found (the real ones to use: AMB, MemoryArena, MEMTRACK, LongMemEval).

What I would test. Already written, twice (Issues 8 and 12): the short/medium/long extraction sweep on your own documents. What’s new to test is the meta-level: take any benchmark name in a vendor deck or community thread you’re about to act on, and spend the ten minutes to find the paper, the repo, and the maintainer. This cycle that check dissolved one instrument and graduated another — a good week’s work for ten minutes.

Elsewhere in AI Engineering

Zhenwu V900 firms in press — and still has no spec sheet (China) — Worth Watching

What happened. (Labelled update on Issue 7.) Tom’s Hardware (September 23) adds the missing structure: the V900 supports up to a 500,000-chip supercluster, a 10T-parameter Qwen model sits on that cluster’s roadmap, and launch is slated for Q1 2027 — with the M890 baseline now quantified (144 GB memory, 800 GB/s interconnect, so the V900’s press specs are +50% on both axes). A companion “PR” chip (128 GB, 1.6 TB/s, 1 PFLOPS FP8) is also reported.

Why it matters. Five press cycles in, the picture has converged everywhere except where it counts: no Alibaba primary spec sheet exists — the HIGH-cost flag enters its fifth cycle, and the numbers stay labeled press-reported-unconfirmed. What the press convergence does establish is the ambition shape: half-a-million-chip clusters and 10T-parameter models are the planning horizon, which is exactly the regime where the memory-bandwidth-per-token arithmetic this desk has tracked since Issue 1 stops being an argument and becomes a procurement constraint.

SAS answers its first question (China) — Worth Reading

What happened. The first community issue on the SAS repo (October 6): can the 64-token block size change at inference? A collaborator answered substantively — retrain the AttnGate at the new block size with the base model frozen, because block size “alters the pooled key representations and block-selection granularity,” so an inference-only change isn’t guaranteed to preserve accuracy.

Why it matters. Two readings. The answer itself confirms the design coupling our paper-notes recorded (selection granularity is trained, not configured — the honest boundary of the method). And the existence of a responsive maintainer, eight checks into a repo with no license, says the project is alive while its legal status stays undefined — now the longest-running license flag in the ledger. A direct license question to that maintainer is the obvious next move.

The October-12 card-edit cluster (China) — Worth Noting

All three load-bearing cards were touched the same day — V4.1-Flash at 09:45 UTC, then the GLM-5.3 pair 28 seconds apart at 12:59–13:00 — and none shows visible content change: V4.1-Flash’s eval tables are unchanged (with a cosmetic contradiction standing: card text still says no Jinja chat template while one ships in the metadata), IndexPool remains absent from the GLM-Flash card on the tenth consecutive check, and Youtu-Parsing-Omni is unchanged since release while its adoption nearly triples along (966 downloads, 88 likes, two community quantizations, five Spaces). The verification-through-PRs layer, meanwhile, now runs on both vendors’ cards — ExtractBench and ParseBench rows are formalized on Z.ai’s too.

Voice cloning graduates into Sarvam’s SDKs (India) — Worth Reading

What happened. (India. Labelled update — the voice-cloning thread’s sixth dated event since September 26.) October 12, per the docs changelog: instant voice cloning is now first-class in the Python and TypeScript SDKs (beta), with “the same call shape as Bulbul Text-to-Speech” — one-shot via voice_cloning.convert, HTTP streaming via convert_stream, WebSocket sessions in a “configure → convert → flush” pattern — plus voice-management helpers (create/list/get/delete) on the same client under /voices/*. Requires the text_to_speech_voice_cloning capability and sarvamai ≥ 0.1.37 (Python) / ≥ 1.1.12 (TypeScript); older alphas lack the streaming APIs.

Why this matters. The arc reads differently now: the API shipped September 26; the prompt-layer skill was dropped in live verification September 28 (Issue 9); and everything since has been code-layer investment — streaming endpoints (October 1), Bulbul persona tooling (October 6), now SDK integration at the same ergonomics as plain TTS. Issue 9’s two readings resolve toward consolidation, not retreat: cloning as a typed client method rather than a scaffolded example. The standing flag stands, though — no entry in this thread has documented consent verification for cloned voices, while AI4Bharat’s open alternative carries explicit card-level consent terms (Issue 11). The call-shape parity with Bulbul also makes the migration cost between synthetic and cloned voice paths near zero — convenient, and precisely why the consent gap deserves closing.

Go deeper: changelog

Sarvam’s chat line consolidated to one model — the carried watch, resolved (India) — Worth Reading

What happened. (India. Carried-item resolution with dating disclosed: the underlying entries are pre-window — August 28 and, mediated, June 5 — found this cycle while closing the sarvam-m watch opened in Issue 9.) The changelog’s August 28 entry states that sarvam-30b “has been fully deprecated” and that Sarvam-105B “is now the only supported Sarvam chat model.” The sarvam-m deprecation itself is dated June 5 on the changelog page as surfaced by search — the machine-readable feed truncates before June, so that date is mediated and labeled as such — with sarvam-m no longer accepted by the Chat Completions API and sarvam-30b (64K) as the then-supported path.

Why this matters. Issue 9 read the direction of travel from example-code wording; the primary now documents the destination: one supported chat model, the hosted 105B. The operational facts for production users: the API rejects both open-weights chat models’ names (sarvam-m since June, sarvam-30b since August 28), while the weights themselves remain downloadable on Hugging Face — self-hosting is the only serving path for them, and the example migration Issue 9 observed (sarvam-m → sarvam-105b) was the vendor walking through its own consolidation. We found no removal date for the weights anywhere: both entries say deprecated, not removed.

Go deeper: changelog

Gemini 4 Argon reached wider rollout — with a dated price clock (US) — Worth Testing

What happened. Google’s next-generation frontier model, Gemini 4 Argon, was announced September 30 — an evening launch this desk’s morning runs missed, uncovered since; both dates stated. The in-window event is October 13’s wider rollout: Pro-tier users began receiving it (availability inconsistent, capacity-constrained, also live on OpenRouter — community-reported, 1,153-point thread). The announcement’s headline spec: a 1M-token output limit (“up from the previous 64K”), with vendor-claimed workloads to match — autonomous C/C++→Rust migrations including “800K+ lines” of the Fuchsia Zircon kernel, fleet memory optimization “freeing 300+ TiB,” a quantum-subroutine claim of beating a published baseline by 40%, and autonomous vulnerability find-validate-patch released without cyber guardrails for trusted defenders (Fairwind Program, plus the US government’s voluntary pre-release process).

Technical signal. Vendor benchmarks, labeled as such: DeepSWE v1.1 at 77.9, stated SOTA — the same benchmark-version lane where this publication’s ledger holds DeepSeek’s 74.2 and GPT-5.6 Sol’s 72.7, all self-run; Zapier AutomationBench #1 at 51.3; LVBench 91.7; CWE-bench tied-first at 68. The rate card, read today: gemini-4-argon at $2 input / $10 output (thinking included) per million through November 30 — doubling to $4/$20 on December 1; context cache $0.10→$0.20 (implicit caching only); batch and flex half; priority 1.8×; no free tier, not on AI Pro. Counter-evidence, carried at equal billing: AA’s index debuts Argon at 53 — tied with GPT-6 Astra and Fable 5.1, below Claude Opus 5.5’s 58 and Sonnet 5.5’s 56 — and the rollout thread praises its writing over Opus 5.5 while calling coding contested (one image→HTML test placed it “just under gpt 6.1 sol”).

Why it matters. A generation jump that lands with a price clock attached: the intro rate dies December 1, a month before Gemini 3.8 Flash’s own scheduled January 1 doubling — anyone benchmarking Argon against a Q4 budget should price both dates. And the evidence shape is the quarter’s recurring lesson: vendor SOTA claims on one hand, a below-the-Opus debut on the only independent index on the other, Real-SWE still silent. Test it on your workload before the intro price does the deciding for you.

Go deeper: announcement · pricing · rollout thread

No Japan items this issue: the scheduled October 14 Japan desk run fired but failed before committing anything; the editor-directed fallback ships three regions rather than filler. Japan coverage returns with the October 18 issue.

Research Radar

No in-window China paper met the read-beyond-abstract bar. N-OPSD is parked after a fourth failed render attempt — it resurfaces only if found organically. The ExtractBench provenance closure (Big Story) is this cycle’s research work.

Repo Radar

No new China repo merited entry; the SAS repo’s first issue (covered above) and the ExtractBench maintainer repo (run-llama/ExtractBench — dataset and eval code, inspected at listing depth) are the cycle’s repo notes, both in the ledger.

My Read on the Week

(Synthesis across the three desks that ran — marked as interpretation throughout; every figure traces to a ledger row.)

1. The measurement layer is the story. Three desks, one lesson in different registers. China’s double verdict: ExtractBench graduated — real, enterprise-grade, and its authors corroborating the short-good/long-bad collapse our ledger traced on two vendors’ cards — while MarathonMemBench dissolved into a phantom, cited by a claim nobody could have checked. The US: Argon announces SOTA (DeepSWE v1.1 at 77.9, vendor-run) and simultaneously debuts at 53 on the only independent index — below Claude Opus 5.5’s 58 — with Real-SWE silent a fifth consecutive read. The annotation that survives both: failure modes are quotable, rankings are marketing — whether the adjacency is a maintainer topping its own benchmark or a vendor timing its own table.

2. Hosted serving now carries dated clocks — and the exit is self-hosting. Google’s rate card kills Argon’s intro rate December 1 ($2/$10 doubling to $4/$20), Gemini 3.8’s own doubling lands January 1, and GPT-5.6 Sol’s promo runs “at least through November 21” — three dated flips inside six weeks across two vendors’ cards, the kind of calendar that quietly reprices every Q4 benchmark. Sarvam ran the same logic to its destination: one supported chat model, the hosted 105B, with the open weights still downloadable but API-rejected — self-hosting is now the only serving path for them, which is exactly what makes the weights’ continued availability worth tracking. The discipline the silicon story has held for five press cycles applies to rate cards too: the primary, read the day you act, beats the number you remember.

3. Two governance gaps, both quiet, both standing. Argon’s autonomous vulnerability finding ships “without cyber guardrails” for trusted defenders — a governance decision embedded in a release note. Sarvam’s voice-cloning thread is now six dated entries deep with zero consent-verification documentation, while the open alternative (AI4Bharat, Issue 11) carries card-level consent terms; SDK-grade ergonomics with call-shape parity to plain TTS makes that gap cheaper to fall into, not smaller. Neither blocks anyone this week; both are the kind of omission that becomes an incident later — the ledgers carry both as open items.

What I’d Watch Next

  1. omni-flash-preview shuts down October 22 — Google’s deprecations page (updated October 12) names gemini-omni-1.1-flash as the replacement; Sunday’s issue carries the final reminder, four days out.
  2. The six-week price clock — Argon doubles December 1, Gemini 3.8 doubles January 1, GPT-5.6 Sol’s promo ends “at least” November 21: three dated flips across two rate cards. Anyone benchmarking on a Q4 budget should price all three dates before quoting a per-task cost.
  3. A direct license question to SAS’s maintainer — the collaborator who answered issue #1 substantively is the path to closing the ledger’s longest HIGH+U item (eighth check; the repo still carries no license).

One Thing I’d Experiment With This Week

Run the ten-minute instrument audit on your own decision stack. This cycle’s biggest finding cost ten minutes and dissolved a claim you might have quoted.

  • Setup: list every benchmark name currently cited in your model-selection docs, vendor decks, and team threads — the ones decisions actually rest on.
  • Conditions: for each, find in under ten minutes: the paper (arXiv or venue), the repo (maintainer, last commit), and the top ranker. Three outcomes: found-and-independent, found-and-vendor-adjacent (annotate: failure modes usable, rankings marketing), or not found (mark fabricated-until-proven, and note who cited it).
  • Measure: the share of your decision-citing benchmarks in each bucket, and the oldest not-found name still in circulation. Decision output: a one-page instrument register for your team — the same table our ledger keeps — and a very short list of claims that need re-sourcing before the next model decision. If the exercise finds nothing phantom, you have unusual hygiene; if it finds one, you just saved the cost of a wrong model choice.

Corrections

[US] 2026-10-14 — Issue 12 overstated the GPT-5.5 retirement’s scope. Issue 12’s migration-calendar item said “GPT-5.5 retires October 14” in API-migration framing, implying readers’ API integrations were on the clock. The changelog entry’s full text (September 14) is narrower: GPT-5.5 “will retire from ChatGPT, ChatGPT Work, and Codex on all plans” — “with the API unaffected.” Verified against OpenAI’s rate card on October 14, the retirement date itself: gpt-5.5, gpt-5.5-pro, and gpt-5.4 rows remain listed and unmarked. The product-surface retirement proceeded as scheduled; no API migration was required. Root cause: the claim shipped from the changelog’s summary line without reading the entry’s scope sentence — the carryover-shaped failure class this playbook’s §1 documents. The deadline preserved in Issue 12’s Watch Next inherits this scope correction. Re-verified against OpenAI’s changelog and the API pricing page, both read October 14.

All vendor-reported figures are labeled as such; if independent evidence doesn’t exist yet, that’s stated rather than papered over. This issue’s desk ledgers live under research/2026-10-14/.

Citation

BibTeX citation:
@online{dimsum_editorial2026,
  author = {dimsum editorial, the},
  title = {The Collapse Graduated — and One Benchmark Evaporated},
  date = {2026-10-14},
  url = {https://news.llm101.in/issues/2026-10-14/},
  langid = {en}
}
For attribution, please cite this work as:
dimsum editorial, the. 2026. “The Collapse Graduated — and One Benchmark Evaporated.” October 14. https://news.llm101.in/issues/2026-10-14/.