The selector finally trains on the loss — and shows where it breaks

Tencent-Hunyuan’s SAS makes Top-K selection differentiable; its own RULER tables are the honest counterweight.

AI Engineering
Architecture
Open Weights
China
India
US

The selector finally trains on the loss — and shows where it breaks: Tencent-Hunyuan’s SAS makes Top-K selection differentiable, with its own RULER tables as the honest counterweight. Issue 5, week of 16 September 2026.

Author

the dimsum editorial

Published

September 16, 2026

Week of September 14–16, 2026 · Issue 5

The Week in One Sentence

A quiet window for launches was a loud one for state: Tencent-Hunyuan’s SAS makes attention selection trainable by the task loss — with its own failure table attached — while Anthropic’s signed compaction and Sarvam’s forward-only data retention turn conversation state and data state into explicit, governed controls.

The Big Story: SAS trains the selector with the LM loss — and shows its own collapse table

What happened. (China.) SAS — Simple Attention Sparsification (paper, September 11; code and runnable AttnGates artifacts landing on the org page in-window, ~September 14) from Tencent HY LLM Frontier with HKUST and HKUST-Guangzhou. (Labelled update discipline: paper September 11, runnable release this window — both dates stated.) The release is small: router-only checkpoints of 33–42M gate parameters that sit on frozen Qwen3-4B/8B/14B backbones. No license appears on the card or in the repo — flagged “Worth Verifying,” not asserted as usable.

Under the hood. The problem SAS attacks is the one this publication has tracked since Issue 1: selecting which KV blocks attend is the bottleneck, and Top-K selection blocks gradient flow, so selectors train on proxies — attention distillation, heuristics. SAS keeps Top-K but adds continuous soft gates on selected blocks, injected as log-space biases inside the attention softmax (o = softmax(qKᵀ + log g)V). The next-token loss itself then trains the ranking — no distillation — and at inference the gates drop away, leaving only Top-K indices. The ablations are the interesting engineering: gates must sit inside the softmax in log space (outside only rescales values); softmax activation (sigmoid saturates, raw logits collapse); soft beats hard Top-K with straight-through estimators (unbounded gradients); and gradients flow only through selected blocks — cheaper, same quality. Training: OpenR1-Math-220K, one epoch, frozen backbone — 33M trainable parameters on 8×H20.

What the evidence says (vendor-run, with the counter-evidence carried at equal billing). At a 1,024-token budget, SAS beats SeerAttention-R by +10.6 to +15.5 points on GPQA-Diamond and +6.0–7.7 on MATH500; at budget 4,096 it matches or exceeds full attention on AIME24 (71.72 vs 71.25 at 4B). Decode speedups: 2.4×/4.6×/5.6× over dense at 64K/256K/512K context (Qwen3-4B, SGLang, single GPU, batch 1; ~13× at 64K batch 8). Trained on math only, it still transfers — LongBench recovered to within 0.4 of full attention at 4,096, and VitaBench agentic metrics beat even dense. Then the authors’ own limitation, which we quote because it is the story: RULER at 128K context collapses — 21.87/23.46/29.95 (4B/8B/14B) against full attention’s 63.81/73.87/82.23 — pooling-based block summaries lose needle-like signal, and Top-K selection cost grows from 21% of step time at 8K to 90% at 512K. And the baselines make the point sharper: Quest scores zero on AIME at 2,048.

Why this matters. This is the sixth design in the selection-convergence ledger (DSA → QSA → IndexCache/KPool → LSA → CSA2 → SAS), and the first to make selection trainable by the task loss — while documenting precisely where the approach does not yet work. The win territory is reasoning and agentic workloads at tight budgets; the loss territory is retrieval at distance, which is the territory 1M-context marketing sells. Both tables ship in the same paper, and that candor is rarer than the speedups. Note also what the ecosystem is converging on: an exact-Top-K operator paper from Tencent’s HPC-Ops library (arXiv:2609.08450, September 8 — pre-window, labelled) attacks the same selection-cost wall SAS names as its own bottleneck.

What I would test. The cheap reproduction this line has never had: 33M routers on a frozen Qwen3-8B you already serve. Run the authors’ RULER configuration at 128K first — if the collapse reproduces on your workload, SAS is a budget tool, not a long-context tool; then run your agentic battery at 2,048 against your dense baseline and price the tokens saved. And watch for the missing piece: nobody has yet published SAS-style learned gates combined with exact Top-K operators.

Elsewhere in AI Engineering

BenchLM’s September refresh: the referee published a scoreboard (China) — Worth Reading

What happened. BenchLM’s Chinese-models leaderboard, last updated September 15 (in-window), ranks 76 models with per-row evidence labels and 90% intervals: Kimi K3 74.6 (Supported, 71.41–77.84) ahead of Qwen3.8-Max 71.8 (Supported), GLM-5.3 67.1 (Estimated), with DeepSeek V4-Pro-0813 sixth at 66.6 (Estimated) and Hy4-preview down at #19, 61.2 (Estimated). V4.1-Flash (released September 10) hasn’t entered yet.

Why it matters. Two readings. First, the independent layer is slowly thickening — a Supported/Estimated evidence label with intervals is exactly the kind of instrument this publication has been asking for since Issue 1. Second, the instruments still disagree: Hy4-preview sits #19 on BenchLM’s aggregate but ranked #8 on Code Arena’s WebDev board September 1 — different constructs measuring different things, both defensible, and a reminder that “where does model X rank” has no single answer. Our verification-gap watch continues.

Mulyankan grew a telemetry spine and a new name (India) — Worth Watching

What happened. (Labelled update on Issue 4’s coverage.) The repo we covered last issue as sarvamai/open-mulyankan is now open-rachana — same repository: the old URL redirects at the API, and the creation timestamp (September 7) is byte-identical. Inside this window it adopted OpenTelemetry as the project’s telemetry SPI — “Adopt OpenTelemetry as the telemetry SPI: ADR-0011, spec and implementation plan” (#90), September 14 — then bootstrapped it in platform/core with a local Collector and Grafana stack (#120, September 15). License re-read this issue per house rules: Apache-2.0, unchanged; description still “Opensourced Codebase for Mulyankan.” (Finalisation note, September 17: the repo moved orgs again on the evening of September 16 — the API now returns it under Bodhan-AI, with creation timestamp, description, and license byte-identical and old sarvamai URLs redirecting. The Bodhan-AI org’s four public repos all date from this month, and none of its materials states a relationship to Sarvam — the transfer is verifiable, its meaning is not.)

Technical signal. A telemetry SPI is M0 plumbing, but the choice is telling for a system whose selling point is verifiable confidentiality: an OpenTelemetry-native core means the observability plane speaks a standard protocol you can point your own collectors at, and the local-Collector-plus-Grafana bootstrap is consistent with (our inference, not a stated guarantee) the confidential-by-default posture — nothing in the commit text implies an external sink. The M1 walking skeleton — identity, audit chain — is still ahead; this is the groundwork that will make those claims observable rather than trusting the README.

Why it matters. The rename shipped with no announcement (none locatable again this cycle, search restored and checked) — the repo remains its own documentation, consistent with how this project has operated since day one. For engineers, ADR-0011 is the first commit-level signal about the observability plane for those sealed-store and hash-chain claims — how they will be supervised, not just asserted, when M1 lands.

Go deeper: open-rachana · ADR-0011 commit

Sarvam shipped data-retention controls (India) — Worth Reading

What happened. September 15, per the docs changelog (read from the primary): organization owners can now self-serve a data-retention period under Settings → Workspace → Data retention — a workspace-wide default plus per-product overrides, live for Model APIs and Voice Agents, with the rest of the product line “still rolling out.” Adjustments apply to “data going forward” only: nothing already stored is deleted early or extended retroactively.

Why it matters. Retention is the boring knob that decides whether an India-centric voice or document pipeline clears a regulated deployment review. Two details matter more than the feature itself. Forward-only semantics mean retention is a policy on ingest, not a backfill — compliance clocks start at cutover, and a migration plan has to name that gap. Per-product granularity means the transcription path can be tightened without touching LLM storage — the override surface, not the default, is where review conversations will live. For teams weighing Sarvam against self-hosting on data-governance grounds, this is a platform-side control that moves that calculus.

Go deeper: changelog entry

Gemini 3.8 Live brought extended thinking to real-time voice (US) — Worth Testing

What happened. September 15, Google’s Gemini Audio Team shipped two live dialogue models: Gemini 3.8 Live, “built for scale and cost efficiency,” and Gemini 3.8 Live Extended Thinking, “built for high-complexity tasks, with increased intelligence and multi-step reasoning.” Both are live speech-to-speech — voice in, voice out, visual input, and background tool/API calls that don’t interrupt the conversation — and Extended Thinking’s addition is that it “reasons and speaks simultaneously,” acknowledging prompts with early verbal cues and narrating progress through multi-step background tasks. The API pricing page, read the day after, lists both (gemini-3.8-live, gemini-3.8-live-extended-thinking) as audio-to-audio models on one shared table: $0.75/M text in, $3.00/M audio in ($0.005/min), $4.50/M text out, $12.00/M audio out including thinking ($0.018/min).

Technical signal. The benchmark story is vendor-cited on third-party instruments — label it as such: #1 on Artificial Analysis’ Speech-to-Speech Quality Index at 82.6 (Extended Thinking), 68.6% on τ-Voice agentic task completion, 35.1% on Sierra’s τ-Voice-banking, 97.7% on Big Bench Audio; the base Live model sits #2 on the Speech Agent Arena. Base Live auto-detects and switches among 97 languages mid-conversation; all generated audio is SynthID-watermarked. Availability is tiered: Live API + AI Studio now, Vertex multimodal live studio in private preview, consumer surfaces (Search Live; Docs/Gmail/Keep Live) rolling out.

Why it matters. The HN thread (363 points, 232 comments) reads like a field report on where live voice actually stands: multilingual practice and low latency praised across languages; a consensus that Gemini still trails Claude and GPT-6 Astra for coding; reliability complaints (context loss in the next message, deep-research hallucination). That matches the positioning — these are dialogue models, not coding models. Two operational notes for API users, both from the day-of reads: the deprecations page (last updated September 5) schedules gemini-omni-flash-preview’s shutdown for September 30, replacement gemini-omni-1.1-flash — Live-API deployments have two weeks to migrate; and 3.8 Flash’s promo prices are dated to double on January 1, 2027, so pin effective dates in any cost model that touches the 3.8 family.

Go deeper: announcement · pricing · HN discussion

Anthropic shipped on-demand conversation compaction (US) — Worth Reading

What happened. September 14, per the platform changelog (read from the primary): the Messages API can now “compact a conversation on demand” in beta — send a top-level compaction parameter (beta header compact-2026-09-04) and the API returns a signed summary block that replaces earlier messages in subsequent requests, with kept turns preserved word-for-word.

Technical signal. The mechanics are the story. The summary is signed — server-attested rather than client-trusted — and the word-for-word guarantee on kept turns pins exactly what compaction may not touch: the boundary between lossy summary and faithful transcript is explicit in the API surface, not implied by a prompt convention. The changelog entry does not state how compaction output is billed; that gap is the adoption question.

Why it matters. This issue’s China sections are all about context economics — budgeted attention, token budgets, win-region maps. Compaction is the same economics applied to conversation state: long-running agent sessions now have a server-side option whose failure mode is auditable (diff kept turns against summarized ones) instead of a hand-rolled summarization prompt failing silently. The decision input the changelog doesn’t give is cost — how compaction output bills against simply re-sending history is the number to capture before adopting at scale.

Go deeper: changelog

The US update ledger: a benchmark revised itself (US) — Worth Watching

What happened. (Labelled update on Issue 4’s US coverage.) Two follow-ups. First: Real-SWE revised its results between our September 13 read and today — Grok 4.6 (Grok Build) moved from 23.8% to 32.5%, from mid-table to rank 3, above Gemini 3.8 Flash (31.2%); every other entry is unchanged (Fable 5.1 still leads at 38.8%), and no revision note, paper, or task release accompanies the change. Second: NVIDIA’s Open-SWE-Traces bumped again September 15 (API timestamp) — downloads ≈27,000, up from ≈24,000 on Sunday — and the card now notes OpenCode and Claude Code harness trajectories “coming soon” alongside v1.2’s Qwen3.8-27B runs.

Why it matters. Issue 4 published the 23.8% figure; this updates it. An independent benchmark revising a score by +8.7 points and changing rank order — silently, with no stated reason — is itself evaluation news, and the counterpoint to the same window’s BenchLM refresh above: one instrument publishes intervals, another revises in place. And the coming-soon harnesses matter because Claude Code is Anthropic’s proprietary harness — open trajectories from it would let Open-SWE-Traces’ exploit-audit methodology (its v1.0 git-hacking filter) run against the most widely used commercial agent harness.

Go deeper: Real-SWE · Open-SWE-Traces

No Japan items this issue: the scheduled September 16 Japan desk run was missed, and the editorial fallback ships three regions rather than filler. Japan coverage returns with the September 20 issue.

Research Radar

The window’s China research story leads the issue as the Big Story (SAS — full read in the packet’s paper-notes). No second China paper met the read-beyond-abstract bar inside the window; the adjacent exact-Top-K operator work (HPC-Ops, arXiv:2609.08450, Sep 8) is pre-window and carried as labelled context inside the Big Story.

Repo Radar

Tencent-Hunyuan/Simple-Attention-Sparsification (China)

  • What’s actually in it: training recipes for Qwen3-4B/8B/14B selectors (stated to fit 8×H20), eval runners (reasoning, LongBench, BFCL, VitaBench) through the sglang-blocksparse fork, and the AttnGates routers on HF (64–81 MiB). Inspection depth: README + file listing, directly inspected; not executed.
  • What you can run: the full train-and-eval loop — the cheapest reproduction entry in the sparse-selection line we cover. Note the wrinkle: VitaBench wants a DeepSeek API key for its user simulator, and each eval boots a server from the SGLang fork rather than a standalone server.
  • What’s missing: a license — nowhere, neither repo nor card (HIGH, Worth Verifying); benchmark scores on the card (results live in the paper); commit dates.
  • Why open it: if you serve Qwen3 at any scale, this is a weekend-scale experiment with published failure modes to check yourself against.

My Read on the Week

1. The selection war entered its training-methods phase. Five entries in the convergence ledger optimized where to select (indexers, budgets, micro-blocks, static modes); SAS — the first entry in that ledger to make selection trainable by the task loss itself, rather than via distillation or heuristics — optimizes how selection learns. When a research line starts refining its training objective rather than its architecture, that is usually the sign the architecture stabilized. The open flank is now operator-shaped: 90% of decode time at 512K is the Top-K itself, and the exact-Top-K operator paper from the same corporate family is aimed exactly there.

2. Candor is becoming the differentiator — at both layers. SAS ships RULER numbers that argue against its own headline; DeepSeek’s card documented ExtractBench’s long-document collapse; Tencent’s Hy4 card self-flags over-verification. The same window’s evaluation layer split the identical way: BenchLM’s refresh publishes 90% intervals and Supported/Estimated labels on every row, while Real-SWE revised Grok 4.6 by +8.7 points and re-ordered its leaderboard silently — no note, no reason, no paper. Vendor failure tables tell you the boundary of a win region without buying the GPU-hours; instrument transparency tells you whether a score is a measurement or a mood. The pattern to price into adoption decisions is who shows their intervals — and who revises in the dark.

3. The window’s real launches weren’t models — they were state controls. No model shipped from the major Chinese labs in three days, yet every desk that published produced a control on a different kind of state: attention state (SAS’s budgeted KV blocks), conversation state (Anthropic’s signed compaction with word-for-word kept turns), data state (Sarvam’s forward-only retention periods), observability state (open-rachana’s OpenTelemetry SPI). None of these is a benchmark headline; each is a knob that decides cost, compliance, or debuggability at scale. The launch cadence is bursty and headline-driven; the state layer underneath compounds continuously. Read the quiet weeks there.

What I’d Watch Next

  1. SAS reproduction, especially RULER at 128K — 33M-parameter routers on frozen bases mean the community can actually check the collapse table; first independent run settles whether it’s method-deep or config-shallow.
  2. The SAS license — still absent from repo and card as of the finalisation re-check (September 17); if it lands permissive, this becomes the default teaching example of trainable selection.
  3. The Gemini Live-API calendar — gemini-omni-flash-preview shuts down September 30 (replacement gemini-omni-1.1-flash), and 3.8 Flash’s promo prices double January 1. Two dated cliffs for anything built on Gemini audio — pin both in the cost model now.

One Thing I’d Experiment With This Week

Map the win region before you adopt any sparse-selection method. SAS’s own tables describe a frontier — wins at low budget on reasoning/agentic, losses at distance on retrieval — but your workload sits somewhere specific on that map.

  • Setup: one frozen Qwen3-8B, the released AttnGates, the authors’ eval harness; three context regimes (8K, 64K, 128K) × three budgets (1024, 2048, 4096).
  • Conditions: two task families per regime — a reasoning/agentic battery (your own, plus BFCL as the published anchor) and a retrieval battery (RULER-style needles plus your real documents). Run dense once per regime as the ceiling.
  • Measure: score delta vs dense per cell, plus realized decode speedup and the selection-cost share of step time (the paper says 21%→90% from 8K→512K — does your hardware agree?). Decision output: a two-column table — the budget at which quality breaks per task family, and the tokens/sec you gain before it breaks. If reasoning holds at 2,048 while retrieval needs ≥4,096 at 64K, you can route: sparse for agentic loops, dense (or pooled-prefix) for document Q&A — and that routing decision is worth more than any single benchmark score.

All vendor-reported figures are labeled as such; if independent evidence doesn’t exist yet, that’s stated rather than papered over. This issue’s desk ledgers live under research/2026-09-16/.

Citation

BibTeX citation:
@online{dimsum_editorial2026,
  author = {dimsum editorial, the},
  title = {The Selector Finally Trains on the Loss — and Shows Where It
    Breaks},
  date = {2026-09-16},
  url = {https://news.llm101.in/issues/2026-09-16/},
  langid = {en}
}
For attribution, please cite this work as:
dimsum editorial, the. 2026. “The Selector Finally Trains on the Loss — and Shows Where It Breaks.” September 16. https://news.llm101.in/issues/2026-09-16/.