The serving engine itself went open
DeepSeek open-sources DeepInfer — the production inference stack, dual-silicon across NVIDIA SM100 and Huawei Ascend 950, 18.6k stars in a day — while Sakana’s QD-Rec evolves recommendations with their reasons.
DeepInfer (Apache-2.0, official DeepSeek org) open-sources the production serving engine — every kernel this desk has tracked assembled, DSpark speculation and SWA bounded replay as env flags, the first dual NVIDIA-plus-Ascend engine we’ve ledgered — surfacing October 21 at 18.6k stars with no published benchmark numbers; Sakana’s QD-Rec ships paper, code, and blog in-window with honest caveats; India re-enters the Bulbul V3 November 10 EOL. Issue 15, coverage window 19–21 October 2026.
Week of October 19–21, 2026 · Issue 15
The Week in One Sentence
A three-day window with two strong signals: DeepSeek open-sourced DeepInfer — the production serving engine itself, every kernel this publication has tracked since Issue 4 assembled into one Apache-2.0 framework that runs on both NVIDIA SM100 and Huawei Ascend 950, 18.6k stars inside a day and not one published benchmark number — and Sakana’s QD-Rec turned recommendation into open-ended evolution, shipping paper, code, and blog with the season’s most honest caveats.
The Big Story: DeepInfer — the engine leaves the building (China)
What happened. (China.) On October 21, DeepInfer surfaced under the official deepseek-ai org: “a high-performance, large-scale inference framework for DeepSeek models” — positioned as the framework running DeepSeek’s own production services, open-sourced for offline inference under Apache-2.0. The repo states no release date, but the reception is its own datapoint: 18.6k stars, 1.9k forks, and 168 open pull requests within a day — the fastest infrastructure adoption this ledger has recorded. The default checkpoint is V4.1-Flash-1017; the stack is built on vLLM (crediting vLLM as primary base, plus SGLang and TensorRT-LLM).
Under the hood — a roll call of everything this desk has tracked. The kernel layer is the Issues 4–10 line assembled: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels. Parallelism: attention data-parallelism with balanced context-parallelism, MoE expert-parallelism with EPLB. The speculation layer is DSpark with confidence-scheduled verification (NEXTN drafting, default length 5) — and the detail that made this desk sit up: SWA bounded replay, the V4.1 tech report’s prefill-compression trick, exposed as operator env flags (DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, keep-tokens 140). Research papers becoming shell variables is the surest sign a technique graduated. Add async scheduling on by default, batch-invariant inference, deterministic sampling, and structured outputs (regex / JSON schema / EBNF), and this is the most complete operator-surface spec we’ve ledgered from any lab.
What the evidence says — including what isn’t there. Two absences, both load-bearing. First, no benchmark numbers are published — prefill/decode pareto charts only (“measured on 4 SM100 GPUs for prefill and 64 for decode”), with run_sweep.sh shipped so anyone can produce the numbers themselves; until someone does, every performance claim about DeepInfer is a chart, not a figure. Second, the release excludes the KV-cache system, disaggregation, and data-parallel load balancing — honestly scoped in the README, and the clearest signal of what the next releases carry. The hardware line is the quiet headline: SM100 (CUDA 13) and Ascend 950 (CANN 9.2) in the same engine — the first official framework we’ve recorded where the Chinese-silicon path is a first-class target rather than a port, which operationalizes everything this desk has written about the silicon thesis since the Zhenwu unveiling.
Why this matters. Open weights made models inspectable; an open engine makes the serving economics inspectable — the scheduling, the speculation acceptance, the replay truncations that decide real cost per token. For operators, the vLLM lineage means the migration path is familiar even where the kernels aren’t; for the ecosystem, 168 open PRs on day one says the serving layer is about to iterate in public. And for the questions this publication keeps asking — whose silicon, at what bandwidth, at what cost — the engine that answers them on both NVIDIA and Ascend is now a git clone away.
What I would test. Run the sweep on your own hardware and publish the table the README didn’t: prefill/decode pareto at your batch shapes, on SM100 if you have it — and on Ascend 950 if you can get it, because the first public NVIDIA-vs-Ascend throughput comparison on identical software would be the most valuable ops datapoint of the quarter. Second: batch-invariant mode plus deterministic sampling is an evaluation-reproducibility feature disguised as an inference option — turn both on and see if your benchmark noise floor disappears.
Elsewhere in AI Engineering
QD-Rec: recommendations that evolve, with their reasons (Japan) — Worth Reading
What happened. (Japan.) Sakana AI released QD-Rec — “Open-Ended Evolution of Recommendations via Quality–Diversity” (Ishida et al. 2026) — with all three artifacts landing inside this window: paper v2 (arXiv:2610.23470, October 19; v1 October 16), the Apache-2.0 code (public October 19), and the blog post (October 20). The reframe is the headline: for high-stakes decisions over a fixed candidate pool — acquisitions, customer targeting, which papers to read — you don’t want one ranked list, you want a diverse archive of well-reasoned options. Every QD-Rec entry is an item paired with a hypothesis: a natural-language reason why that item fits the decision-maker’s needs.
Under the hood. Proposed entries face novelty-first admission: an LLM judge checks novelty against the three nearest archive neighbors — “A novel entry is admitted immediately; no comparison is made” — and only a non-novel entry fights, in a blind pairwise duel against its single nearest neighbor (both entries anonymized, scores withheld, so the judge can’t favor the newcomer). Generation conditions on three exemplars sampled from the archive — archived hypotheses literally become stepping stones, the open-endedness pattern (ShinkaEvolve, AB-MCTS) this lab has been building for two years, now applied to decisions instead of programs. Item selection is a Bayesian Thompson-sampling bandit with a diversity bonus (β=1, distance = 1 − max cosine to the archive), warm-started from seeds, its posterior updating even on discards.
What the evidence says. Domains: M&A over 6,332 companies (Disney and JPMorgan acquirer profiles), synthetic-customer marketing, 17,511 ML papers — and the honest instrument, Goodreads children’s books scored by real human ratings. Vendor-run results with standard deviations: M&A quality 2.91±0.05 vs 2.51 for the best ablation, with the archive more than doubling (214 vs 85 entries); papers essentially tied on quality but largest archive (789) and best diversity; and — the counter-evidence carried — on the human-rated Goodreads domain all variants are statistically similar (~4.12–4.19), QD-Rec distinguished only by archive size. The paper’s own caveats are unusually forthright: runs cost $13–62, with >90% of the cost difference being proposer input; and the headline “no MMR/DPP baseline reaches any QD variant” comes with the disclosure that the baselines saw zero ratings while QD variants saw up to 1,000 — an uncontrolled comparison, flagged by the authors themselves.
Why this matters. The recommendation reframe — archive over ranking, reasons as first-class artifacts — is exactly the shape the agentic-decision quarter wants: when a model must justify choices to a human, the hypothesis is the product. The honest-instrument tie is the finding to respect: on real human preferences, evolution bought breadth, not quality. (The Japan desk’s closing experiment — scoring the reasons on your own reading — is preserved at the end of this issue.)
The deadline that never made print: Bulbul V3, November 10 (India) — Worth Watching
What happened. (India. Carried item, published nowhere until now: drafted for Issue 14 and cut by the joint cap — the deadline is under three weeks out and existed only in a research ledger. Verified from the changelog’s primary feed October 18 and re-verified October 22.) The changelog’s October 14 entry states that Bulbul V3 “reaches end of life on Voice Agents on November 10, 2026 — agents must migrate to Bulbul V4,” with a volume slider for agent speech (0.7–2.0, default 1.0) in the same entry; October 16 brought dual-channel call recordings — every call in stereo, user on the left channel, agent on the right, “enabled for all calls.”
Why this matters. A dated EOL on a voice engine inside a voice-agent platform is a production deadline: V3-voiced agents stop working on a fixed date. The migration path has been paved for weeks — V4 persona naming and catalog, v4-flash guidance in the official skills, the voice-cloning SDK sharing Bulbul’s call shape (Issues 12–13) — so the deadline lands on prepared ground. And the re-entry is itself the lesson about the joint cap: Issue 14’s enforcement cut this item to hold 5–8, the date missed Watch Next, and a production cliff briefly existed in print nowhere. Dated cliffs should survive cuts — this desk carries the November 10 date in every ledger until it passes or the vendor moves it.
Go deeper: changelog
Hy-Memory ships a GGUF (China) — Worth Watching
What happened. (China.) Issue 14’s memory-writer specialist arrived in llama.cpp format this week — Hy-Memory-30B-A3B-GGUF updated October 20/21, days after the FP8 build. A 60 GB-class MoE writer in the local-runtime ecosystem is a small event with a large implication: memory-specialist models are now consumer-runnable, not just vLLM deployments.
Why this matters. The specialist line (parsing, retrieval, memory-writing) keeps landing in GGUF within days of release — the local-community surface is where permissive Chinese-lab artifacts compound fastest.
US desk — a zero-item window, stated rather than padded (§10.5). The October 19–21 window was swept and nothing above the bar surfaced: OpenAI’s October 19 news was fundraising-class, and the nearest model releases (Meta’s Muse Wonder pair, Microsoft’s MAI batch) date to the prior week. The desk’s carried deadlines are the actionable output: the October 22 retirement cluster — gemini-omni-flash-preview plus three veo-3.1 generate previews — executed as scheduled (replacement: gemini-omni-1.1-flash; verified from the deprecations page October 22). The price clock stands: Argon’s intro rates end December 1, GPT-5.6 Sol’s promo runs “at least through November 21,” Gemini 3.8 Flash doubles January 1; grok-voice-2.0’s deprecation remains undated; Sonnet 4.5 retires November 30. Sunday’s run re-sweeps the window from primaries.
No further Japan item met the bar this window: the remainder was an event-recap post, a CI-bot chore on PFN’s MN-Core sample repo, metadata-only pushes (including a fifth consecutive from ALE-Bench), and pre-release activity around NII’s full-duplex speech line (llm-jp-moshi) with zero shipped artifacts — all dated in the ledger.
Research Radar
The window’s research story is QD-Rec (Japan) — read in full from the paper’s HTML, and carried in full in Elsewhere. DeepInfer (China) ships no paper — README and a citation entry only, with the reproduction sweep script as its evidence path. No second paper met the read-beyond-abstract bar.
Repo Radar
deepseek-ai/DeepInfer (China)
- What’s actually in it: the full engine — kernel integrations (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, TileLang), parallelism (attention DP/CP, MoE EP+EPLB), DSpark speculation with confidence-scheduled verification, SWA bounded replay flags, async scheduling, batch-invariant and deterministic modes, structured outputs — plus eval, examples, tests, and the sweep harness. Inspection depth: README in full; code not executed.
- What you can run: offline inference for V4.1-Flash-1017 on SM100 or Ascend 950 tonight;
run_sweep.shreproduces the (unpublished) pareto charts. - What’s missing: the KV-cache system, disaggregation, and DP-replica load balancing — scoped as not-in-this-release; numerical benchmarks; and a paper.
- Why open it: the production serving stack of a frontier lab, dual-silicon, Apache-2.0 — the reference implementation for everything this publication has written about serving economics since Issue 4.
SakanaAI/QD-Rec (Japan)
- What’s actually in it: the full method implementation — archive with novelty-first admission and blind duels, the diversity-aware bandit, evolutionary operators — plus domain apps (customers, papers, and the paper’s others), profiles, and seeding scripts. Inspection depth: README in full, code tree not executed.
- What you can run: the papers app against your own profile tonight: one OpenRouter key,
uv sync, four commands to a growing archive. Paper configs use claude-sonnet-5 + gpt-5.6-terra; the repo defaults to the cheap tier (gpt-6-luna) for trials. - What’s missing: the human-ratings domain (Goodreads) is not in the public app set — the honest-instrument tie can’t be re-run from the repo as shipped; no cost telemetry beyond the paper’s $13–62 figures.
- Why open it: the most runnable artifact in the QD line — and the archive-grows-while-quality-ties shape on real human preferences is the finding any deployer should reproduce before trusting the M&A margins.
My Read on the Week
(Synthesis across the desks that published — marked as interpretation throughout; every figure traces to a ledger row.)
1. The stack is open end-to-end now — and the last layer was the engine. (China.) Weights went open (this desk’s whole ledger), then the kernels (FlashMLA’s dual-silicon releases, Issue 10), then the memory and specialist organs — and now the serving engine itself, scheduler to speculation, with the Chinese-silicon path as a first-class target. DeepSeek’s release pattern is deliberate disclosure in layers, each layer making the previous one’s claims checkable; what remains closed (KV-cache, disaggregation) is effectively the roadmap. When the engine is open, “cost per token” stops being a vendor quote and becomes a reproducible measurement — which is exactly why the missing benchmark table will not stay missing.
2. Open-ended search found its deployment shape: decisions, not programs. (Japan.) Sakana’s QD line has evolved programs (ShinkaEvolve), coordinated frontier models (AB-MCTS), and now evolves justifications — archives of item-plus-reason pairs where novelty admits and blind duels replace. The mechanics transplant cleanly because the payload changed: a hypothesis is just a program that runs in a human’s head. The honest-instrument result is the guardrail the whole field should copy — on real human ratings the archive grew but quality tied: evolution reliably buys diversity, and only sometimes buys quality. Buy it for the breadth; audit it for the margin.
3. Candor is compounding into a house style — and it’s load-bearing. QD-Rec’s paper flags its own uncontrolled baseline; DeepInfer’s README scopes its own absences (no KV-cache, no disagg, no numbers); India’s re-entry carries its own verification dates. When artifacts arrive with counter-evidence attached, the reader’s job shifts from hunting for the catch to weighing the one they handed you — and this publication’s ledger gets cheaper to keep honest.
What I’d Watch Next
- DeepInfer’s first external numbers — the 168 open PRs and
run_sweep.shmake the first independent throughput table a matter of days; the first public SM100-vs-Ascend-950 comparison on identical software would be the ops datapoint of the quarter. - Unlicensed-code watch, both hemispheres — MASS at 294★ (Japan carry) and SAS at ten license checks (China carry): community uptake on explicitly unlicensed code keeps compounding in two ecosystems at once.
- llm-jp-moshi (Japan carry) — NII’s full-duplex speech line is visibly pre-release; a release lands squarely in the voice-agent quarter.
One Thing I’d Experiment With This Week
Run QD-Rec on your own reading, and score the reasons, not the ranking. (Japan desk’s closing experiment, preserved.) The repo’s papers app is the cheapest deployed QD system available, and the experiment it invites is the one the paper couldn’t run: a real user grading real hypotheses.
- Setup: write your “Me” profile (research questions, methods, what makes a paper useful — the repo’s schema). Seed, embed, and run 300–500 evaluations at the repo’s cheap defaults; budget $15–40 and an evening.
- Conditions: two arms, same profile: the full method, and the repo’s Novelty ablation (no evolution, no bandit). Then blind-review 30 sampled archive entries from each: abstract read, then a 1–5 on whether the hypothesis — the stated reason this paper fits you — was right.
- Measure: hypothesis accuracy per arm; archive size and diversity (the repo logs both); papers-you’d-actually-read per $10. Decision output: whether the evolved justifications beat naive-novelty ones on your own preferences — the exact cell where the paper’s human-rated domain showed a tie, now measured on a user who can grade reasons in context. If evolution ties again, you have the second honest-instrument datapoint; if it wins, you have a working research-discovery engine for the cost of a takeout dinner.
(The China desk’s counterpart experiment — reproducing DeepInfer’s unpublished pareto on your own hardware, ideally across both silicon paths — is specified in the Big Story above; the two together make a fine week for anyone with GPUs and an evening.)
All vendor-reported figures are labeled as such; if independent evidence doesn’t exist yet, that’s stated rather than papered over. This issue’s desk ledgers live under research/2026-10-21/ (Japan, India, China, and the US zero-item ledger).
References
Citation
@online{dimsum_editorial2026,
author = {dimsum editorial, the},
title = {The Serving Engine Itself Went Open},
date = {2026-10-21},
url = {https://news.llm101.in/issues/2026-10-21/},
langid = {en}
}