The sparsity that pays for everything else in DwarfStar
DeepSeek V4 Flash keeps 97.5% of its parameters in routed experts that fire 2.34% of the time. Five design decisions in DwarfStar, the DeepSeek V4 engine from antirez, all collect on that one gap.
TL;DR. DeepSeek V4 Flash has 284B parameters and 277B of them, 97.5%, sit in routed MoE experts. Only 6 of each layer’s 256 experts fire per token, so 2.34% of that mass does the work. DwarfStar spends that one imbalance five separate ways: it quantizes by architectural role rather than tensor type, pages experts off the SSD, compiles a profile-guided hot-expert table into the binary, keeps the expert cache byte-identical at any budget, and keys its disk KV cache on rendered bytes because tokenization does not round-trip. All of it is legal only because the engine refuses to load models it does not know.
DeepSeek V4 Flash has 284 billion parameters. Work through the config and 277 billion of them, 97.5%, sit in routed mixture-of-experts weights. On any given token, 6 of each layer’s 256 experts fire, so 2.34% of that 97.5% does the work and the rest is inert.
Almost every interesting decision in DwarfStar, the DeepSeek V4 inference engine Salvatore Sanfilippo has been building in public since May, is a way of collecting on that one imbalance. It runs a 284B model at 39 tokens/sec on a 128 GB MacBook, and a 2-bit build of it fits in 81 GB. I spent a day reading the C, and what makes the engine interesting is less any single trick than how many separate wins come out of the same structural fact.
The engine is also deliberately not a general GGUF runner. It loads three model families, validates their exact tensor layout, and rejects everything else. That constraint is what makes the rest legal.
Where the parameters actually are
The shape table in ds4.c gives Flash 43 layers, 256 routed experts per layer, 6 active per token, one shared expert, n_embd 4096 and an expert FFN width of 2048.
Each expert holds gate, up, and down matrices, so 3 × 4096 × 2048 ≈ 25.2M parameters per expert, 6.44B per layer, and 277B across the model.
That leaves roughly 7B for everything else: attention, the router, the shared expert, the projections, embeddings. Those 7B parameters decide which experts run and how their outputs get combined, and they participate in every token. The 277B are where the bytes live; the 7B are where the behavior lives.
Once you have that split in front of you, several things follow that would be reckless to assume about an arbitrary model.
Quantize by role, not by tensor
The 2-bit builds quantize only routed experts, gate and up at IQ2_XXS, down at Q2_K.
Router, shared expert, attention, and projections are left alone.
The arithmetic works out: 277B at the ~2.23 bits/weight that mix averages is about 77 GB, plus 7 GB for non-routed weights at 8 bits, so ~84 GB against a shipped file of ~81 GB. Nearly all the compression comes from the part of the model that a given token mostly skips, and none of it touches the part that runs every time.
llama.cpp can express a split like this; it has per-tensor-type overrides and its own mixed-precision recipes. The difference is that those recipes are heuristics over tensor names that have to be defensible across every architecture GGUF supports, while DwarfStar hardcodes the role mapping for models it can name, validates that the loaded file matches, and refuses the file otherwise. Narrowness buys verification here rather than expressiveness. The README’s claim that the 2-bit quants “behave well, work under coding agents, call tools in a reliable way” is a claim about specific files, not a general quantization result.
Page the experts off the SSD
If routed experts are 97.5% of the bytes and 2.34% of the compute per token, they are the obvious thing to stop keeping in RAM.
Under --ssd-streaming the non-routed weights stay resident, and routed experts live in a fixed-size in-memory cache backed by reads from the GGUF on a miss.
That is what lets a 64 GB MacBook run a 284B model at all.
Prefill survives this better than generation does. A long prefill can batch and overlap its expert reads, while every single generated token re-routes and can miss again, which is why the README steers you toward larger caches when decode throughput matters.
Some experts run far more often than others
This is the decision I did not expect, and it is the one worth stealing.
Sparse access alone would justify a cache. What justifies a preloaded cache is that some experts are much hotter than others, consistently, across workloads. DwarfStar ships that measurement inside the binary.
ds4_streaming_hotlist.inc and its GLM sibling are generated tables of {layer, expert} pairs with a one-line header: “Generated from ds4 expert hotlist profiles; sorted by hits/weight.”
There are 6,436 entries for Flash, 6,884 for PRO, and 6,501 for GLM 5.2.
At startup the engine walks the table for whichever variant it loaded and seeds the expert cache in that order, so the cache begins warm instead of cold.
The profiler that produces those tables is in the same repo. ds4_metal.m counts expert selections per {layer, expert} during a run, merges them with any existing profile, sorts, and writes the list back out at exit.
Feed that output back into the build and you have profile-guided optimization, applied to weight residency rather than to branch layout.
Two details around it show the idea had contact with reality. Automatic preload is capped at 4096 experts, with the reason stated in the code: large caches “can otherwise spend startup doing thousands of preads into shared Metal buffers and trip the system watchdog before decode begins.” And on Apple silicon there is a neater trick still, where the cache is seeded from layer weights that prefill has already mapped into GPU-visible memory, “rather than rereading it after prefill.” The hot seed rides along on work the engine was doing anyway.
The cache is never allowed to become a correctness structure
Size the expert cache too small and it thrashes. The code computes exactly when, and what it does with that number is the part I liked:
/*
* Below one token's routed working set (uniform routed layers
* x experts used) every token evicts entries it is about to
* reuse, and prefill serves layer overflow through mapped
* model views. Output stays byte-identical at any budget
* (the addr-table kernels read the same bytes either way);
* only throughput collapses, so warn instead of refusing.
*/
const uint64_t min_experts =
(uint64_t)(routed - boosted) * DS4_N_EXPERT_USED;
if (min_experts != 0 &&
e->ssd_streaming_cache_experts != 0 &&
e->ssd_streaming_cache_experts < 2u * min_experts) {
fprintf(stderr,
"ds4: WARNING: SSD streaming expert cache (%u experts) is "
"under twice the per-token routed working set (%u layers "
"x %u experts = %llu); expect heavy thrashing below "
"%.2f GiB\n",
e->ssd_streaming_cache_experts,
routed - boosted,
DS4_N_EXPERT_USED,
(unsigned long long)min_experts,
(double)(2u * min_experts * slab_expert_bytes) /
1073741824.0);
}min_experts is one token’s routed working set, 43 layers × 6 experts = 258 for Flash, and the check fires below twice that.
The warning converts the threshold into the actual GiB figure, so a user who under-sizes the cache gets a number to act on rather than advice to try something bigger.
Byte-identical output at any cache budget means the cache is purely a performance structure. Under-size it and you get a slow engine and a warning, never a quietly worse model. That property is easy to state and easy to lose, and plenty of systems lose it by letting a memory-pressure path silently change numerics.
Tokenization does not round-trip, so the KV cache keys on bytes
The last one has nothing to do with experts and is my favorite.
Chat APIs are stateless, so agent clients resend the whole conversation every request, and a server that re-prefills all of it burns the user’s time on tokens it has already processed. ds4-server keeps a disk KV cache so useful prefixes survive session switches and restarts.
The obvious key for such a cache is the token-ID prefix. DwarfStar keys on the SHA1 of the rendered byte prefix instead, and the README explains the failure it is dodging: “the model may have generated one token whose decoded text is later sent back by a client as two canonical prompt tokens.”
Sampling picks token IDs. The client gets text. When that text comes back and is re-tokenized, the canonical segmentation need not reproduce the IDs the model actually emitted, so an identical conversation can present a different token sequence. Key on IDs and you miss a cache entry you legitimately hold. Key on decoded bytes and the hit survives, after which the stored token IDs from the checkpoint are treated as authoritative and only the new suffix gets tokenized.
The same non-determinism is handled again one layer up. The server keeps a bounded map from tool-call IDs to the exact block of tool-call syntax the model sampled, so a restarted server can re-render client history byte-for-byte even if the client reordered the JSON arguments in between.
There is also a small, very deliberate I/O choice: cache files use ordinary read/write rather than mmap, “so restoring cache entries does not add more VM mappings to a process that already maps the model.” When your process has an 81 GB file mapped, you stop treating address space as free.
How they know the quantization didn’t break it
A 2-bit routed-expert quantization is a strong claim, and the repo backs it with an eval design I had not seen before.
Rather than wikitext perplexity or a few sampled answers, they collect 100 continuations from the official hosted DeepSeek and GLM APIs with top_logprobs=20, then score each local GGUF by the negative log likelihood it assigns to that exact official continuation, token by token.
The stated reason is that it “avoids judging quality from one sampled answer.”
The comparator reports per-case NLL wins, how often the local greedy token matches the API’s, average greedy common prefix, and agreement with the API’s top-20 alternatives.
The target is the hosted model’s own behavior rather than a generic corpus, which is the right target when the question is “did my quantization move this model,” not “is this model good.”
The same harness scores full-residency against SSD-streaming runs, so the streaming path is checked for numerical drift and not just for speed.
Separately, ds4-eval runs a 92-question capability set, and the README is careful that it “should not be reported as an official GPQA, SuperGPQA, AIME, or security benchmark score.”
What the narrowness actually bought
Each decision needs a fact a general runner cannot assume: which tensors are routed experts, that they dominate size, that their selection is skewed and profilable, that the cache is outside the numerics, and how this tokenizer behaves on a round trip. DwarfStar gets all five by naming its models and rejecting the rest, and it pays for them in scope, since a new model means new shape entries, new validation, and a new profiling run to regenerate the hotlists. The README is upfront that models will be dropped when better ones arrive.
The transferable idea is the profiling one. We have decades of practice profiling code to decide what stays hot in cache, and MoE routing makes weights behave like code paths, with a measurable, skewed, workload-dependent access distribution. Compiling a hot-weight profile into the binary is a natural response, and I have not seen many engines do it.
Status and further reading
A few honest notes on what is verified here and what is not.
The throughput numbers are the author’s, not mine. The 790 t/s prefill and 39 t/s generation figures come from ds4-bench on a MacBook Pro M5 Max with 128 GB, published in the repo’s speed tables. I have not reproduced them, and reproducing them needs hardware I do not have.
The parameter arithmetic is mine and is checkable. 43 × 256 × 3 × 4096 × 2048 comes straight from DS4_SHAPE_FLASH in ds4.c. It lands at 277B of a stated 284B, and the implied file size (277B at ~2.23 bpw plus ~7B at 8 bits ≈ 84 GB) sits within about 4% of the shipped ~81 GB, which is the sanity check that the split is what the README says it is.
Quality claims are the project’s own. The official-continuation NLL harness is a good design, but I ran none of it. Treat “the 2-bit quants behave well” as a claim about specific published GGUF files, backed by a methodology you can inspect, not as a general result about 2-bit MoE quantization.
The project was built with heavy LLM assistance. The README says so directly, with humans leading the ideas, testing, and debugging. It is part of what makes the codebase worth reading rather than a caveat against it.
The primary sources are the DwarfStar repository and its README, which is unusually detailed about the reasoning behind each mode; the quantized GGUF files on Hugging Face; and the DeepSeek V4 Flash model card for the architecture the engine targets. The files worth opening first are gguf-tools/deepseek4-quantize.c for the role-based quantization, ds4_ssd.c for the streaming cache plan, ds4_streaming_hotlist.inc for the compiled-in profile, and ds4_kvstore.c with the “Disk KV Cache” README section for the byte-prefix keying.