How Data Mixture Affects Pre-training
Published:
How Data Mixture Affects Pre-training
DeepSeek-V4 has been out for a while now, but if you scroll through the community discussion, you can hardly find anyone seriously talking about how good it actually is on pre-trained knowledge. Open the technical report, though, and something interesting stands out: the things DeepSeek most wants you to notice are all crammed into the very first figure (Figure 1) — on the left, a knowledge benchmark that almost nobody mentions anymore, SimpleQA-Verified; on the right, its long-context speed demo. Knowledge on one side, speed on the other — exactly the two directions pre-training is racing along today. And within its own scale class, DeepSeek does both remarkably well. The thing this figure is really about, underneath, is data — but more on that later.
Figure 1: Header figure (Figure 1) of the DeepSeek-V4 technical report. Left red box: SimpleQA-Verified benchmark results. Right red box: inference FLOPs and KV-cache efficiency in long-context settings.
If Figure 1 is the “knowledge” and “speed” that DeepSeek curates for you to see, the community leaderboards give a reference that nobody curated on its behalf. On AA-Omniscience — a hard-knowledge benchmark that scales almost monotonically with model size — DeepSeek still stands out: its smaller / flash variant beats a long list of far larger competitors, making “a small model can still carry a lot of knowledge” concrete. But this race is far from over: Kimi has caught up recently and is now trading blows with DeepSeek on the hard-knowledge column. The small-model contest still has a long way to go.
Figure 2: AA-Omniscience Accuracy cross-model leaderboard (top 28). DeepSeek’s small / flash models stand clearly ahead of a number of much larger competitors; Kimi has also closed in.
Bojie Li wrote a paper noting that knowledge benchmarks like SimpleQA are essentially impossible to improve through SFT. So here’s how I read it: setting aside the caveat that hallucination-oriented RL may drive the model to refuse more (and pull knowledge accuracy down), a SimpleQA score is basically locked the moment pre-training ends. Look back at the neglected SimpleQA-Verified in DeepSeek’s header figure — the reason it’s placed front and center is exactly that it measures a model’s pre-trained knowledge most directly; and that knowledge is set by the data mixture during pre-training, not patched in afterwards. So we’re back to the real topic of this post: data mixture and ratio.
So where do these gains actually come from? Changes to architecture or training method can each chip in something — I won’t deny that. But my own sense is that the more fundamental, and more underrated, part still comes from data mixture. Or, in the words of Microsoft AI’s MAI-Thinking-1 technical report (MAI-Thinking-1: Building a Hill-Climbing Machine, Microsoft AI, 2026), “semantic deduplication.” My view is that these aren’t really two things — just the same thing at different scales. Data mixture decides how much of each source and domain goes in; semantic deduplication decides, at a finer grain, which near-duplicate text isn’t worth feeding again. One handles breadth, the other handles redundancy, but both answer the same underlying question: what data, and at what weight, actually makes it into pre-training.
Back when I was at LongCat in 2024, I already had a fairly crude intuition about this: rather than arguing about which data to feed, you sort every piece of data into a few buckets on the axes of “density × learnability × count,” and assign a different sampling rate to each bucket. That idea was partially formalized in our ACL 2025 paper Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies⁶ — through a two-axis “data density × diversity” framework. The figure below (from Figure 5 of that paper — sorry the sketch is so ugly here; it looked a lot nicer in the published version) visualizes, under two topics (Objects / Math), how clusters in embedding space differ in density and diversity:
Figure 3: Chen et al., ACL 2025, Figure 5. Within a single corpus, clusters split along the axes of density (how packed the points are inside a circle) and diversity (how similar those points are to one another).
Project those axes onto the sampling rate, and you end up with four cases:
- Large n, easy to learn, relatively high density — e.g. the blue cluster under “Objects,” the kind of high-frequency, formulaic factual phrasing that abounds on the web. The model gets it at a glance; a second pass isn’t going to teach it anything new. Sample at a very low rate.
- Large n, hard to learn — e.g. the orange cluster under “Math”: n is already big, individual samples look very different from one another, and yet every single one of them is hard. One pass isn’t enough; the model needs to see them repeatedly. Sample at a relatively high rate so they keep showing up across the pre-training budget.
- Small n, easy to learn, relatively high density — n is small but the samples are highly similar to each other. The cluster is already saturated; more near-duplicates only buy you redundancy. Sample at a lower rate, and reclaim that budget for something else.
- Small n, hard to learn — small, sparse, low-density, and each sample looks unlike its neighbors. Almost always important long-tail data: rare exam-grade concepts, niche professional cases, recent events. Sample at an extremely high rate, otherwise under uniform sampling it might never get enough exposure in the entire pre-training run.
Put plainly: sampling rate shouldn’t be allocated linearly with raw data volume. It should follow the triple (density × learnability × count) — discount the easy-and-plentiful clusters, boost the hard-and-rare ones.
But honestly, turning this idea into practice is full of difficulties, and they concentrate in two places:
- It is hard to control the granularity of clustering. Naively running k-means on plain embeddings is itself problematic — the granularity is hard to tune, and documents that merely share a similar format (a templated web page, a series of reports from the same template) get pulled into one cluster, so the resulting buckets don’t always hold up semantically.
- Once you have clusters, it’s hard to quantify a cluster’s density. People have mostly used loss for this — but my own feeling is that we still need a newer metric, because different data carries a natural loss bias (long text, code, and math each have different loss magnitudes and descent curves, so comparing raw loss directly is misleading). My own hunch is that using a small model to “watch” the loss-descent curve could characterize “how much learnable headroom is left in this cluster” far better than a single loss value — but that pipeline again looks fairly resource-hungry.
These are the two directions I’ll be pushing on in the near future.
Back to the beginning: that header Figure 1 DeepSeek put first has, in a way, already drawn the whole argument of this post. The quality of a model’s pre-trained knowledge is locked in at the data-mixture step — it can’t be patched up in post-training, and SFT can barely lift it. The reason SimpleQA-Verified gets neglected is probably exactly that it’s too “naked”: it rewards no tricks, it only measures what and how much you actually fed the model. From data mixture, to semantic deduplication, to our attempt at bucketing data by density × learnability × count and giving each bucket its own sampling rate — all of it is really answering one question: within a finite pre-training budget, what data deserves a second, or tenth, look? To be clear, this framework is nowhere near usable yet — clustering granularity and density quantification are both still unsolved. This post is just my own observations and intuitions, not anything close to a settled conclusion. But I do think it’s the right road.
References
- Bojie Li. Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity. Pine AI, 2026. arXiv:2604.24827.
- Microsoft AI. MAI-Thinking-1: Building a Hill-Climbing Machine. Microsoft, 2026.
- DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. arXiv:2606.19348.
- Artificial Analysis. Omniscience Evaluations. https://artificialanalysis.ai/evaluations/omniscience.
- Lukas Haas, Gal Yona, Giovanni D’Antonio, Sasha Goldshtein, Dipanjan Das. SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge. Google DeepMind / Google Research, 2025. arXiv:2509.07968.
- Zhengyu Chen, Siqi Wang, Teng Xiao, Yudong Wang, Shiqi Chen, Xunliang Cai, Junxian He, Jingang Wang. Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23881–23899, Vienna, Austria, July 2025. Association for Computational Linguistics. Link.