Experiment: Distill Sonnet 4.5 On Solving Mathematics Challenges

Experiment summary.

Frontier models perform well on almost any task, but are extremely expensive. Most production workloads are just one narrow task run many times. If a small open model can be brought up to the frontier model's standard on that one task, the bill for every subsequent request falls significantly.

This experiment is an exploration of whether distilling a model to reduce that cost works. The task is competition mathematics, drawn from the Hendrycks MATH benchmark[1], which consists of 12,500 problems each with a human-written solution. We chose it because its correctness is mechanically verifiable.

We use Sonnet 4.5 as the teacher model, and the students are Qwen3 base checkpoints[2] at the 0.6B and 8B parameter scale.

These models were chosen to demonstrate our distillation capability under a $100 budget, but the same approach works with a stronger teacher such as Fable or Opus, given a larger allocation.

Below are two example problems, with the answers the grader checks against. The first is geometry, the second an algebraic system:

A rectangular box P is inscribed in a sphere of radius r. The surface area of P is 384, and the sum of the lengths of its 12 edges is 112. What is r?

answer: 10

If x, y and z are real numbers for which x + y - z = -8, x - y + z = 18, and -x + y + z = 30, then what is xyz?

answer: 1320

Results.

Sonnet is the teacher model, and measured 86.55% accuracy on the 498-problem hold-out test.

Qwen3-8B started from SFT distillation around 14 points lower, but matches closely after SFT distillation plus GRPO. What is interesting is that distillation added a lot more performance to the small Qwen3-0.6B model than to the larger one. Our hypothesis is that Qwen3-8B already has some of that distillable knowledge preserved in its parameters. An open question is how a full distillation, i.e. just continuing SFT training on Sonnet 4.5 traces, would compare with reinforcement learning, but calling Sonnet is rather expensive, so our budget capped it at 3,000 samples.

Another point is that RL did not seem to help Qwen3-0.6B much after distillation, though it could also be that SFT had already saturated the upper bound of what this model can do. A follow-up ablation would compare it with the same model trained on reinforcement learning from the get-go.

System setup.

We split the experiment into four layers. The control layer does the holdout split, coordinates the other layers, and generates the final report. The training layer runs the training jobs for both SFT and RL, and the storage layer holds the weights and metrics between them. The teacher is handled through OpenRouter, and provides both the reasoning samples and the baseline answers for the final report.

Why distill at all.

If we look at the final distilled and fine-tuned model, Qwen3-8B costs roughly 20 times less than Sonnet, on an estimated 1,000 requests at an estimated 1,000-token cap per request (about 1 million tokens).

Closing thoughts.

We demonstrated that a small open-weight model, trained with distillation and RL, can match significantly more expensive models on a complex task such as solving maths problems. Industry agentic tasks with multiple steps or tool calls are essentially just text, so the trace logs can be used to distill smaller models in a similar manner, as DeepSeek-R1[8] showed at a far larger scale. This approach also scales well to stronger teacher models like Opus or Fable, as done here.

References.

1. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. arXiv:2103.03874. The 12,500-problem benchmark this experiment trains and scores on.
2. Qwen Team. (2025). Qwen3 technical report. arXiv:2505.09388. The base checkpoints used as students.
3. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531. The original statement of the idea: train a small model against a large one's outputs rather than against labels alone.
4. Kim, Y., & Rush, A. M. (2016). Sequence-level knowledge distillation. arXiv:1606.07947. Distillation by supervised training on the teacher's own generated sequences, which is exactly what the SFT stage here does.
5. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155. Supervised fine-tuning on demonstrations as a stage in a language-model pipeline, with reinforcement learning layered on top of it.
6. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., et al. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300. Introduces GRPO, including the objective reproduced above.
7. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347. The clipped surrogate objective GRPO keeps, minus the critic it drops.
8. DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948. GRPO against verifiable rewards at scale, and distillation of the result into smaller open students.