How to evaluate the performance of a theorem proving model?

Evaluating the performance of theorem proving models is an interesting topic, and to be honest, just looking at the accuracy rate may not be enough. Just like the recent DeepSeek open source Prover-V2 model reached such an amazing figure of 88.9% pass rate on MiniF2F test set, but if we carefully disassemble this result, we will find that there are more questions worth thinking about behind. For example, did the model really master the logic of mathematical proofs, or did it just memorize some specific solution patterns? This question has plagued many researchers.

Limitations of the benchmark test set

Current mainstream assessment methods rely on standardized math test sets, such as MiniF2F, PutnamBench, and others. But this is like judging a student by his or her final exam scores - it reflects partial competence, but does not measure true math literacy. Interestingly, DeepSeek-Prover-V2-7B on PutnamBench instead solved some topics that the 671B model couldn't handle, which shows that a large number of parameters is not always a panacea.

One detail I noticed: the skewed distribution of topics in the test set can lead to biased evaluation results. For example, some datasets focus too much on number theory questions, which can make models that specialize in that area seem to perform particularly well. deepSeek's release of ProverBench, which contains 325 questions covering both competitive and undergraduate math, is worthwhile to learn from this diversified design.

In-depth assessment of the quality of proof

Just counting "pass/fail" is a bit too simple, isn't it? The formal proofs generated by a good proof model should also be readable, innovative and generalizable. It's interesting to look at DeepSeek's technical report, which distinguishes between CoT and non-CoT models - the former generating proofs with a complete chain of thought, and the latter generating streamlined code directly. Experiments show that the CoT model works better, suggesting that we should focus on the model's reasoning process when evaluating it.

I especially agree with the DeepSeek team's approach - they not only evaluate the final result, but also check whether the structure of the proof is reasonable, and whether the decomposition of subgoals is clear. It's like a teacher correcting a math problem not only to see if the answer is right or wrong, but also to see if the solution steps are reasonable. Their 7B model is said to have accidentally developed a special skill for dealing with finite-base problems, an "evolved" ability that is easily overlooked in traditional evaluations.

A more comprehensive evaluation system

To fully evaluate a theorem proving model, several key dimensions may need to be considered: first, generalizability, whether it can handle unseen question types; second, innovativeness, whether it can provide novel proofs that no human expert would have thought of; and robustness, whether it can still give correct proofs for slight variations in mathematical expressions.DeepSeek reports that they collected hundreds of high-quality training data, and that such diversity of data is critical to ensuring the reliability of the assessment.

In the end, evaluating theorem-proving models is like evaluating a mathematician - simply looking at how many papers have been published is far from enough. We need a more three-dimensional and dynamic way of evaluating it, both in terms of whether it solves problems, but also in terms of how it solves them, and to what extent it can be used by example. Perhaps this is what teams like DeepSeek are exploring.