Recreating Minecraft Is Not a Benchmark

The author argues that “demo-benchmarks” like recreating Minecraft or generating a pelican SVG are easy to overfit, so they no longer measure true model capability. Because static, public test sets quickly become part of training data, only dynamic or hidden evaluations—such as LiveBench, ARC-AGI, or private portions of Humanity’s Last Exam—can reveal genuine performance. Consequently, while demo-benchmarks make compelling content, they should not be used as the primary metric for assessing AI models.