Hacker News
Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
FrontierHarness v1.0 achieved a median cost of $3.24 per successful task on Runta, increasing to $3.24 when failures are included. The benchmark involved 15 passes and a cache hit rate that does not reduce cost, as a 300-turn cached failure still incurred high expense. The study compared nine harnesses—Codex v0.148.0, DeepSeek v0.1.0-rc.8, Claude Code v2.1.237, Pi v0.84.2, Oh My Pi v17.4.0, Kimi Code v0.37.2, Exo v0.1.0, OpenCode v1.18.19, and Hermes v0.20.4—under identical vCPU, memory, disk, and state conditions.