Claude AI Failed 650 Times...Then Beat The Human Record

Summary

An unreleased version of Claude was given the task of solving the Riemann hypothesis, a long-standing mathematical problem about the distribution of prime numbers. No human has ever been able to prove it, and Claude did not solve it either. However, it was able to improve a related bound beyond the human record, moving from 41.6 percent to 67.2 percent, an increase of 25.6 points in one go. Mathematicians such as James Maynard and Andrew Sutherland reacted with surprise and described the result as a genuinely interesting mathematical contribution and a massive leap forward.

The experiment began when a non-mathematician named Jarred prompted the AI to try a famous unsolved math problem. The first 650 tries did not work. During the process, Jarred provided messages of encouragement such as keep going and believe in yourself. Claude had internet access available but did not need to use it during the key breakthrough run. It went down many wrong roads first, learned from them, and recovered. After about 37 minutes of radio silence, the first crucial result appeared, and the AI itself noted that the result was suspiciously good.

The full technical paper titled More Than Two Thirds of the Zeros of the Riemann Zeta Function Are Simple and on the Critical Line is available, co-authored with Dr. Caro Joona Ifair. Additionally, a Lean 4 formalization of the proof is available and can be automatically verified. The video notes that similar encouragement prompts were previously used to help Claude disprove the Jacobian conjecture.

To manage and evaluate machine learning applications, Weights and Biases offers Weave, a lightweight toolkit. Weave allows developers to log and debug inputs, outputs, and traces, drill into failure modes to find hallucinations, and use evaluations to measure performance. Developers can try out Weave at wandb.me/papers.