livenerf
A benchmark for tracking frontier model performance changes over time after release
SITE CAPTUREBased on information published on the service's website. Selection and review policy
WHAT IT DOES
What does it do?
livenerf measures model performance changes over time using fixed prompts, a fixed CLI, exact graders, and raw logs. On a frozen evaluation panel, it tracks item-level score differences from the baseline and output token counts. The current v0 uses a logged-in Claude Code session, Python 3.11 or later, and uv.
KEY FEATURES
Key features
- Run repeated evaluations with fixed prompts and a CLI
- Continuously retain raw execution logs
- Compare item-level score differences with the release-week baseline
- Measure statistical drift using cluster standard errors
- Track output token counts for each sample
- Support hourly evaluation runs and a fixed rotating panel
EDITOR'S NOTE
Why did it stand out?
It establishes a baseline soon after release and observes changes in the distribution over time with fixed prompts, a CLI, and graders. Its explicit statement that there are no actual results yet also reflects an effort to turn discussion of performance decline into reproducible measurement.