Can it do it again?
A coding agent fixing one bug is encouraging. I want to know what happens on the next attempt, with another model or a slightly different task. That is what led me to put more repeatable checks around the setup.
Start small
The smallest checks look at whether provider replies and tool calls are read correctly. Then come quick connection checks and longer tasks against actual code. Each layer answers a different question, and the cheap failures should be easy to find.
I care about consistency as well as the best result. A task that works once in several attempts needs different treatment from one that succeeds reliably.
Use failures to improve the setup
These checks cannot prove that an agent will handle every future job. They can tell me when a provider change breaks a tool, when a task has become flaky, or when yesterday’s working setup has regressed. That is enough to make them useful.