A pelican riding a bicycle is an excellent conversation starter. It is quick to request, easy to share, and some failures are obvious at a glance. It tells you much less about whether an agent can safely fix your application's import flow.
The Astra quality discussion is a good reason to keep a few repeatable tasks of your own. You do not need a public leaderboard or a large evaluation service. You need a preserved starting state, an outcome you can check, and a record of what happened.
OpenAI's evaluation guidance recommends task-specific evaluations drawn from real use, with human judgment used to calibrate automated scoring. The small test pack below applies that idea to day-to-day Coding Agent work. It is a suggested method, not a benchmark result or a claim that one model wins.
First decide which claim you want to test
“My current setup fails this task more often” and “the model has deteriorated since launch” require different evidence.
To compare two current configurations, give them equivalent inputs now. To establish a historical regression, you also need a preserved earlier baseline: the task, files, instructions, tool environment, and actual results. Memories of an unusually good first week do not provide that baseline.
If you do not have older records, start collecting them. You can still choose the setup that works better today. Label the finding as a current comparison and keep the new material for the next suspected regression.
Also distinguish model evaluation from whole-system evaluation. Changing from one model to another inside the same harness changes fewer variables than moving to a different app with different tools and instructions. Both comparisons can be useful, provided you name what you actually changed.
Build five cases from work you recognize
Start with small cases that expose different kinds of failure. Replace the examples below with representative tasks from your own repository.
The last case needs a saved sequence of messages and tool results, or a reproducible setup script. Giving one agent a long existing conversation and the other a freshly written summary would test different inputs.
Do not include tasks whose answer is already in the current working tree. Do keep the acceptance checks outside the agent's editable target when practical, so it cannot “fix” the assessment by weakening it.
Make one case concrete before adding more
Consider this hypothetical bug: a request helper replaces an explicit timeout of 0 with the default because it uses a truthiness check.
A useful task brief specifies that zero is a supported value, omitted input keeps the default, and existing validation for invalid values must remain. It asks for the smallest appropriate correction and the relevant verification.
Grade the resulting behavior with at least the zero case and the omitted-value case. Inspect the diff for unrelated changes. If the agent edits the test to accept the old behavior, the task has failed even if its own test command is green.
This case is intentionally small. It checks whether the agent can trace a requirement into code and prove the change. A large open-ended task would make it harder to see which part broke.
Keep the starting conditions comparable
For each case, preserve the code revision and any uncommitted fixture changes. Run attempts in independent worktrees, copies, or equivalent isolated environments. Two agents modifying the same live directory cannot provide an independent comparison.
Record the harness and client versions, model identifier, reasoning setting, speed mode, tools, permissions, dependency versions, and effective instructions. A missing permission or failing dependency install can otherwise masquerade as a reasoning failure.
Decide the context condition in advance. Fresh-session attempts and long-session attempts can both matter, but should be separate groups. If network lookups are essential to the task, record what was retrieved or use a stable fixture where possible. Keep the intended test representative of the work rather than controlling away the behavior you care about.
Repeat enough to find a signal, within a budget
One success and one failure tell you what happened twice. They do not establish a reliable difference between two systems.
For an initial screening exercise, try three to five independent attempts per selected case and configuration if the cost is acceptable. That is a practical starting point, not a statistical guarantee. Preserve every attempt. Stop at the planned budget instead of continuing until a preferred configuration wins.
Alternate the order of configurations across runs. If every A run happens in the morning and every B run during an evening incident, time becomes an uncontrolled variable. For high-stakes choices, a small screening set should lead to a broader evaluation with an appropriate sample-size analysis.
The purpose of this first pass is narrower: identify a repeatable failure worth investigating, or a working alternative for today's tasks.
Grade artifacts before reading the model label
Where practical, hide configuration labels from the reviewer until the patch or output is graded. Write the acceptance conditions before seeing the results. This makes an impressive explanation less likely to overshadow a broken implementation.
Automate checks that have a reliable pass condition. Inspect UI states in the browser. Review changes to permissions, data handling, and task scope explicitly. A serious unauthorized change should remain a critical failure; averaging it into a neat overall score can hide the problem.
Keep a worksheet like this:
Environmental blocks should be reported separately, including how often they occurred. Do not quietly discard them from an end-to-end reliability comparison. A model-only question may require rerunning after the environment is repaired; a workflow comparison may need to retain them as part of the user experience.
Report the small result honestly
A useful finding reads like this hypothetical example: “On the preserved timeout fixture, configuration A passed two of five attempts; B passed four of five. Both ran from the same revision with these settings. The sample is small, and we are using B for this task while investigating A's failures.”
That tells another person what to reproduce. “B is twice as intelligent” does not follow from the result.
If the question is whether the backend model changed, add routing evidence. If the question is cost, include all attempts in the cost per accepted task. Public Astra benchmarks can help select candidates, while your task pack checks whether those candidates fit your project.
In Agent.Space, a Workspace can keep the project and task brief available to multiple Sessions. Prepare independent file copies or branches for attempts that write, then give each Session the same starting brief. Shared files are useful for continuity; isolation is something you must arrange for the comparison.
Keep the failing fixtures after the immediate decision. The next client update, instruction change, or model incident will be much easier to assess when you already have work with a known starting point and a checkable finish.
This article presents an original evaluation workflow. The timeout bug and sample pass counts are illustrative, not results measured by Agent.Space. Official guidance checked September 15, 2026.