The fastest way to lose a quarter on an AI project is to pick the model architecture before you have an eval set. The same anti-pattern shows up across the market: a team has stood up a proof of concept, the demo went well in the room, and the conversation has already moved on to which agent framework to use and how to chain the tools. Nobody has written down what “working” means on this data.
You will discover what working means anyway. You will discover it the expensive way, during integration, when the first stakeholder runs a real query against the real system and the answer is wrong in a way the demo never showed. By then the team has committed to the framework, the prompt scaffolding, and the model vendor. Backing any of that out is a rebuild.
What an eval set is, concretely
An eval set is a fixed collection of inputs paired with the criteria you will use to judge the outputs. For a customer service triage agent it might be 300 historical tickets, each annotated with the correct routing decision, the relevant policy that applied, and the expected escalation path. For a contract review system it might be 80 clauses annotated with what to flag, what to ignore, and what the safe edit would be. For an internal Q&A system it might be 200 questions with the authoritative answer pulled directly from the source documents.
The set has to come from your data, not a public benchmark. The point is to measure quality on the actual job, not on a job that resembles the actual job from a distance. Public benchmarks are useful for ranking vendors at a coarse grain. They cannot tell you whether the system can do your work.
Why this comes first
Three things change the moment you have an eval set on your own data:
-
The architecture conversation gets shorter. Most “should we use framework X or Y” debates dissolve when you can run both against the same 300 inputs in an afternoon. The answer is almost always boring. Pick the one that scores better on your data; ignore the rest of the discourse.
-
You can ship without holding your breath. A change that improves the system on the eval set is a real improvement. A change that drops the score is a regression you found before production found it. You stop relying on whoever was in the room when the model was last tested.
-
Stakeholders can sign off on something concrete. “The system answers 91% of tier-2 tickets within policy on the held-out set, with full source citation, and routes the remainder to a human within 90 seconds” is a sentence you can put in front of compliance. “We tried it and it seemed to work” is not.
What we see when teams skip it
The pattern is consistent. A team spends six to eight weeks building agent scaffolding, integrating tools, tuning prompts, and demoing to stakeholders. Quality is judged in vibes. The system goes into a limited rollout. A stakeholder runs a query that no one happened to test. The answer is confidently wrong. Trust collapses overnight.
The recovery costs more than the original build. Now the team has to retrofit evals onto a system that was not designed to be tested, untangle which prompt is responsible for which behaviour, and rebuild stakeholder confidence from a worse starting point than they had at the kickoff. That recovery work commonly costs several times the original build, because the original build has to be largely undone first.
What the first week of a project should look like
If we are running a diagnostic, the first artifact we produce is the eval set. We pull 100 to 500 representative examples from your historical data with your subject matter experts in the room. Each one gets an expected outcome, a rationale, and an edge-case classification. The set then becomes the ground truth for every architectural decision that follows.
The set is also versioned. When the business changes, the set changes. When a new failure mode shows up in production, that example gets added to the set so the next regression catches it. The eval set is a living artifact, not a deliverable you write once and forget.
The economics
The eval set is the cheapest single thing in the project. It is a week or two of focused work with people who already know the domain. The architectural decisions it informs cost ten to a hundred times more to revisit later. The deployments it derisks cost more again to recover when they go wrong in front of the customer.
There is no version of this calculation where “we will add evals later” wins. Later is the most expensive time to add them, because by then the system has been built around the assumption that evals were not necessary. Start with the evals. Let everything else follow.
What this means if you are about to start
Before you choose an agent framework, write the eval set. Before you choose a model, write the eval set. Before you choose a vector database (the search index underneath retrieval), write the eval set. Before you do anything that costs more than a week to undo, write the eval set.
Then choose whatever scores highest on your data and move on. The rest of the discourse can wait.