PRACTICAL AI GUIDE
I ran the same 14 business questions against the same AI agent three times. The official score went from 0% to 79% to 93%. Neither higher score told me what I needed to know. At 79%, three rejected answers looked identical in the score, yet two were correct answers with missing columns and one was a return rate of 56.89% against a verified 4.11%. At 93%, the only rejection was a correct answer written in a different format. AI agent evaluation is the work of finding those differences before your users do.
Short answer: evaluate an AI agent with a frozen set of real business questions, an answer key you build outside the agent, two separate scores, and a written reason for every failure. A single accuracy number is a starting point. The failures behind it decide whether people can rely on the agent.
This guide uses my own Databricks Genie business-context experiment and the follow-up semantic layer for AI experiment as the worked example. I am not a developer. I lead data engineering teams, and I wrote this for the business owner who will be asked, “Is the agent ready?”
What is AI agent evaluation?
AI agent evaluation is a repeatable test of whether an agent gives the right business result for the questions people will actually ask, using an answer key the business trusts. It measures correctness, behavior under ambiguity and the cost of each failure. A confident reply is not evidence of a correct one.
Vendors use the word “evaluation” for several things: model benchmarks, developer test suites and dashboards that grade responses with another model. Those have their place. The version I care about as a manager is narrower. Before an agent goes in front of people who will act on its answers, can I show that it handles our definitions, our data quirks and our ambiguous questions? If not, which failures remain and how bad are they?
That framing changes who does the work. The data team can build the harness that runs the tests. Only the business can say what the right answer is.
Why can’t you trust a single AI accuracy score?
A single score counts a harmless miss and a dangerous one the same way. In my 79% round, two of the three failures were correct answers missing columns, and the third was a return rate nearly 14 times the verified value. In my 93% round, the only failure was a correct answer.
Here is what happened. In Round B, Databricks accepted 11 of 14 answers. Two of the three rejected answers were right but left out supporting columns, so my review counted 13 of 14. The third rejection was the real problem. Genie wrote the correct return-rate formula in words, then generated SQL that joined two detail tables at the wrong grain. It returned 0.568925, which reads as 56.89%. The verified return rate was 4.11%.
The benchmark caught this one because I had an answer key. A user asking the same question in a chat window would see a plausible number with two decimals and no warning. Nothing about 56.89% looks broken on a screen.
In Round C, I moved the metric math into governed Databricks Metric Views. The tool score rose to 13 of 14. The one rejection was Genie returning 0.041064 while my answer key stored 4.11. Those are the same return rate written two ways. My review counted 14 of 14.
In both rounds, the tool’s number was accurate about what it measured. It measured whether result sets matched. It did not measure whether the business answer was right or how much a wrong answer would cost.
How do you evaluate an AI agent step by step?
Name the decisions, freeze the data, write realistic questions, build the answer key outside the agent, run in fresh sessions, score twice, classify every failure, then change one thing and rerun. The answer key is the slowest step and the one the business has to own.
- Name the decisions. List the questions people will act on. For my fictional B2B company these were quarterly sales, return rate, regional target shortfall, active customers, top customers and revenue concentration. Start from decisions, not from features.
- Freeze the data. I checked the fictional records for contradictions first. I repaired orders dated before a customer signed up, shipments after the reporting cutoff and returns recorded before shipment. Then I stopped changing the files. If the data moves during the test, you cannot tell whether the agent or the data changed.
- Write the questions. Use the wording real users would use, plus alternate wording for the same meaning. Add traps on purpose: joins that can multiply rows, explicit overrides, ambiguous terms and a false premise.
- Build the answer key yourself. I wrote verified SQL for every fixed-answer question and checked the results independently. None of it went into the agent’s instructions. An answer key generated by the agent under test proves nothing.
- Run in fresh sessions. Same settings each time. Fresh chats stop earlier answers from steering later ones.
- Score twice. Keep the tool’s score exactly as reported. Then review each response as a business user would. Report both numbers and say which is which.
- Classify every failure. A wrong number, a missing column and a format mismatch are different problems with different fixes. The next section has the categories I use.
- Change one thing and rerun everything. Between my rounds, only the context changed: first plain-language business rules, then governed metrics. Rerun the full set, including questions that passed, and record the date and settings.
What questions should an AI agent benchmark include?
Include plain questions, alternate wording, grain traps, explicit overrides, ambiguous terms, false premises and at least one question that asks why something happened. Each type catches a different failure, and a benchmark with only easy lookups will overstate readiness.
Two results in that table taught me the most.
The ambiguity test failed twice, but differently. I asked, “Which region performed best in Q2 2026?” With plain-language rules, Genie silently chose target attainment and named West. With governed metrics, it silently chose net revenue and named South. Both answers were mathematically valid under the metric it picked. Neither asked what “best” meant. The right behavior was a clarifying question, and Genie had done exactly that for “Who are our best customers?” in the same runs.
The false-premise test passed. I asked why sales fell in June. Genie checked the governed data and reported that June net revenue had actually risen by $641,399.10 compared with May. An agent that invents a reason for something that did not happen is more dangerous than one that returns an error.
How should you score AI agent results?
Keep the tool’s score unchanged, add a separate business review, and tag each failure with a type and a severity. Severity should reflect what a person would do with the wrong answer, so a plausible wrong number ranks above a correct answer in the wrong format.
These are the failure types I now use. Every one of them appeared in my two experiments.
| Failure type | Example from my tests | Severity |
|---|---|---|
| Plausible wrong answer | Return rate of 56.89% against a verified 4.11% | High. People act on it. |
| Silent assumption | “Performed best” answered without asking which measure | High when the choice changes the answer |
| Setup error | An intermediate run scored 8 of 14 because the wrong metric view was attached | High, and invisible to users |
| Correct but incomplete | Right region and shortfall, two supporting columns missing | Low |
| Format mismatch | 0.041064 against 4.11 | Low for the agent. Fix the answer key. |
| Causal overreach | A correct breakdown of a June to July decline described as “primarily due to” | Medium. Wording implies proof the data lacks. |
The setup error deserves its own note. One intermediate run in my semantic-layer test scored 57%. The agent was fine. I had attached the wrong object, so it could not see the governed target measures, and my customer-activity definition still had a grain problem. If I had reported that run, I would have blamed the approach for my configuration mistake. Record the configuration with every score.
Copy this AI agent benchmark template
This is the structure I keep for each question. A spreadsheet is enough. The point is that every column is filled in before the run, except the last three.
| Column | What goes in it | Example |
|---|---|---|
| ID and question | Exact wording a user would type | Q5: What was our return rate in Q2 2026? |
| Type | Plain, alternate wording, grain trap, override, ambiguity, false premise, cause | Grain trap |
| Business definition | The rule the answer depends on, in the owner’s words | Returns recognized in the period divided by gross revenue from completed orders in the same period |
| Expected result | The verified answer, or the expected behavior for ambiguity | 4.11% |
| How it was verified | Who checked it and how | Independent SQL, reviewed by the metric owner |
| Pass rule | What counts as correct, including tolerance and format | Within 0.01 percentage points. Decimal or percent both accepted. |
| Tool result | Pass or fail exactly as the platform reported | Fail |
| Business review | Your judgment of the answer | Wrong. 56.89%. |
| Failure type and note | Category from the table above, plus one sentence | Plausible wrong answer. Cross join at the wrong grain. |
Writing the pass rule before the run would have prevented my format mismatch. I learned that from the 0.041064 result, not before it.
The error rate you measure here is also the input that decides whether an agent pays for itself. I show how in AI agent cost.
Who should own AI agent evaluation?
The business owner of each metric should own the answer key and the pass rules. The data or engineering team should own the test harness, the configuration record and the reruns. Risk or compliance should review the ambiguity and false-premise cases for decisions with real consequences.
This split matters more than any tool choice. In my view, the fastest way to stall an AI rollout is to make the technical team the judge of business correctness. They will either refuse to sign off or sign off on the wrong thing. Neither is their fault. The definition of net revenue belongs to finance, and the definition of an active customer belongs to whoever is accountable for that number.
This is the same decision-rights problem I describe in The Data-Driven Executive. An agent does not remove it. It makes it visible faster.
What does AI agent evaluation not tell you?
A passing benchmark says the agent handled your test set on the day you ran it. It does not prove the agent will stay correct as data, instructions and platform releases change, and it says little about behavior outside the questions you wrote.
Three limits are worth stating plainly:
- Drift. My tests are dated September 12 and October 2, 2026. Genie has changed since. Rerun the full set whenever the model, the instructions, the data model or the platform changes. For what to watch after launch, see my guide to AI agent observability.
- Coverage. Fourteen fixed-answer questions and a handful of ambiguity and stress tests were enough to expose several failure types. They are not a universal accuracy rate for Genie or for semantic layers.
- Actions. My agent only read data. An agent that can update records needs tests for permissions, refusals and instructions hidden inside documents. I describe those risks in what an AI agent harness does, but I have not run that test yet, so I have no results to report.
Limits of this evidence
I ran both experiments myself in a personal Databricks Free Edition workspace on fictional B2B sales data that I created for the purpose. The dates were September 12, 2026 for the business-context rounds and October 2, 2026 for the governed-metrics round. “Tool score” means the Databricks Chat-mode benchmark. “My review” means my manual classification of whether the business answer was correct, and I report it separately from the official score every time. Agent-mode results used a different evaluation method, so I kept them out of the main comparison.
Official resources on agent evaluation
Anthropic: Demystifying evals for AI agents
Databricks: Genie Agent benchmarks and monitoring
NIST AI Risk Management Framework
If you are building the business case for an agent pilot, the evaluation plan belongs in the approval, not after it. AI to ROI for Business Leaders covers pilot gates and stop criteria, and the free AI ROI Calculator builds the conservative case Finance will ask for.
What are you building, what data will the agent answer from, and which question are you most worried it gets confidently wrong?