Databricks Genie Tutorial: How Business Context Improved Accuracy From 0% to 93%

PRACTICAL AI EXPERIMENT

This Databricks Genie tutorial tests a simple question that matters far more than whether an AI tool can write SQL: does the agent understand what the business means? I gave the same Genie Agent the same six tables and fourteen benchmark questions twice. The data stayed fixed. The questions stayed fixed. The meaningful change was the business context.

The result was substantial. With neutral table information, Databricks scored the benchmark at 0%. After I added the company’s definitions for revenue, returns, active customers, targets and reporting periods, Databricks reported 79% accuracy, or eleven accepted answers out of fourteen.

However, my manual review found that two of the three rejected responses still answered the business question correctly. Therefore, thirteen of fourteen answers were business-correct by my review, or 93%. The remaining failure was more important than the score: Genie stated the correct return-rate formula but generated SQL that returned 56.89% instead of the verified 4.11%.

New to Databricks Genie? Start with my Databricks Genie tutorial for beginners. It introduces the interface, source-table selection, business instructions, prompts and generated SQL through a simpler walkthrough. Then return here for the six-table benchmark and a closer look at what works well and what can still fail.

Databricks Genie tutorial experiment at a glance

There is no universal accuracy number for Databricks Genie. Results depend on the data model, business definitions, instructions, trusted logic and the questions users ask. In this controlled experiment, the official benchmark moved from 0% to 79% after business context was added. Meanwhile, my separate manual review classified thirteen of fourteen fixed-answer questions as business-correct.

Does business context improve Databricks Genie accuracy?

In this experiment, yes. The data and questions stayed the same while the official benchmark moved from 0% to 79% after company-specific definitions were added. However, this does not mean every Genie implementation will improve by the same amount. It shows how strongly business context affected this controlled test.

Data access and business understanding are different problems. A table can clearly identify an order date, ship date, status and price while leaving the meaning of revenue unresolved. For example, one company may recognize sales when an order is placed, while another may use the shipment date. Returns may also be deducted from the original sale period or from the period in which the return occurred.

The same problem appears in other metrics. For example, “active customer” could mean a customer with an order in the last thirty, sixty or ninety days. “Highest-value customer” could refer to lifetime revenue, recent revenue, margin or order frequency. Likewise, “best-performing region” could mean the most revenue, the fastest growth or the highest target attainment.

Because of those differences, I wanted to test whether Genie would answer consistently without the definitions, how much its performance would change after receiving them, and whether clear instructions were enough to prevent incorrect SQL.

Meet the fictional company and its data

Before looking at the agent, it helps to understand the business it was asked to analyze. The experiment represents a fictional B2B company selling products to small, mid-market and enterprise customers across four US sales regions: Central, East, South and West. The reporting data covers customer signups, orders, individual order lines, products, returns and monthly regional sales targets.

I loaded the six Delta tables into workspace.genie_sales. Although the names are simple, each table has a different grain. As a result, a careless join can duplicate revenue or returns.

How customers and orders connect

First, the customer table supplies the descriptive context that business users recognize. A customer belongs to one segment and one region, and can place many orders.

Next, the orders table supplies the customer, order date, ship date and status. Therefore, this is where Genie must decide which date controls a calculation and whether cancelled or pending orders belong in revenue.

How revenue is built from order lines

Order items hold the values needed for the revenue calculation. Each line can have a different quantity, price and discount. Therefore, gross revenue in this experiment is calculated at line level as quantity × unit_price × (1 - discount_pct) before it is aggregated.

After that, product IDs become meaningful by joining to the product master, which supplies product names and categories. As a result, category questions depend on this table.

Why returns and sales targets need different treatment

In contrast, returns are linked to the original order and product, but their financial effect is recognized on the return date. That timing rule matters because a product can be sold in one reporting period and returned in another.

Meanwhile, sales targets sit at a different grain: one row for each region and month. Therefore, revenue must first be aggregated to that same region-month level before it is compared with the target.

In practice, this is the task Genie received. It had to select the correct date, include only the relevant order statuses, calculate discounted line revenue, subtract returns in the correct period and respect the grain of the target table. Therefore, the benchmark tested those decisions rather than only checking whether the agent could produce valid SQL.

To reproduce the previews, open Catalog, expand workspace.genie_sales, select each table and open the Sample Data tab. These previews did not require a Genie prompt.

How I controlled the Databricks Genie experiment

I also added three ambiguity questions in Agent mode. However, these did not have SQL answers because the expected behavior was to ask the user for clarification instead of calculating a result from an assumed definition.

The exact prompts used in the benchmark

I entered the following fourteen questions as Chat-mode benchmarks. In addition, alternate wordings were included to check whether the same business meaning produced the same result.

Revenue and target prompts

Customer prompts

Metric and override prompts

Ambiguity prompts

Finally, I entered these three questions in Agent mode. Passing meant asking for clarification instead of silently choosing a definition.

Preparing the data for a fair test

To limit avoidable differences in the calculations, I loaded the tables with explicit decimal, integer and date types.

Next, I checked the fictional records for contradictions before loading the final dataset. I repaired orders dated before customer signup, completed orders with shipment dates beyond the reporting cutoff and returns recorded before shipment. I then froze the files and calculated the expected answers with verified SQL.

As a result, the questions tested the agent rather than known defects in the sample data. They covered quarterly revenue, gross sales, return rate, active customers, trailing-twelve-month customer value, customer concentration, regional target performance, product-category performance and an explicit request to use ship date instead of order date.

Can Databricks Genie query tables without business definitions?

Yes, Genie could read the tables and generate SQL, but that was not enough to reproduce the company’s intended answers. At baseline, the agent had no company-specific definition for the metrics being tested. For example, it was not told that an ordinary request for “sales” meant net revenue, or that returns should be recognized using the return date rather than the original order date.

Consequently, Databricks reported 0% accuracy, or zero accepted answers out of fourteen. This did not mean Genie was unable to write SQL. Instead, the results did not match the definitions embedded in the verified benchmark answers.

That distinction is important. The baseline measured agreement with this fictional company’s rules, not general SQL ability. In fact, several benchmark questions were intentionally impossible to interpret correctly without definitions that had been withheld from Round 1.

What business context did I add to Databricks Genie?

For Round 2, I added one organized block of instructions to the same Genie Agent. The most important rule defined net revenue as gross revenue from completed orders in the reporting period minus returns recognized during that period.

In addition, gross revenue meant quantity multiplied by unit price after discount but before returns. Sales reporting used order date unless the user explicitly requested ship date. An active customer needed at least one completed order during the ninety calendar days ending on the stated date.

Finally, the instructions defined highest-value customers using trailing-twelve-month net revenue, explained the monthly and regional grain of sales targets, and specified how to calculate target attainment and missed-target dollars.

Round 2 instruction prompt

I pasted the following text into the agent’s instruction field without changing the benchmark questions:

Business context
This is a fictional B2B company. Reporting data is complete through 2026-08-31.

Business definitions
1. Completed order revenue
Include only orders where orders.status = 'COMPLETED'.

2. Gross revenue
Gross revenue = SUM(order_items.quantity * order_items.unit_price * (1 - order_items.discount_pct)) for completed orders.

3. Returns
Recognize a return in the period containing returns.return_date.
Return value = SUM(returns.return_amount).

4. Sales / revenue
Unless the user explicitly asks for gross sales or gross revenue, interpret "sales" and "revenue" as NET REVENUE.

5. Net revenue
Net revenue for a reporting period = gross revenue from completed orders whose orders.order_date falls in the period MINUS return_amount for returns whose returns.return_date falls in the period.

6. Date basis
Use orders.order_date for sales reporting unless the user explicitly asks to use ship_date. If the user explicitly asks to use ship_date, use orders.ship_date for the order-revenue portion. Returns are still recognized using returns.return_date.

  

Customer and target rules

7. Active customer
An active customer as of a specified date is a customer with at least one COMPLETED order in the 90 calendar days ending on and including that date.

8. Highest-value customer
When the user asks for "highest-value customers", rank customers by trailing-12-month NET REVENUE ending on the specified as-of date.

9. Top-N customer revenue share
For an as-of date, calculate each customer's trailing-12-month net revenue. Top-N share = net revenue of the top N customers / total net revenue for all customers over the same trailing-12-month period.

10. Sales targets
sales_targets.target_revenue is monthly and regional. Aggregate targets over the same months and region as the revenue being compared.

11. Target attainment
Target attainment = net revenue / target_revenue.

12. Missed target by dollar amount
For regions below target: missed target amount = target_revenue - net revenue. The region with the largest positive missed-target amount missed by the most dollars.

13. Return rate
Return rate for a period = return_amount recognized in that period / gross revenue from completed orders in that period.

  

Ambiguity and period rules

Ambiguity rules
- If the user asks for "best customers" without defining best, ask which metric they mean.
- If the user asks "how much did sales grow?" without specifying comparison periods, ask for the periods.
- If the user asks which region "performed best" without specifying the performance metric, ask whether they mean revenue, target attainment, growth, or another measure.
- Do not silently invent a business definition when the question is genuinely ambiguous.

Period conventions: use calendar quarters. Trailing 12 months as of August 31, 2026 is September 1, 2025 through August 31, 2026 inclusive. All amounts are USD. Gross revenue in this experiment is after discounts but before returns, a company-specific convention. Aggregate regional revenue and regional monthly targets separately to the same region and period before comparing them.

Databricks recommends using a focused set of instructions for context that should apply across the agent. It also supports verified example queries and functions when a calculation needs more control. However, I kept those out of this round because I wanted to isolate the effect of plain-language business definitions.

How accurate was Databricks Genie after adding business context?

Databricks accepted eleven of the fourteen fixed-answer questions after the business rules were added. Therefore, the displayed benchmark score increased from 0% to 79%. My later manual review reached a different figure because I judged whether each response answered the user’s business question, while the Chat-mode benchmark compared generated result sets against the SQL answers I supplied.

Why did Databricks report 79% while my review found 93%?

I reviewed the three rejected responses because a platform score does not explain whether the business conclusion was wrong, the SQL was wrong, or the output simply differed from the expected shape.

The first two answers directly answered what the user asked. However, the platform rejected them because the result sets did not contain all columns in the ground truth. Therefore, I counted those as business-correct answers while retaining the official Databricks score separately.

Can Databricks Genie generate incorrect SQL?

Yes. The genuine failure in this experiment was the return-rate question. Under the instructions, return rate meant the return amount recognized during the period divided by gross revenue from completed orders during the same period. Using that definition, the correct Q2 result was 4.11%.

However, Genie created one common table expression containing order-item revenue and another containing individual returns. It then cross-joined the detail rows and summed the multiplied result. As a result, the generated output was 0.568925, or 56.89% when interpreted as a percentage.

The written formula was correct, but the SQL implementation was not. This is the part of the experiment I found most useful. Business instructions greatly improved the results, yet they did not protect the calculation from a join-grain error.

Therefore, for important metrics, I would add verified SQL examples or governed functions and keep benchmark tests around them. I would also review generated SQL for join behavior, aggregation order and date boundaries before using the answer in an operating decision.

When should Databricks Genie ask for clarification?

Genie should ask for clarification when a reasonable business user could interpret the same term in more than one materially different way and the missing definition would change the answer. Therefore, the last part of the experiment tested exactly that. I added three questions in Agent mode and provided evaluation notes describing the expected behavior.

Genie correctly asked what “best customers” meant and requested the comparison periods for sales growth. However, for regional performance, it silently selected target attainment and named West. South had the highest net revenue, so a different reasonable interpretation would have produced a different answer.

When is Databricks Genie accurate enough for business use?

I would use Genie as a conversational layer over governed analytics for exploration and for recurring questions that have been tested. However, I would not treat every newly generated answer as a certified metric automatically. In this experiment, 56.89% looked like a normal numeric result even though the verified return rate was 4.11%.

For metrics used in financial, customer or operating reviews, I would start with governed definitions, explicit grain and relationships, realistic benchmark questions and verified logic for the calculations that matter most. In addition, I would inspect generated SQL for join behavior, aggregation order and date boundaries, then rerun benchmarks when the data model or context changes.

Human review still has a role. Automated comparison quickly exposed result-set differences, while manual review showed that two rejected answers had actually answered the user’s question correctly. Therefore, the two methods were measuring different things, and I would keep both in a production evaluation process.

Databricks’ current guidance follows a similar testing pattern. Its documentation recommends realistic questions, alternate phrasings, review of generated SQL and benchmark suites that are rerun as the agent changes. In addition, for recurring calculations, Genie supports example SQL and trusted assets whose logic has been verified by an author. See Databricks guidance for a Genie Agent and Databricks benchmark and monitoring documentation.

This follows the same pattern I saw in my AI accounts payable experiment and AI contract-review experiment. Supplying source data is only part of the task. The system also needs the rules used by the people who review that data. Then its output needs to be tested against known cases.

What changed in Databricks after this experiment?

I ran this test on September 12, 2026. Since then, Databricks has described Genie Ontology as a shared context layer that combines governed Unity Catalog meaning with context drawn from assets such as metric views, dashboards, SQL queries and Genie Agents. That direction reinforces the problem tested here: table access alone does not tell an AI system which business definition or source should be trusted.

Databricks has also published an internal 28-question enterprise data-analysis benchmark in which Genie with Ontology answered 84.5% correctly on the first attempt, compared with 52.4% for the strongest general-purpose coding agent in that evaluation. However, those are Databricks-reported results from a different benchmark, not results from my experiment. Therefore, they should not be treated as a universal Genie accuracy rate. See Databricks’ published benchmark and Genie Ontology announcement.

My test should also not be read as an evaluation of the full Genie Ontology setup. I did not separately build and test ontology snippets, metric views, Unity Catalog Pages or a production collection of company assets. Instead, I deliberately focused on whether adding explicit business instructions changed answers over the same six-table dataset.

I later tested that question separately in a semantic layer for AI experiment, replacing the raw-table setup with governed Databricks Metric Views and rerunning the same 14-question benchmark.

Limitations of this Databricks Genie test

What this Databricks Genie experiment showed

If I continue this experiment, I would test a third round with one verified SQL example for the return-rate calculation, then rerun all seventeen questions. In addition, I would add fiscal-calendar rules, customer hierarchies and questions that combine several metrics in one request.

For this test, the main result was clear. Business context changed the Databricks benchmark from zero accepted answers to eleven of fourteen, while my manual review found thirteen of fourteen answers business-correct. However, the remaining SQL failure matters because it shows why business definitions, verified logic, benchmarks and human review solve different parts of the same problem.

This experiment was run on September 12, 2026. Databricks product behavior and benchmark scoring may change as Genie Agents are updated. For current product details, see Databricks official documentation: Genie Ontology, Databricks official documentation: benchmark monitoring and Databricks official documentation: Genie Agent best practices.