A billing pipeline can run successfully and still charge the wrong customer. A usage field can remain an integer while its meaning changes from individual tokens to thousands of tokens. Data contracts help teams make these expectations explicit before another system depends on the data.
I care about this from a data leadership perspective because a successful job run tells me very little about whether the business can trust its output. The questions are more useful: What does each record represent? Which changes require review? Who responds when the data breaks the agreement?
What is a data contract?
A data contract is an agreement between a data producer and its consumers that defines the data’s structure, meaning, quality expectations and responsibilities. It also explains how changes and violations will be handled, with automated checks enforcing the requirements that can be tested.
The producer supplies the data. The consumer uses it in a report, application, model or business process. The same team can be a consumer of an upstream dataset and a producer of a transformed dataset further downstream.
Think of a usage feed delivered to billing. Knowing that usage_units is an integer is useful. Knowing that it counts individual tokens, excludes test requests and represents one metered event is what makes the number usable.
A simpler example is a spreadsheet sent from Sales to Finance each morning. Finance expects one row per completed order, amounts in US dollars and cancelled orders excluded. Those expectations form the basis of a data contract, whether the delivery is a spreadsheet, a database table or a stream of events.
The Open Data Contract Standard provides a common format covering areas such as schema, quality, roles, support and service expectations. A standard helps teams describe the agreement consistently. A separate implementation must execute its checks and handle failures.
Why do data contracts matter?
Data contracts matter because downstream systems depend on assumptions that an upstream team can change without realizing the consequences. Making those assumptions explicit gives teams a basis for reviewing changes, detecting violations earlier and assigning responsibility before incorrect data reaches an important business decision.
A renamed field may cause an obvious failure. A changed definition can be harder to notice. If an upstream team starts including trial accounts in an active-customer feed, the pipeline may keep running while a management metric changes meaning.
Inventory provides another example. A warehouse reports available_quantity = 100. Does that mean 100 individual items, 100 cases or stock remaining after customer reservations? The number can be technically valid under all three interpretations, while purchasing decisions differ. The contract should settle the unit and definition before the buying team uses it.
The problem predates today’s platforms. Teams have long maintained interface specifications, database constraints, reconciliation checks and service agreements. Data contracts bring these expectations together around a particular data handoff.
My view is that their value comes from the conversation and the controls they create. A file named contract.yaml cannot settle a disputed business definition or make someone respond to an incident.
What does a data contract look like in practice?
A practical data contract describes a specific dataset, its record grain, important fields, business definitions, quality rules, delivery expectations and accountable roles. It should also state what the consumer can rely on, which changes need coordination and what happens when the producer cannot meet the agreement.
Consider a fictional subscription business that bills for measured usage. The metering service produces usage events. The billing pipeline consumes them and joins them to a customer reference. This is an illustrative scenario, not a description of a particular company’s architecture.
| Dataset | Record grain | Relationship and purpose |
|---|---|---|
| Usage events | One row per metered request | Connects to the customer reference through customer_id; supplies quantities for billing |
| Customer reference | One row per customer | Identifies valid customers and the applicable illustrative pricing information |
For the usage feed, the agreement might contain the following requirements. These are example choices, not universal rules for every subscription business.
| Contract element | Illustrative agreement | Why it matters |
|---|---|---|
| Grain and identity | One metered request per event_id; identifiers remain stable during retries | Makes duplicate detection possible |
| Business meaning | usage_units counts individual tokens, never thousands of tokens | Prevents an agreed unit from remaining implicit |
| Required fields | event_id, customer_id, occurred_at, usage_units and unit | Defines the information the consumer needs |
| Relationships | customer_id must exist in the approved customer reference | Flags events that cannot be assigned through that reference |
| Time and delivery | Timestamps use UTC; late events follow an agreed correction process | Clarifies period assignment and incomplete deliveries |
| Failure response | Withhold the affected billing batch and assign investigation to the metering owner | Connects detection to an operational decision |
| Change handling | Review unit changes, required-field removals and altered event meaning before release | Gives consumers time to adapt |
For example, an illustrative usage record might contain event_id = E101, customer_id = C021, usage_units = 1200 and unit = token. A required-field check can detect a missing customer identifier. A relationship check can detect C021 if it is absent from the customer reference. Neither proves that this request actually belonged to C021.
Written down, a simplified version of that agreement might look like this. It is a custom illustration, not an Open Data Contract Standard document or executable configuration for the downloadable experiment.
dataset: usage_events
grain: one row per metered request (event_id)
owner: metering team
consumers: [billing]
fields:
event_id: {type: string, required: true, unique: true}
customer_id: {type: string, required: true, references: customer_reference.customer_id}
occurred_at: {type: timestamp, required: true, timezone: UTC}
usage_units: {type: integer, required: true, min: 0, meaning: individual tokens}
unit: {type: string, allowed: [token]}
on_violation: withhold billing batch, notify metering owner
changes_requiring_review: [unit, required fields, event meaning]In this example, event uniqueness must be checked within the incoming batch and against previously published event identifiers. The word unique does not implement that history check by itself. The downloadable experiment uses a separate custom JSON contract and Python validation code.
The grain deserves attention before the rules. One row per request and one row per customer per day are different datasets. A uniqueness check that makes sense for one can reject valid records in the other.
How is a data contract different from a schema or data-quality test?
A schema describes the structure of data, while a data-quality test checks a particular condition. A data contract connects those technical expectations to business meaning, producer and consumer responsibilities, delivery commitments and change handling, so the checks have an agreed purpose and a response when they fail.
| Component | Question it answers | Usage-feed example |
|---|---|---|
| Schema | What fields and types are expected? | usage_units is an integer |
| Quality check | Does this data satisfy a test? | No duplicate event_id values appear in the batch |
| Data contract | What has been agreed, and how is it maintained? | Event identity survives retries; replay checks use published history; violations delay billing and have an owner |
A sales amount makes the distinction easy to see. The schema says amount is a decimal. A quality test checks that it is nonnegative. The contract explains that it is the order value after discounts, before tax, in US dollars, and assigns responsibility for changing that definition. All three contribute different information.
You may already have useful checks in your pipelines. Turning them into a contract involves agreeing on their meaning and consequences with the producer and affected consumers. Rewriting every existing test into a new tool is not a prerequisite.
A schema registry can support the agreement, and some registries support additional rules and metadata. Confluent’s data-contract documentation describes structure, integrity constraints, metadata, policies and evolution. The capabilities available depend on the implementation.
Who owns a data contract?
The producer should be accountable for the data it supplies, while consumers help define the requirements their workflows depend on. A named role should maintain the contract, and business owners should approve consequential definitions and exception decisions rather than leaving those choices entirely to the pipeline developer.
In the billing scenario, the metering owner can explain how usage is captured. The billing owner can explain which events should be charged and how corrections affect customers. The data team implements the checks and provides evidence when they fail.
Suppose Sales changes the definition of an active customer from someone who bought in the last 12 months to someone who logged in during the last 30 days. The engineer can update the query, but the decision affects business reporting. The responsible business owner should approve the definition, while producers and consumers coordinate the change and its effective date. My data governance framework covers how to assign those decision rights.
I would make those responsibilities specific. Who investigates an unexplained usage change? Who can approve a temporary exception? Who decides whether corrected data requires a revised bill? Calling all three teams owners does not answer these questions.
In the fictional business story in AI to ROI for Business Leaders, the business leader encounters competing ownership claims from Sales, IT and Finance. The useful move is to assign concrete responsibilities for the data the business needs. A data contract applies that discipline to a particular handoff.
Where should data contract checks run?
Data contract checks should run where they can detect a violation before an affected consumer relies on the data. Depending on the requirement, that can mean reviewing proposed changes before deployment, validating incoming records or batches, and monitoring published datasets for delivery, volume or distribution problems.
These checks answer different questions. A release check can identify removal of a required field. An ingestion check can flag an unknown customer. A scheduled monitor can detect that an expected delivery never arrived. One cannot automatically substitute for the others.
Consider a daily order feed expected by 7 a.m. The deployment check catches a proposed rename of customer_id. The ingestion check flags an order with an unknown customer. At 7 a.m., a delivery monitor checks whether the feed arrived at all. An empty folder does not contain a bad record for the ingestion validator to inspect.
Every check also needs evidence appropriate to its scope. Detecting a replay across batches requires access to prior event identifiers. Detecting a missing delivery requires a delivery expectation. Checking only the records that arrived cannot establish that all expected records arrived.
What should happen when a data contract is violated?
A violation should trigger an agreed response based on the consequence of using the affected data. The response may block publication, quarantine records, retain the last approved dataset or raise a warning, but it should identify an owner, preserve evidence and define the conditions for recovery.
I would be cautious about silently dropping invalid billing events. The remaining records might look clean while the invoice becomes incomplete. Blocking the whole batch avoids that particular publication risk, but delays valid records too.
For an internal dashboard, the agreed response might retain yesterday’s approved figures and display their date. For customer billing, the team might withhold the affected batch until an owner investigates. For a missing optional product description, a warning might be enough. These are example policies; the consequence of using the data should determine the response.
There is no universal failure policy. An exploratory dashboard and customer billing can justify different responses. The business owner should understand the tradeoff, and an exception should have an approver, a reason and an expiry or review point.
Can data contracts prevent every data problem?
Data contracts cannot prove every business fact simply by validating fields and types. Their protection depends on the requirements expressed, the checks actually executed and the evidence available, so missing events, incorrect attribution or an undisclosed change in measurement can pass a contract that does not test those conditions.
A known customer identifier proves that the customer exists in the reference. It does not prove that the usage belongs to that customer. A positive integer proves that a quantity satisfies a format and range rule. It does not prove that the producer measured it correctly.
Suppose a feed contains 99 valid orders, but the source system recorded 100. Every delivered row can pass its checks while revenue is understated. A source control total can expose the missing order. Simply requiring nonblank order identifiers cannot, because the absent record never reaches that check.
Stronger contracts can include completeness controls and reconciliation requirements. They still need an independent signal to test against. If the feed and its reference both inherit the same upstream mistake, agreement between them can be misleading.
This is where human review remains useful. People should review disputed meaning, new business behavior and consequential exceptions. The contract makes the routine expectations inspectable; it does not remove business judgment.
When should a team start using data contracts?
Start with a data handoff where a failure has a clear business consequence and the producer and consumers can agree on a small set of expectations. Prioritize recurring defects, consequential definitions and changes that have surprised downstream teams, then add coverage as actual failures reveal what is missing.
A useful first candidate is a daily sales feed that repeatedly confuses order date with shipment date. Agree on which date drives the report, how returns are represented and who approves corrections. A rarely used archive with no active consumers may deserve less attention. I would prioritize the consequence of a failure over the number of columns a dataset contains.
I would begin with one dataset, its grain, a few important definitions, the checks needed for its intended use and a named failure-response owner. Then deliberately test a broken delivery and a proposed change. A contract that nobody has exercised is still an assumption about how the organization will respond.
In my data contract validation experiment, I test 13 usage-billing scenarios with local Python and SQLite, comparing an unguarded pipeline, schema checks and a broader contract gate. The follow-up includes the code, example data, measured results and the problems that still passed validation.
For the broader leadership decisions around ownership and trustworthy information, see The Data-Driven Executive. For assessing whether your data can support a particular AI use case, read my data readiness for AI guide.
Official resources
- Open Data Contract Standard: specification sections and examples for describing a contract.
- Confluent data-contract documentation: one implementation of schema, rules, metadata and evolution controls.
What are you building, what data does it depend on, and which blocker would you want a data contract to catch? Share your scenario in the comments.