Data Contract Validation: What My Billing Experiment Caught and Missed

Data contract validation blocked 8 of 11 deliberately broken usage-billing batches in this experiment. Three still passed and produced incorrect customer charges. Both valid control batches passed, and the same billing calculation was used across all three pipeline configurations.

The largest escaped error reduced the correct invoice from $6.70 to $4.30. Every delivered record passed validation. One usage event was missing, and the gate had no independent completeness check.

I wanted to examine a management question behind the technical checks: what evidence allows a team to trust a billing feed? A pipeline can reject obvious errors and still publish a plausible number that is wrong.

If the concept is new to you, start with my guide to data contracts. This follow-up tests a small implementation and explains what its failures reveal.

What did this data contract validation experiment test?

The experiment tested whether explicit validation rules could stop defective usage data before a billing calculation published customer charges. It compared an unguarded pipeline, a schema gate and a broader contract gate across 13 controlled scenarios, including duplicates, replayed events, changed units, incorrect customers and an omitted event.

The question was not whether a contract file could describe a dataset. It was whether executing its rules changed the outcome, and which errors remained outside those rules.

The recorded run took place on October 8, 2026, using Python 3.12.14 and SQLite 3.53.1. All customers, events and prices are fictional teaching examples. No employer data, model calls, paid APIs, cloud services or Databricks workspace were used.

Download and run the experiment

The package contains the CSV scenarios, custom JSON contract, Python runner, billing SQL, acceptance checks and recorded results. It requires Python 3.10 or newer and uses only the standard library, so no package installation is needed. Enter your email below and I’ll send you the download link.

Extract the ZIP, open a terminal in the extracted data-contract-experiment folder, and run:

python experiment.py
python verify_experiment.py

On Windows, use py if python is unavailable. On macOS or Linux, use python3 if needed. The first command runs the experiment; the second checks its expected outcomes. The runner reads the shipped files without repairing them.

What data feeds the billing calculation?

The billing calculation uses usage events joined to a customer reference containing illustrative rates. Each event represents one metered request, while each customer-reference row represents one customer; a separate reference ledger supplies the expected event population for post-run reconciliation and is excluded from the validation gate.

The fictional business charges for token usage. That is a convenient example of measured consumption, not an attempt to reproduce a particular AI provider’s pricing or billing architecture.

Customer-reference preview

Clean usage and reference-ledger preview

The clean usage file and reference ledger contain these same six records. The ledger remains unchanged when a scenario alters the incoming file.

In the scenario metadata, the clean batch has no prior events. The replay scenario supplies E001 as already published, then sends a batch containing E001 again. The reference population still contains each event once.

How did I compare the three pipelines?

Each scenario ran through three configurations with identical customer prices and downstream billing logic. Only the validation gate changed: the baseline had none, the schema gate checked basic structure and types, and the full contract gate added identity, reference, range, declared-unit and billing-window requirements.

There were two valid controls: the unchanged batch and a batch with an additional permitted column. Eleven other scenarios introduced one defect each. Thirteen scenarios across three configurations produced 39 pipeline runs.

The baseline is deliberately weak. Existing production pipelines may already have equivalent controls without calling them a data contract. These results compare the three implementations here, not every approach to data engineering.

Contract configuration

This excerpt shows selected executable settings from contract.json. The complete file also records illustrative ownership roles, business meaning, change handling and failure policy. The format is custom and is not ODCS compliant.

{
  "allow_extra_columns": true,
  "integer_field": "usage_units",
  "minimum_usage_units": 0,
  "allowed_units": ["token"],
  "billing_window_start": "2026-10-01T00:00:00Z",
  "billing_window_end_exclusive": "2026-10-02T00:00:00Z"
}

The end of the window is exclusive: an event at midnight on October 2 does not belong in this batch. This example deliberately rejects out-of-window events; a real late-arrival process would need an agreed correction path.

Replay validation

The validator initializes its seen identifiers from supplied prior history, then checks incoming events. These are excerpts from the same function in experiment.py:

seen={r['event_id'] for r in prior}

# Inside the incoming-row loop:
if r['event_id'] in seen:
    errors.append(prefix+'duplicate or replayed event_id')
seen.add(r['event_id'])

A within-batch uniqueness check alone would miss an identifier previously published in another batch. This experiment supplies that history as a fixture; it does not implement a durable production deduplication service.

How were the correct customer charges calculated?

The correct complete invoice is $6.70, calculated by adding each customer’s tokens, applying that customer’s illustrative rate per 1,000 tokens and rounding once per customer to cents. Expected charges were calculated separately from the billing query so a matching total alone would not hide incorrect customer attribution.

SELECT c.customer_id, SUM(u.usage_units) AS total_units, c.rate_cents
FROM usage AS u
JOIN customers AS c ON u.customer_id = c.customer_id
GROUP BY c.customer_id, c.rate_cents
ORDER BY c.customer_id;

Python converts the aggregated quantity and rate into a charge using Decimal and ROUND_HALF_UP. This same calculation runs after every gate that accepts the batch.

The rates are invented teaching prices. This simplified calculation excludes subscriptions, tax, discounts, commitments, refunds, multiple currencies and separate input/output token prices.

Which defects did data contract validation catch?

The full contract gate blocked missing fields, nonnumeric usage, duplicate and replayed events, unknown customers, negative usage, a declared unit change and an out-of-window timestamp. The schema gate blocked only the missing field and nonnumeric usage; both valid controls passed all three configurations.

In the table, “Blocked” means no invoice was published for that batch. It does not mean a $0 invoice. “Error” means the unguarded consumer failed during processing.

Two distinctions matter. First, the missing column and text quantity already caused baseline failures. Earlier validation improved their location and explanation; it did not rescue a silently incorrect bill in those cases.

Second, the declared-unit and timestamp defects happened to preserve the baseline dollar total because the billing query ignored those fields. Numerical agreement did not make the input compliant. Checking only $6.70 would miss that distinction.

Across the 11 defective batches, the baseline published seven incorrect customer-charge outcomes, the schema gate also published seven, and the full gate published three. The eight blocked cases are results on these chosen fixtures, not a general detection-rate estimate.

Why did three incorrect bills still pass?

Three incorrect bills passed because the contract gate did not verify measured quantities against source evidence, confirm correct customer attribution or reconcile the expected event population. It checked whether the delivered records satisfied specified rules, leaving those three business facts outside its enforced coverage.

Undisclosed unit change

E002 changed from 20,000 tokens to 20 while retaining the token label. Twenty is a valid nonnegative integer. The gate had no evidence that the producer had changed its measurement convention. C002’s charge fell from $2.10 to $1.50, reducing the total by $0.60.

Wrong but valid customer

E002 moved from C002 to C001. Both identifiers exist, so the relationship check passed. C001 was charged $1.40 instead of $1.00; C002 was charged $1.50 instead of $2.10. The net error was only $0.20, concealing larger errors for the individual customers.

Missing event

E006 disappeared. The other five records remained valid, but C003’s charge fell from $3.60 to $1.20. The total dropped by $2.40. Record validity could not prove batch completeness.

The missing event is the failure I would emphasize in a business review. A green validation result can mean “the delivered records passed these checks.” It cannot automatically mean “all billable usage was delivered.”

How were the results validated?

The results were validated with separately calculated customer charges, eight automated acceptance tests and post-run reconciliation against the reference ledger. A fresh extraction of the downloadable package reproduced all 39 result records; the reconciliation step flagged all three errors that escaped the contract gate.

The acceptance tests cover baseline billing amounts, blocked publication, early versus downstream failures, the three escaped errors, unchanged supplied history, per-customer rounding and result-record counts.

Reconciliation compares event identifiers and the associated customer, quantity, unit and timestamp. It runs after processing and does not block publication in any of the three configurations. Treating it as part of the full gate would overstate what that gate achieved.

The reference ledger is a controlled fixture generated from the clean example, not a separately observed production source. The exercise assumes trustworthy reference evidence. If two real feeds inherit the same upstream error, agreement between them may still be wrong.

The recorded outputs and tests support reproducibility. They do not constitute an independent human sign-off or prove production billing readiness. Business reviewers would still need to approve definitions, authoritative references, exception decisions and recovery handling.

Would I use this approach in a real billing pipeline?

I would use the tested checks as one layer of billing protection, with completeness controls, reconciliation and owned exception handling added for the business facts they cannot establish. I would not approve a release simply because the contract gate returned no errors or because the final invoice total looked plausible.

The useful lesson is specific: quality rules can change publication behavior, but their scope must match the decision. A customer-existence check establishes existence. A quantity-type check establishes type. Neither establishes correct attribution or measurement.

A stronger contract can require reconciliation. The missing coverage here is a limitation of this implementation, not an inherent inability of all data contracts to describe stronger requirements.

There are operational gaps too. The replay scenario uses supplied history rather than durable transactional storage. This experiment does not test concurrent writers, production retries, scale or incident recovery. Rejecting a whole batch delays valid usage as well as defective usage, so someone must own investigation and release.

My next test would add independently supplied batch control totals and a reconciliation gate, then repeat these cases and introduce new ones. Hard-coding an expected six-row count would only fit this fixture.

The responsibility question connects to The Data-Driven Executive: what information can the business trust, and who can decide when it is fit for use? For the broader concept, return to the data contracts guide.

Official resources

• Open Data Contract Standard: a broader standard for describing data agreements.
• Python sqlite3 documentation: the database interface used in this experiment.
• Python Decimal documentation: decimal arithmetic and rounding.

What are you building, what data does it depend on, and which blocker would you test first: duplicates, missing records, changed meaning or incorrect attribution? Share your scenario in the comments.