An AI pilot should test what happens when a useful model meets a real business process. In a small accounts payable experiment, I asked ChatGPT to compare fictional vendor invoices with purchase orders. It returned the expected classification for all 12 invoice records. That was encouraging. It was also a controlled test with readable, simple documents that I had created myself. No AP team used the output in its daily queue, and no payment or ERP action took place.
That gap is where an AI pilot begins. A demonstration asks whether the task is possible. A pilot asks whether the workflow helps people make better decisions under the conditions they actually face, without weakening the controls around those decisions.
Here is how I would design that next step for invoice exception review. The same questions apply if your use case is a sales brief, support draft or purchasing recommendation: What improves, how will you measure it, who checks the answer, and what happens when the system is wrong?
What counts as an AI pilot?
An AI pilot is a limited test of a business workflow with real users, approved data, defined success measures and a clear decision at the end. It includes the work before and after the model: finding source records, presenting evidence, correcting mistakes, handling exceptions and recording what happened. A proof of concept can establish that a model handles selected examples. A pilot must establish whether the whole process is useful and safe enough to operate.
That distinction matters in my AP example. The test on fictional data showed that a language model could read PO notes and distinguish some allowed differences from genuine exceptions. It did not measure handling time, labor savings, performance on poor scans, access control, user adoption or downstream accounting errors. Calling its 12 out of 12 result a successful business pilot would skip the questions that matter most.
In AI to ROI for Business Leaders, I use a fictional demand forecasting story to explore this gap. Janet and Lia face a recommendation that would change a purchase order, so a person must check the evidence before the business acts. That is a teaching scenario in the book, not a reported incident from an operating company. The operational lesson is straightforward: the pilot boundary must include the point at which a recommendation could become a real commitment.
If you have not yet chosen which workflow to pilot, the AI prioritization framework shows how to rank candidates on value that survives review time and adoption.
Start with one decision in one workflow
The proposed AP pilot has a narrow job: prepare an invoice review queue. For each invoice, the system compares the document with its referenced purchase order and, where applicable, the goods receipt and approved AP policy. It returns a match candidate, an exception with evidence, or an uncertain case for review. It cannot approve a payment, change a vendor record, post a transaction or write to the ERP.
The AP manager owns the business measure and the reviewer owns each disposition. Finance, IT and security agree on the permitted documents, access, retention and audit trail before anyone uses live records. If an invoice refers to an unknown PO, the system says the reference was not found. A similar PO may be shown as a possible lead, never as a confirmed replacement.
This is a real business use case because AP teams already spend time resolving invoice differences, and the output has a clear consumer: the person who reviews the queue. The benefit remains a hypothesis until a pilot measures whether that person can resolve the same work accurately with less effort.
The published test gives two useful examples of what would enter such a queue. An invoice for 40 monitors against a 100-unit PO was an allowed partial shipment because the PO note permitted it. Another invoice cited a PO number absent from the supplied workbook, so the correct response was to flag the missing reference. A similar PO was a lead for a reviewer, not proof of a match. Those were fictional examples, and neither included a goods receipt or a live ERP check.
Get the records and joins right before testing AI
A useful pilot needs more than a folder of PDFs. It needs a reliable way to connect each invoice to the records that govern its treatment. The exact fields depend on the company, but the following is a practical starting map.
| Record and grain | What the reviewer needs | Key check |
|---|---|---|
| Invoice, one row per invoice line | Vendor, invoice number, PO reference, amount, currency, date and line details | Can each line be read, and is the vendor identity verified? |
| PO, one row per PO line | Approved item, quantity, unit price, terms and special notes | Does the referenced PO exist and belong to the correct vendor? |
| Goods receipt, one row per receipt line | Quantity received, date and related PO line where receiving is required | Does the receipt support the billed quantity? |
| AP policy, one row per rule and effective period | Effective tolerance, duplicate and approval rules | Which rule applied when this invoice entered the process? |
| Review outcome, one row per reviewed invoice | System suggestion, source evidence, human correction, disposition and time spent | Can the team reconstruct the decision later? |
Use explicit identifiers to join invoices to POs and receipts to PO lines. Do not treat a fuzzy text match as a verified join. For missing, conflicting or unreadable records, route the invoice to a person. Ordinary software should handle arithmetic and fixed tolerances; a language model may help interpret notes and explain why a case needs attention. Source references let the reviewer check that explanation.
Measure the work, the mistakes and the cost
Before turning on the pilot, measure a representative sample of the current process. Include routine invoices and difficult ones, such as partial shipments, duplicates, missing POs and poor-quality scans. The AP manager and reviewers should agree on what counts as a material exception. Then compare the assisted process with that baseline using the same kinds of cases.
| Measure | How to calculate it | Why it matters |
|---|---|---|
| Handling time | Minutes from opening an invoice to a recorded review disposition, including corrections | A quick AI response does not prove the person saved time. |
| Missed exceptions | Material problems the assisted process marked as routine, checked against an independent review | These can create payment or control risk. |
| False alarms | Valid invoices unnecessarily sent for exception review | They can create more work than the tool removes. |
| Reviewer correction | Share of suggestions changed by reviewers, with reasons | It reveals where source data or interpretation fails. |
| Full operating cost | Integration, model use, review, support, security and change-management costs | Savings must exceed the cost of running the process. |
Track these by invoice type and document quality, not only as one overall average. A pilot that works on clean purchase-order invoices but fails on service invoices has a narrower useful scope. Set the acceptance criteria from the current baseline, business risk and available sample size before examining the pilot result. A universal accuracy or adoption percentage would give a false sense of precision.
The financial case is also more than a count of hours theoretically saved. Count only capacity that the team can actually redeploy or cost it can actually avoid. Include the time spent investigating false alarms and reworking mistakes. Then compare the net annual benefit with setup and continuing costs. The AI ROI roadmap walks through a concrete AP calculation, payback period and the benefits that should not be claimed.
Run the pilot in stages
1. Define the boundary and baseline
Write down the precise invoice types, suppliers, users and systems in scope. Name an AP owner and a technical owner. Establish the current handling time, exception rate, rework and cost with a sample that includes messy cases. Agree on the review standard, pause conditions and decision date before showing the team model output.
2. Test in shadow mode
With approved access, let the system prepare suggestions while the normal AP process continues. Compare its output with independently checked outcomes. This catches wrong joins, unsupported explanations and missed exceptions without changing how invoices are posted. Protect vendor and financial information according to company policy, restrict access, and record what data entered the service and how long it is retained.
3. Give reviewers an assisted queue
After the shadow test meets the agreed safety conditions, show the suggestions and source evidence to a limited AP group. Reviewers accept, correct or reject each result. Record corrections and elapsed time in the ordinary workflow. Keep posting and payment approval in the existing ERP process with its authorized roles and separation of duties.
4. Decide with the whole scorecard
At the end, compare quality, total review effort, cost, user experience and control findings with the baseline. Include failures and excluded invoice types in the decision record. A narrow, useful result may justify expanding to one more supplier group or invoice type. It does not justify granting autonomous payment authority.
Put security and responsible use inside the workflow
Financial documents can contain bank details, addresses, prices and contract terms. Before using live data, confirm that the chosen service and configuration satisfy your organization’s vendor, privacy, security and retention requirements. Give the pilot only the access required to read approved records and write review suggestions. Log access, model inputs or references as appropriate, outputs, human changes and final dispositions so issues can be investigated.
Prompt text inside an invoice is untrusted document content. It must not be allowed to override system instructions, request other records or trigger actions. Validate retrieved records and calculations outside the model, and treat its prose as a suggestion requiring evidence. If a source is missing, the system should say so. If a material exception is missed, sensitive data appears outside its approved boundary, or an unauthorized action occurs, pause the affected flow and use the existing manual process while the cause is investigated.
Also ask who bears the cost of mistakes. An AP tool that shifts work to suppliers or delays legitimate payments may appear efficient on an internal time measure while making the overall process worse. Review disputed and delayed invoices as part of the pilot, and let reviewers report problems without pressure to accept the system’s suggestion.
When should you scale, revise or stop?
Scale the bounded workflow when it meets the agreed quality and security conditions, reduces total effort or another measured cost, and can be supported by named owners. Expand scope gradually and keep checking performance by invoice type.
Revise the design when the failure has a specific fix: a missing goods-receipt join, a poorly presented evidence field or a rule that should be handled by code. Retest the affected cases before expanding.
Stop or pause when the system misses material exceptions, cannot maintain its data boundary, creates unauthorized actions, or adds more total work than the benefit can justify. A stopped pilot can be a useful result if it identifies the blocker before a broad rollout.
AI pilot questions
What is the difference between an AI proof of concept and an AI pilot?
A proof of concept tests whether an approach can handle selected examples. A pilot tests whether the full, limited workflow improves a real process with its users, data, costs and controls. My 12-record AP experiment on fictional data is a proof of concept, not a live pilot.
How long should an AI pilot run?
Long enough to see a representative mix of work and compare it fairly with the baseline. The calendar depends on transaction volume, month-end patterns and the kinds of exceptions you need to observe. Set the sample, review checkpoints and decision date in advance rather than assuming every company needs the same number of weeks.
What should an AI pilot measure?
Measure the business outcome and the full cost of reaching it. In AP, that means total handling time, missed material exceptions, false alarms, correction work, operating expense and any downstream rework. Examine the difficult cases separately from the average.
Can an AI pilot use real company data?
Yes, if the organization approves the purpose, access, service configuration, data handling and retention. Begin with the least data and access needed. Where those approvals are unavailable, continue with fictional or appropriately prepared data and describe the work as a controlled test.
The decision to make before you build
The first question is not whether AI can read an invoice. My controlled test suggests it can handle some simple comparisons and business notes. The decision is whether a bounded system can help an AP reviewer resolve actual invoices faster and accurately enough to justify its full cost. That requires an owner, a baseline, representative records, independent checks and a clear human authority boundary.
My book, AI to ROI for Business Leaders, develops pilot and guardrail decisions as part of a full execution roadmap.
What business workflow are you considering for a first AI pilot? Share the decision it would help someone make, the data you already have and the blocker you are trying to resolve.