What review step keeps an automated AI workflow accurate after it goes live?

An AI job can go wrong a little at a time for weeks, and nothing tells you. The review step that fixes this is a weekly AI check on each live job. Once a week, AI reads everything the job did, compares it to the plain rules you gave it, and writes a one page report. Each item is marked fine, look, or stop. You read only the marked items. By "workflow" I mean a job you have handed to AI that now runs by itself, such as sending order updates to customers or filling in a Monday report.

What goes wrong in the weeks after launch

Nothing breaks loudly. That is the problem. The job keeps running and keeps looking busy, so nobody checks it.

Three things happen. First, what the job reads changes. A supplier renames a column in their sheet, a new product gets added, a price list is updated. The AI keeps working from what it was told on day one. Second, small errors repeat. One wrong line in a customer message is an accident. The same wrong line in forty messages is a policy. Third, nobody owns it. The person who set it up moves on, and the job is treated like a finished machine instead of a new hire. I wrote about that in why you should treat AI like a hire.

A job can also pass every check you wrote and still be wrong, because the checks measure the wrong thing. I covered that in a post on checks that stay green while the work is wrong. The review has to look at the work itself, not only at whether the job ran.

Which checks the AI can run on itself, and which need a person

The AI can read every item the job produced, not a sample. It can compare each item to your rules, notice when this week looks different from last week, and tell you what changed in the data the job reads. A person reading by hand does not have the hours for that.

The AI cannot decide whether a rule is still the right rule. It cannot tell whether a message sounds like you on a bad week. It cannot undo something that has already gone to a customer. Those need a person. The split is simple: the AI checks facts and consistency, and a person decides anything that is about judgment or cannot be undone. AI that audits the AI goes further into how that second check works.

Here is the decision tree, which is a set of questions where each answer sends you down a different path. Start with the live job and ask what it touches. This is illustrative, with no figures.

Does it send messages to customers?

• The AI reads every message each week and marks anything sent twice, any promise about price, refunds or delivery dates, and any message that sounds unlike the rest.
• A person reads every marked message, and signs off before any change to the wording or the rules.
• Look now, not Monday, if a customer complains or a message goes out twice.

Does it move numbers into reports?

• The AI re-adds the report from the source figures each week and marks any total that differs, even slightly.
• A person looks only at marked totals, and at any week the source sheet changed shape.
• Look now if two reports disagree.

Does it touch stock records (your counts of what is on hand)?

• The AI checks each count against the shipments and sales it can see, and marks every count it cannot match.
• A person looks at marked counts, and decides whether the count or the record is wrong.
• Look now if the AI cannot explain a gap.

None of those?

• A monthly AI check. A person looks only when something is marked.

How often to review

Match the schedule to the damage a mistake can do. For the first four weeks after launch, have the AI check every day and read the report yourself once a week. After that, check weekly for anything that touches customers, money, or stock, and monthly for low stakes work. After any change to the job, or to the data it reads, go back to daily for two weeks.

What the report should show

Keep it one page and the same shape every week, so your eye learns where to land. It should have four parts. One line per job: fine, look, or stop. What changed since last week. Each marked item, with the rule it broke and a fix ready for you to approve. And one sentence on what the AI could not check. If you cannot read it in five minutes, tell the AI to shorten it.

Run the review today

In the older days you would open the sent messages, the report, and the stock sheet. You would read through them, compare them to the source figures, write down what looked off, and email whoever built the job. Most owners skip it, because it eats an afternoon. If you did it, three hours later you would have your answer.

Today, give this to Claude (the AI assistant I use) and you have your answer in under a minute. (A prompt is just the instructions you type in; you can paste this one as it is.) To try it once, paste the job's last week of work underneath it, together with the source figures it was built from.

You are reviewing a job that AI does for my business. Underneath these instructions I will paste the last week of what the job produced, along with the source figures it used.

First, say in one sentence which kind of job it is: it sends messages to customers, it moves numbers into reports, it updates stock records, or something else.

Then check every item, not a sample.
For customer messages, mark anything sent twice, any promise about price, refunds or delivery dates, and any message that sounds different from the rest.
For reports, re-add the figures from the source figures and mark every total that differs, even slightly.
For stock records, mark every count you cannot match to a shipment or a sale shown.
For anything else, mark what differs from the earlier items.

Then write a one page report. Give one line per job saying fine, look, or stop. Say what changed compared with the earlier items. List each marked item with the rule it broke and your suggested fix. End with one sentence on what you could not check. Keep it short enough that I can read it standing up. Do not change anything yourself.

After that, you stop pasting. Hand this one line to whoever set up your job, or to Claude in its scheduled tasks setting:

Every Monday at 7 AM Pacific, collect the last week of work from my live job and the source figures it uses, follow the review instructions above, and email me the one page report.

Then your only task is reading the marked lines. Read the whole report the first week. By the fourth week you will read only the marked lines.

If you would rather have this set up for you, the Workflow Build page describes how that works.