The AI Automation Playbook: What to Automate, What to Skip

Every team claims it is "automating with AI" by early 2026; the ones with results share a pattern. They did not automate everything that could be automated — they scored candidates, shipped one narrow workflow with guardrails, and measured ROI before the next. The automation graveyard is full of projects that died because they chose the wrong first target or skipped the measurement. This playbook is the order of operations that avoids both.

Score automation candidates before building

• Frequency: how often does the task happen? Monthly tasks can't amortize the build cost.
• Repeatability: is the input structured and the output checkable? Unstructured free-for-alls fail.
• Cost of error: can a mistake be caught and fixed cheaply? Not for anything irreversible or regulatory.
• Volume of human time: minutes per task × tasks per week is the number that justifies the work.
• Evalability: can you score success automatically? If not, you cannot prove the automation works.

Pick a first automation with a 90% success bar

Your first automation should be narrow enough that the AI succeeds almost always: extract data from an email and file it, draft a routine reply for approval, summarize a weekly report. The goal is not the time saved — it is building trust and the feedback loop. A first automation that fails visibly poisons the whole program. Save the clever multi-step agents for the third or fourth project, after the pipeline patterns are proven.

Guardrails are the product

1. Human-in-the-loop for anything that sends, spends, or commits — approval on actions, not just on output.
2. Output validation against a schema before any downstream system consumes it.
3. Source grounding: the model answers from retrieved documents, with citations, for anything factual.
4. Rate and budget caps: a per-run token budget and a kill switch so a runaway loop cannot cost a week of budget.
5. Audit trail: log every automation run, input, output, and approval so problems are explainable.
6. Rollback: every automation ships with a manual mode, and it stays a flag flip away.

Cost per run is a design input, not an afterthought

An automation that saves 10 minutes but costs $0.50 in tokens only wins if the human's time is worth more than $3 an hour. Compute cost per run honestly — including retrieval calls, draft retries, and the occasional escalation to a frontier model — and compare it against the cost of the manual step, including the opportunity cost of your best people doing it. For high-volume automations, route the happy path to a small model and escalate only the tail.

Measure ROI in units you can defend

• Minutes saved per run, multiplied by runs per week — conservative and auditable.
• Throughput: units processed per day before vs after, which captures scale gains humans can't match.
• Quality: error rate or rework rate before vs after, from the eval you built at the start.
• Coverage: work that previously didn't happen (weekly follow-ups, data cleanup) now happening consistently.
• Cost: hard token spend per run, so the savings number has a price tag attached.

Expand only after the measurement says so

The playbook rule: no second automation until the first one shows its ROI number for two full cycles and its error rate is under your threshold. Each new automation reuses the same harness — candidate scoring, guardrails, cost model, eval — so the program compounds instead of scattering. Teams that follow this ship four or five automations a year that keep running; teams that don't ship twelve that get turned off.

FAQ