Guide · AI
Most AI automation projects fail on selection, not technology. The tasks worth automating share a specific shape, and the ones that burn money share a different one.
A task is a good automation candidate when it is repetitive, structured, deadline-driven, and cheap to verify. That last one carries most of the weight: if checking the output takes as long as producing it, automation has moved the work rather than removed it.
| Automate | Keep human |
|---|---|
| Drafting a first version | Sending anything that commits you |
| Extracting fields from documents | Deciding what the fields mean for the relationship |
| Classifying and routing inbound messages | Handling the angry one |
| Chasing missing paperwork on a schedule | Negotiating price |
| Summarising a long thread | Making the call the thread was about |
| Monitoring dates and flagging expiry | Deciding what to do when something expires |
Everyone has an intuition about what to automate and the intuition is usually wrong, because it is driven by which task is most annoying rather than which is most valuable. Score candidates on five criteria, weighted. Maximum score is 55.
| Criterion | Weight | Score 1 | Score 3 | Score 5 |
|---|---|---|---|---|
| Frequency | ×3 | Monthly or less | A few times a week | Many times a day |
| Structure of the input | ×3 | Free-form, no pattern | Semi-structured, varies by source | Consistent shape every time |
| Verification cost | ×2 | Checking takes as long as doing | Checking takes half the time | Checking is a glance or a code assertion |
| Reversibility if wrong | ×2 | Irreversible and customer-facing | Recoverable with an apology | Nothing leaves the building |
| Input-format stability | ×1 | Upstream format changes constantly | Changes a couple of times a year | Has not changed in years |
Then apply thresholds: 38 or above, build it. 28 to 37, build it but run it in shadow mode indefinitely. Below 28, leave it alone. Three worked scores, on tasks that look superficially similar:
CANDIDATE A - drafting follow-up emails to a lead list Frequency 5 x3 = 15 (dozens per day) Structure 4 x3 = 12 (lead record fields, consistent) Verification 4 x2 = 8 (skim against the record) Reversibility 2 x2 = 4 (it reaches a real person) Format stability 4 x1 = 4 --------------------------------------------- TOTAL 43/55 → BUILD CANDIDATE B - writing the monthly board narrative Frequency 1 x3 = 3 (12 times a year) Structure 2 x3 = 6 (different story every month) Verification 2 x2 = 4 (you must read every word anyway) Reversibility 2 x2 = 4 Format stability 3 x1 = 3 --------------------------------------------- TOTAL 20/55 → LEAVE IT ALONE CANDIDATE C - extracting fields from inbound invoice PDFs Frequency 5 x3 = 15 Structure 5 x3 = 15 (invoices have a fixed vocabulary) Verification 5 x2 = 10 (assert totals match line items) Reversibility 4 x2 = 8 (writes to a record, nothing sent) Format stability 3 x1 = 3 (each vendor redesigns eventually) --------------------------------------------- TOTAL 51/55 → BUILD THIS FIRSTCandidate B is the one people build, because writing the board narrative is the task the founder resents most. Candidate C is the one that pays, because verification is nearly free and nothing it produces can embarrass anyone. Resentment is not a prioritisation signal.
Illustrative numbers throughout — substitute your own. The point is the shape of the calculation, in particular that the review time after automation is never zero and the maintenance line is never zero either.
TASK: drafting outreach follow-ups BEFORE Frequency 60 per week Manual time per item 6 minutes Weekly time 60 x 6 = 360 min = 6.0 h Annual time 6.0 x 52 = 312 hours Error rate (sent wrong) ~3% = 1.8 items/week AFTER Review time per item 1.5 minutes Weekly time 60 x 1.5 = 90 min = 1.5 h Annual time 1.5 x 52 = 78 hours SAVING Weekly 6.0 - 1.5 = 4.5 hours Annual 4.5 x 52 = 234 hours COST Build 28 hours Maintenance 1 h/month = 12 h/year Inference (local model) ~0 marginal PAYBACK Build hours / weekly saving 28 / 4.5 = 6.2 weeks NET, at an illustrative internal value of $30/hour Value of hours saved 234 x 30 = $7,020 Less build (one-off) 28 x 30 = $840 Less maintenance (annual) 12 x 30 = $360 --------------------------------------------------- Year one net $5,820 Year two onward $6,660Three things this calculation catches that hand-waving does not. A payback measured in weeks is a build; a payback measured in years is a hobby. A review time of zero is a lie, and pretending otherwise is how projects come in at half the promised saving. And a task that occurs twice a month cannot clear any reasonable build cost no matter how much you dislike it.
The architecture that consistently works is two-tier. A cheap local model does the volume — drafting, tagging, extraction, overnight batches — at zero marginal cost. A frontier model does the judgement: the final pass, the hard reasoning, anything customer-facing. Between them sits the piece most people skip.
| Tier | What it does | Marginal cost | What it must never do |
|---|---|---|---|
| Local model on your own hardware | Bulk drafting, tagging, extraction, classification, overnight batches | Electricity only | Be trusted without a gate; be the final word on anything sent |
| Automated quality gate (code, no model) | Hard invariants, deterministic checks, quarantine | Effectively zero | Contain judgement. Every rule must be a boolean |
| Frontier model | Hard reasoning, final polish, ambiguous cases, anything customer-visible | Per token, meaningful at volume | Run on every item when 90 percent of items are easy |
| Human | Exceptions, irreversible actions, relationships | The scarce resource | See the whole queue. They should see only what the gate could not resolve |
The economics of the split are the point: the cheap tier absorbs the volume, the expensive tier absorbs the difficulty, and the gate decides which is which. Running the frontier model on everything is the most common way to make a working pipeline uneconomic. Setting the cheap tier up is covered in the local LLM guide, keeping its output machine-checkable in the structured output guide, and cutting the cost of the expensive tier in the prompt caching guide.
Generated output at volume is worthless without an automated check, because the failure rate is never zero and one bad artefact reaching a customer costs more than the whole batch saved. A gate is a set of hard invariants, checked in code, that quarantines anything failing them.
| Invariant | What it catches | How it is checked |
|---|---|---|
| Required fields present and non-empty | Truncated generations, silent API failures | Presence and length assertion per field |
| No placeholder text or unfilled template variables | Leaked brackets, XX, TODO, lorem ipsum, [NAME] | Regex blocklist over the rendered output |
| Names, amounts, and dates match the source record exactly | The single most damaging class of error: confident wrong specifics | String equality against the source record, not similarity |
| Length and format within bounds | Rambling output, missing structure, wrong file type | Character and line counts; schema validation |
| Banned-phrase list | Anything that would embarrass you, plus spam-filter triggers | Case-insensitive substring match |
| Deduplication against what has already been sent | The same person receiving the same message twice | Hash or key lookup against a send log |
| Recipient and identifier sanity | Wrong record joined to right template | Cross-field consistency check between record and rendered text |
| Encoding and whitespace clean | Mojibake, doubled spaces, stray markdown artefacts | Normalisation pass plus a character-class assertion |
Everything failing the gate goes to a quarantine folder rather than out the door. Volume without a gate is not throughput, it is liability at scale.
Candidate 2 is the important one, because a human skimming a batch of eighty would very likely catch [MONTH] and would very likely miss Sacramento. The city is plausible, well-formed, and completely invented. That is precisely the failure class a string-equality check catches for free and human review does not.
Shadow mode means the pipeline runs end to end — generate, gate, log — and sends nothing. It exists so that the decision to go live is made against data rather than against optimism. Give it a defined exit condition before you start, or it runs forever or ends the first time you get impatient.
| Measurement | What it tells you | Bad result means |
|---|---|---|
| Gate pass rate | How good the generator is | Below 85 percent: fix the prompt or the inputs, not the gate |
| Escape rate in audited passes | How good the gate is | Anything above zero: add an invariant, do not go live |
| False-positive rate in audited failures | Whether the gate is too strict | Above 20 percent: you are quarantining good work and will start ignoring the queue |
| Time per human review | Whether the saving is real | If review is not much faster than doing it manually, the ROI was fiction |
| Distribution of failure reasons | Where to spend the next hour | One reason dominating usually means one prompt fix removes most failures |
Every one of these will happen. Knowing the list in advance is the difference between a gate designed against reality and a gate designed against imagination.
| Failure | What it looks like | Detection | Mitigation |
|---|---|---|---|
| Confident invention | A plausible city, date, or figure that appears nowhere in the source | String equality against the source record | Never allow a specific to originate in the model |
| Placeholder leakage | [NAME], XX, TODO, lorem ipsum surviving into output | Regex blocklist | Fail closed; never strip and send |
| Wrong record joined | Right template, wrong customer's details | Cross-field consistency assertion | Pass one record object, never loose variables |
| Silent truncation | Output stops mid-sentence at a token limit | Minimum length plus terminal-punctuation check | Raise limits; assert on completion reason |
| Upstream format drift | A vendor redesigns a PDF and extraction returns nulls | Sudden change in gate pass rate | Alert on pass-rate deltas, not just on absolute failures |
| Duplicate send | The same recipient contacted twice in a week | Send-log key lookup before dispatch | Dedup at dispatch time, not at generation time |
| Encoding damage | Smart quotes turning into mojibake in a plain-text channel | Character-class assertion | Normalise to a known character set at render |
| Partial batch | A run dies halfway and is re-run from the start | Idempotency key per item | Make every write idempotent before you make it automatic |
| Tone drift | Technically valid output that does not sound like you | Human sample review only | This one genuinely cannot be automated. Keep sampling |
| Phase | What you do | Time | Exit condition |
|---|---|---|---|
| 1. Instrument | Count occurrences, time the manual version, price an error | 3–5 h | You have frequency, minutes, and error rate written down |
| 2. Specify the gate | Write the invariants as assertions before any generation exists | 2–4 h | Every rule is a boolean a machine can evaluate |
| 3. Build the data path | Get the source records into a structured form the generator can read | 8–20 h | One record object, all fields, no scraping at generation time |
| 4. Build the generator | Prompt, template, render. Cheap tier only | 4–10 h | It produces plausible output on 20 real records |
| 5. Shadow mode | Run, gate, log, audit, iterate | 1–3 weeks elapsed | Pass rate and escape thresholds met across three batches |
| 6. Human-fired sends | Output goes out only when a person clicks | Ongoing | Weeks of clean batches and a reviewer who is bored |
| 7. Selective automation | Auto-send only the highest-confidence slice, humans keep the rest | Ongoing | You would bet your own money on the pass rate |
Two rules that hold across all seven phases. Write the gate before the generator — if you cannot specify what correct output looks like in code, you are not ready to generate it. And keep a human at the point of irreversibility until the pass rate has been stable for long enough that you would bet on it. Most of the value arrives at phase 6, not phase 7, and phase 6 carries a fraction of the risk.
Less than people expect on tokens and more than they expect on plumbing. Illustrative figures for a single high-volume drafting pipeline built by one person on hardware they already own.
| Line item | One-off | Recurring | Notes |
|---|---|---|---|
| Model inference, local tier | $0 | Electricity, effectively noise | Requires hardware you already have. If you do not, this line becomes a purchase |
| Model inference, frontier tier | $0 | Per token; material only if you route everything through it | The gate exists partly to keep this line small |
| Integration and data plumbing | 8–20 h of build | — | The largest genuine cost, and the one nobody budgets |
| Gate specification and testing | 2–6 h | — | Cheap, and the highest-return hours in the project |
| Storage and logging | ~$0 | Pennies | A send log and a quarantine folder. Do not skip because it is cheap |
| Monitoring and alerting | 1–3 h | — | Alert on pass-rate change, not just on crashes |
| Maintenance | — | 0.5–2 h/month | Upstream formats change. Budget for it or the pipeline rots |
| Human review time | — | Scales with volume until the gate is trusted | This is the line that determines whether the ROI was real |
| The first bad artefact that escapes | — | Unbounded | Which is the entire argument for the gate |
Which is the honest summary of this whole field right now — the model is the cheap part.
A worked example of all of this in production is FreightDesk AI: local model drafts overnight, an automated audit quarantines anything failing its checks, a human sees only exceptions. The freight-specific version of the same problem is in the back office guide, the retrieval half is in the RAG guide, and if you want the build done rather than described, that is the services page.
FAQ
Tasks that are repetitive, structured, deadline-driven, and cheap to verify — drafting first versions, extracting fields from documents, classifying and routing messages, chasing paperwork on a schedule, summarising threads, and monitoring expiry dates. If verifying the output costs as much as doing the work, automation has moved the work rather than removed it.
Score candidates on frequency, input structure, verification cost, reversibility if wrong, and format stability, with frequency and structure weighted heaviest. Build anything scoring high, and leave low scorers alone regardless of how annoying they are. Resentment is not a prioritisation signal, which is why founders build the monthly board narrative instead of the daily document extraction.
Multiply frequency by minutes saved per item to get weekly hours, subtract the review time that still exists after automation, and divide build hours by the weekly saving to get payback in weeks. A payback measured in weeks is a build and a payback measured in years is a hobby. Never model review time as zero.
Anything that commits you or depends on a relationship: sending final communications, negotiating price, handling complaints or claims, and deciding what to do when something the system flagged actually happens.
A cheap local model handles high-volume, low-difficulty work such as drafting, tagging, and extraction at zero marginal cost, while a frontier model handles judgement, hard reasoning, and anything customer-facing. An automated quality gate sits between them and decides which items need the expensive tier.
Because generation at volume always has a non-zero failure rate, and one bad artefact reaching a customer can cost more than the entire batch saved. A gate checks hard invariants in code — required fields, no placeholder text, exact string matches for names and amounts, deduplication — and quarantines anything that fails.
Shadow mode runs generation and the gate end to end while sending nothing, so the go-live decision is made against data. A workable exit condition is a gate pass rate at or above 95 percent with zero escapes in 30 audited passes, sustained across three consecutive batches, and even then the next step is human-fired sends rather than full automation.
Less than expected on model inference, especially with a local model on hardware you already own, and more than expected on integration — connecting data sources, defining invariants, handling exceptions, and maintaining it when an upstream format changes. Budget half an hour to two hours a month of maintenance forever. The model is the cheap part.