How to Tell If an AI Worker Is Ready for a Specific Role (A Readiness Checklist for Operators)

Most teams evaluating AI workers start with the same question — *"what can it do?"* — and stop there. That single question is the reason so many pilots look brilliant in a demo and stall in production three weeks later. The demo shows you the AI worker handling the five inputs the sales team rehearsed. It says nothing about input six.

If you're comparing vendors for a specific role in your business — a support queue, an intake function, a research or back-office workflow — the useful question isn't *"is this AI worker capable?"* It's **"is this AI worker ready for this role, in this business, under our constraints?"** Those are different tests, and the second one is the one that predicts whether the deployment survives contact with real work.

Below is a practical way to run that test. It's written for the person who has to sign off, not for the person selling.

First: separate "the model" from "the worker"

A capable underlying model and a ready worker are not the same thing. Every serious vendor now uses a strong model. What differs — and what you're actually buying — is everything wrapped around it: the workflows it's allowed to touch, the guardrails, the escalation paths, the integrations, the observability, and how the vendor behaves when something breaks.

If a vendor's answers are all about model capability and never about operating discipline, you've learned something. Note it and move on.

The five tests that actually predict readiness

1. Failure behavior, not demo behavior

This is the single most discriminating question you can ask. A polished demo tells you the agent can handle prepared inputs. It tells you nothing about what happens when a customer types something unexpected, uploads a malformed file, or asks the agent to do something outside its scope.

Ask directly:

- When the worker hits a task outside its scope, what happens? Does it stop, escalate, guess, or fail silently? - What's the escalation path, and can you configure it per workflow? - Show me a real (or anonymized) failure log. Not a hypothetical — an actual incident and what the vendor did about it. - What's your documented process when the system produces a wrong answer that reached a customer?

Score vendors on this above all else. A worker that fails *loudly and safely* is worth more than one that fails rarely but invisibly.

2. Scope boundaries for *this* role

Generic capability claims collapse the moment you name an actual role. So name it. Write down, in one page, what the worker is allowed to do in this position and — just as importantly — what it must never do without a human. Then make the vendor respond to that page, not to their slide deck.

Useful probes:

- Which parts of this workflow would you refuse to automate, and why? - What would you need from us (data, access, documentation) before this role could go live? - Are there tasks in this role you'd explicitly recommend keeping with a human?

A vendor who can't tell you what *not* to automate either doesn't understand the role or is telling you what you want to hear. The willingness to scope down is a readiness signal in the vendor.

3. Integration and data reality

Most stalled AI projects stall here, not at the model. Before you believe a timeline, verify what the worker actually needs to connect to — your helpdesk, CRM, ticket system, database, or a human who currently exists because nothing talks to anything.

- Which of these systems do you have a native connector for, and which need custom work? - Who owns the integration — you or the vendor — and how is that priced? - What happens when one of our systems is down or returns bad data? - Can we pilot with read-only access before you get write access to anything?

Be honest with yourself about your own data readiness too. A large share of employers report they haven't implemented AI specifically because their company's data isn't ready. If that's you, the fix isn't a different vendor — it's sequencing the pilot around a workflow whose data is actually usable.

4. Governance, security, and compliance posture — early, not last

Compliance is the dimension teams most often leave as a final checkpoint, and it's the one that most often forces a re-do. Bring it into the first conversation.

At minimum, ask which of these the vendor holds and can evidence, and note that the relevant ones depend on your industry and what the worker touches:

- SOC 2 Type II - HIPAA BAA availability (if any health data is involved) - GDPR data processing agreements (if EU personal data is involved) - PCI-DSS compliance (if any payment-adjacent workflow is involved)

Then go past the checkmarks:

- Where does our data live, and is it used to train anything? - What's your retention and deletion policy, and can it be configured? - Who at your company can access our data, and under what logging? - Can you sign our DPA, or do we sign yours?

A vendor who answers these plainly and quickly is telling you how the whole relationship will go. One who deflects to a legal call is telling you something too — not necessarily fatal, but factor it in.

5. Measurement: what does "ready" actually look like on day 30?

Readiness is only meaningful against a definition of done. Agree on it before you start.

- What does success look like at 30 and 90 days, in numbers we both accept? - What baseline do we measure against — current handle time, resolution rate, error rate, cost per task? - What's the fallback if it underperforms that baseline? - Who reviews outputs during the first weeks, and how do we ramp that down?

If a vendor resists committing to a measurable definition of success, treat that as your answer. It's kinder to find out in the scoping call than in month four.

The checklist, in one place

Copy this into your evaluation notes and fill it in per vendor:

**Scope & failure** - [ ] Documented, configurable escalation path - [ ] Vendor can show real failure-handling examples - [ ] Vendor can articulate what *not* to automate in this role - [ ] Silent-failure risk addressed explicitly

**Integration & data** - [ ] Named connectors for our actual systems - [ ] Clear owner for custom integration work, with pricing - [ ] Behavior when our systems fail or return bad data - [ ] Read-only pilot option available

**Governance & security** - [ ] SOC 2 Type II (if applicable) - [ ] HIPAA BAA / PCI-DSS / GDPR DPA as applicable to our workflows - [ ] Training-on-customer-data position stated clearly - [ ] Retention and deletion configurable - [ ] Access logging and least-privilege controls

**Commercial & measurement** - [ ] Pricing model understandable at our expected volume - [ ] Agreed 30/90-day definition of success with baseline - [ ] Fallback plan if it underperforms - [ ] Human review plan for the first weeks

If a vendor can't answer plainly, that *is* the answer

You don't need a perfect score to move forward — you need clarity on the tradeoffs. The vendors worth shortlisting are the ones who answer the awkward questions directly, scope down when it's the right call, and can show you how their worker behaves when things go wrong.

A worker that isn't ready for *your* role isn't a bad worker. It's just the wrong fit, and the whole point of this exercise is to find that out before you've signed, not after.

If you're working through this evaluation for a real role in your business and want a second set of eyes on the scoping, the team at aiworker.today reviews applications from operators who are assessing exactly this.