Is Your AI Worker Ready for the Role? A Role-by-Role Readiness Checklist for Operators Comparing Vendors

Is Your AI Worker Ready for the Role? A Role-by-Role Readiness Checklist for Operators Comparing Vendors

Most AI readiness checklists ask whether *your business* is ready for AI. This one asks a narrower, more useful question: **is this specific AI worker ready to do this specific job in your business?**

That distinction matters when you're comparing vendors. A vendor can look impressive in a demo and still be a poor fit for the role you need filled — and the gap usually shows up in places a demo never touches.

Two questions, not one

There are really two readiness questions in any AI worker decision, and weak evaluations tend to over-index on the first:

1. **Is the vendor's product credible?** (security, governance, references, contract terms) 2. **Is this worker ready *for this role*?** (can it actually handle your inputs, your edge cases, your escalation paths)

The research on vendor evaluation is fairly consistent here: the second question is the one that gets skipped, and it's the one that decides whether the deployment works. What follows is structured around role fit first, vendor posture second — the reverse of how most demos are run.

Part 1: Define the role before you talk to vendors

Before you compare anything, write down the role in the same terms you'd use to hire a person. An AI worker taking on a role needs, at minimum:

- **A defined scope.** What tasks are in, and what's explicitly out. - **Named inputs.** The document types, message formats, files, and systems it will receive. - **A definition of "done."** What a correct output looks like, in specifics, not adjectives. - **An escalation path.** Who or what handles what the worker can't or shouldn't. - **A failure tolerance.** What's the cost of a wrong answer here — an annoyance, or a compliance event?

If you can't write these down, you're not ready to evaluate vendors yet, and no demo will get you there. This is the part of readiness that lives on your side of the table, and it's also the part most vendor comparisons quietly assume away.

Part 2: Score the worker on failure, not on the demo

The most useful guidance in the vendor-evaluation literature is blunt: **score vendors on how they handle failure, not on their demo.** A polished demo shows the worker can handle the inputs the sales team rehearsed. It tells you very little about input six — the unexpected customer message, the malformed file, the request outside the scope entirely.

So build a failure-focused test:

1. **Hand every vendor the same test set.** Same knowledge base, same example inputs, same edge cases, same scoring rubric. This is the single most important thing you can do for comparability, and it's the approach that evaluation research converges on — no single metric tells you whether an agent truly works well, so consistency of the test matters more than the polish of any individual run. 2. **Include the ugly inputs on purpose.** Blank fields, contradictory instructions, oversized or corrupt files, questions in a language you didn't mention, requests that should trigger escalation. 3. **Ask what happens after a wrong answer.** Who notices? How quickly? Is there a log you can review? Can a human correct it and does the correction stick? 4. **Watch the handoff.** When the worker escalates, does it have what your human team needs, or does it dump a transcript and walk away?

Take notes on behavior under stress. That's your score, not the happy path.

Part 3: Ask governance questions early, not at the contract stage

Compliance posture is often treated as a final checkpoint before signature. The vendor-evaluation research argues the opposite: **evaluate it early**, because a mismatch here ends the conversation and you shouldn't spend three weeks of pilot time discovering it.

Depending on the role, the questions to put to every vendor include:

- Do you hold **SOC 2 Type II** — and can I see the report, not just the badge? - Is a **HIPAA BAA** available if the role touches health information? - Will you sign a **GDPR data processing agreement**? - Are you **PCI-DSS compliant** for any payment-adjacent workflow? - More generally: which regimes apply to *my* role, and can you answer each in plain language?

The most useful rule of thumb from the research: **if a vendor can't answer these plainly, that's your answer.** You don't need to become a security auditor to notice evasion.

Part 4: Run the six-dimension check

A useful structure for a full evaluation tests six dimensions rather than one. The exact categories vary by framework, but in practice they map to:

1. **Capability** — does it do the role's core work accurately? 2. **Reliability under real inputs** — the failure testing above. 3. **Integration and data** — can it reach the systems the role needs, with your data quality as it actually is today? 4. **Governance and compliance** — the questions in Part 3. 5. **Operability** — can your team monitor, correct, and update it without vendor hand-holding? 6. **Commercial fit and exit** — pricing at your volume, and what happens if you leave.

Weight the dimensions for *your* role. A customer-facing role should weight reliability and governance heavily. A back-office role might weight integration and operability more. What you shouldn't do is let capability alone dominate — that's exactly the over-indexing the research warns about.

Part 5: Check your own readiness, honestly

Vendor evaluation has a mirror image, and it's worth being candid about it. Survey data cited in AI readiness research suggests **more than half of employees don't feel adequately trained to work with AI**, and **just under half of employers say they haven't implemented AI because their company's data isn't ready.**

Those are not reasons to stop evaluating vendors — they're inputs to the evaluation. Specifically:

- **Data readiness.** If the role depends on a knowledge base or dataset you know is messy, ask each vendor how they handle that *as it is*, not after a cleanup project. - **Team readiness.** Who on your side owns the worker after go-live? If nobody is named, that's a gap the vendor can't fill. - **Training.** Ask what role-based training is included, and for whom.

The rest of the standard readiness list — business alignment, infrastructure, culture, change management, pilot design, ROI measurement — applies. Just make sure it's informing the vendor shortlist rather than becoming the reason the shortlist never gets built.

A one-page checklist to take into vendor calls

**On the role** - [ ] Scope, inputs, definition of done, escalation path, and failure tolerance written down - [ ] The role named in the same terms you'd use to hire for it

**On the test** - [ ] Identical knowledge base, inputs, and rubric given to every vendor - [ ] Deliberately bad inputs included - [ ] Post-failure behavior and logging reviewed - [ ] Escalation handoff examined

**On governance** - [ ] SOC 2 Type II report requested - [ ] HIPAA BAA availability confirmed (if applicable) - [ ] GDPR DPA discussed - [ ] PCI-DSS confirmed for payment-adjacent roles (if applicable) - [ ] All answers plain and non-evasive

**On the six dimensions** - [ ] Capability, reliability, integration, governance, operability, commercial fit each scored - [ ] Weighting set *before* the first call, based on the role

**On your side** - [ ] Data readiness assessed against the role's actual requirements - [ ] Internal owner named - [ ] Training and change management planned

What to do with the results

If a vendor can't answer the governance questions plainly, move on. If two vendors score similarly on capability but diverge on failure handling and operability, the divergence is the decision.

And if you get to the end of this checklist and realize the role isn't defined sharply enough to test against — that's a normal place to be, and a better place to start than picking a vendor and working backwards.

--- *This draft is prepared for human review. No claims about specific vendors, pricing, or outcomes have been made beyond what the cited research supports.*