The pattern we see across deployments is consistent enough to be a playbook. Agents that go from pilot to production in under three months share five traits. Agents that stay stuck in pilot past six months are usually tripped up by the same five missteps. This is what we've seen work.
Phase 1: Pick one job, end to end
The single biggest predictor of an agent's success in production is how narrowly it was scoped. The agents that ship are the agents that own one job end to end — triage inbound leads, draft support replies, schedule customer check-ins. The agents that get stuck are the agents that "help the team" — vague scope, overlapping with human roles, impossible to evaluate.
A good first agent has three properties: it touches a high-volume workflow, the workflow has a measurable outcome, and a human is comfortable approving its actions before they execute. Lead triage meets all three. "Write me a blog post" meets none.
Phase 2: Audit the data first
Before the agent is built, the data it will read needs to be audited. We've seen too many deployments where the agent works correctly on a sample dataset and hallucinates in production because the production data has fields the training data never saw.
Audit the data the agent will touch. Tag what's clean, what's noisy, and what's missing. Decide explicitly which fields the agent is allowed to read, which it can write, and which are out of bounds. Write the policy down. The security team will ask for it.
Phase 3: Shadow mode for 30 days
The agent runs in shadow mode for 30 days before any real action. The agent's proposed actions are logged and presented to the human team, but the human team performs the real action. The agent's actions are measured against the human's actions.
Shadow mode produces three pieces of information: the agent's alignment rate (how often its proposed action would have been the same as the human's), its coverage rate (what percentage of cases it had a confident answer for), and its failure mode (the cases where it was wrong and why).
The failure modes are where the surprises live. Aggregate agreement rates flatter an agent badly: one that matches human decisions on the easy majority can still fail hard on the residual, and those are exactly the cases you would not want a customer to meet. Shadow mode exists to surface that tail before anyone is exposed to it — so read the disagreements, not the headline percentage.
Phase 4: Human approval for anything irreversible
Once the agent leaves shadow mode, every action that is hard to reverse requires human approval. Sending an email is reversible (the recipient can be told it was a mistake). Updating a CRM record is reversible (the field can be restored). Sending an invoice is irreversible (the customer has paid). Closing a support case is borderline (the customer has stopped waiting).
The agent's approval surface is the human team's existing approval surface — a Slack channel, an email inbox, a custom UI. The agent proposes, the human disposes, and the platform logs the decision.
Phase 5: Continuous evaluation
Agents are not "ship and forget" software. Models change, data drifts, requirements shift. An agent that performed well last quarter can quietly degrade this one, and without monitoring the first signal is a customer complaint rather than a dashboard.
Continuous evaluation means every agent action is scored by an independent model against a quality benchmark. The scores are aggregated. Drift is detected within hours, not quarters. The team sees the score on a dashboard before the customer feels the quality change.
What fails
Three patterns kill agent deployments:
- Vague scope: an agent that "helps the team" is impossible to evaluate and impossible to trust. Pick a specific job.
- No shadow mode: agents that go straight from prompt engineering to production handle 100 cases fine and 1 case catastrophically wrong. Shadow mode surfaces the catastrophic case before a customer sees it.
- Approval fatigue: agents that require approval for every action train the human team to click "approve" without reading. The approval becomes rubber stamp. The agent's actions become unaccountable. Pick the approvals that matter; automate the rest.
What comes after the first agent
The first agent is the hardest. It builds the team's muscle memory for governance, the platform's muscle memory for the audit trail, the customer's trust in non-human actors on their account.
Once the first agent is in steady state, the second agent is faster to ship — the playbook is now organizational memory. Within a year, a team that started with one qualified agent typically runs five to ten agents, each scoped narrowly, each governed by the same playbook. That's the operating-system metaphor becoming concrete.
Read the agent developer documentation or talk to our solutions team about deploying your first agent.