Did the system find the correct policy section?
loading…
Here is how I would take a broad automation request and turn it into a pilot the support team can actually run.
Initial request “We have too many support tickets. Build an agent that answers customers and completes common support tasks.”
There are several projects hiding in that sentence. Drafting an answer for an employee is one risk level. Sending it directly to a customer is another. Changing an order or issuing money raises the bar again. Treat those capabilities as separate scope decisions.
Before drawing boxes, watch agents resolve real tickets. The official process is useful; the actual clicks, handoffs, workarounds, and exceptions are what you need to build against.
The workflow is usually messier than the diagram Stakeholders often describe the official process. Real operators reveal the actual process, including copy-paste work, missing data, exceptions, unofficial spreadsheets, and manual approval steps.
Assume order-status questions make up 30% of volume. The work is repetitive, measurable, and read-only: look up an order, explain the latest shipment event, and escalate exceptions. That is a sensible pilot. Refunds and fraud are not.
A pilot target worth measuring Automatically draft a cited answer for order-status tickets, reducing agent handling time by 40% while keeping factual accuracy above 95% on an approved evaluation set.
Order-status tickets, read-only order lookup, draft response, citations, human approval.
Refunds, address changes, cancellations, fraud, direct customer sending.
Low-confidence or unsupported cases remain in the existing agent queue.
Order state, policy, ticket history, and the customer message have different owners, freshness requirements, and access risks. Model them separately.
| Source | Purpose | Access pattern | Main risk |
|---|---|---|---|
| Order API | Current order and shipment state | Real-time tool call | Wrong customer or stale response |
| Support policy | Approved explanation and escalation rules | RAG retrieval | Outdated or irrelevant passage |
| Customer message | Intent and requested help | Prompt input | Ambiguity or prompt injection |
| Ticket history | Prior contact and promises | Structured context | Excessive PII exposure |
A safe pilot keeps humans in the loop and preserves the existing fallback path.
Create a representative ticket set before the pilot. Include ordinary requests, missing data, policy exceptions, hostile wording, unsafe instructions, and examples that must go back to a human. Protect customer data while keeping the cases realistic.
Did the system find the correct policy section?
Does the draft match the real order state?
Did it answer, ask for clarification, or escalate correctly?
Did it stay within approved support rules?
Is the response clear and useful to the customer?
Did it reduce handling time without adding review burden?
| Workstream | Likely effort | Important assumption |
|---|---|---|
| Workflow discovery and data access | 1 week | Customer can provide SMEs and test access quickly |
| Core agent and order tool | 1–2 weeks | Order API is documented and stable |
| RAG and policy ingestion | 1 week | Policies are centralized and current |
| Evaluation and iteration | 1–2 weeks | Historical examples are available |
| Support-platform integration | 1–2 weeks | Draft workflow is supported by the platform |
| Pilot rollout and monitoring | 1 week | Small pilot group and human review remain in place |
A usable estimate A controlled, human-reviewed pilot is approximately five to eight weeks for a small team. The largest schedule risks are customer data access, support-platform integration, and policy quality. Direct autonomous actions are not included and would require a separate safety and authorization phase.
What this case teaches A strong FDE does not start by making the agent powerful. They start by making the workflow understandable, measurable, narrow, and safe.