Chapter 01
The challenge
A two-person support team was spending most of its day categorising and routing tickets rather than answering the hard ones.
Getting from there to a working product was not only a matter of writing code. A few things made this project harder than it first looked:
- Help-centre articles were out of date in places, so naive retrieval produced confident wrong answers
- The team would not accept a tool that sent anything to a customer unreviewed
- Ticket volume was spiky, which made an uncapped per-request cost a real risk
- There was no baseline for what a good answer looked like
Chapter 02
What we did
Retrieval over their help centre and past resolutions, a classifier for routing, and a drafted reply a human approves before it sends.
1.Evaluation set first
Several hundred past tickets with their accepted resolutions became the test suite every prompt and retrieval change is scored against.
2.Grounded drafts with sources
Replies are drafted only from retrieved articles and past resolutions, and cite them so an agent can check in seconds.
3.Human approval by default
Drafts appear inside the helpdesk the team already used. An agent sends, edits or discards. Nothing goes out on its own.
4.Routing and cost control
A small model classifies and routes, the larger model drafts only when needed, and repeated questions are served from cache.
Chapter 03
The outcome
Headline result
First-response time cut by more than half
- Draft replies accepted without edits on routine tickets
- Evaluation suite run on every prompt change
Chapter 04
Built with
The tools chosen for this project, each for a reason specific to the problem rather than out of habit.
- Claude
- Python
- FastAPI
- pgvector
- PostgreSQL
