How to triage and escalate support tickets with an LLM

The pain points

The message is not enough. "My invoice is wrong" is billing for one customer and a retention case for another whose contract renews next month. The deciding facts live in your CRM: plan, open tickets, renewal date. A prompt that never sees them has a ceiling that no rewording can lift.

Escalations are rare and expensive. A prompt can be right on almost every ticket and still miss the few that needed a senior engineer today. Aggregate accuracy hides that completely.

The first queue is not the right queue. Tickets get reassigned. If you train on the queue a ticket first landed in, you train on the old router's mistakes.

Two prompts, one score. Summarize-then-route is a sensible design. But when the route is wrong, was the summary missing the detail, or did the router misread it?

The taxonomy churns. Queues get split, merged and renamed. Labels from last quarter disagree with today's org chart.

How to do it

Label from the final queue, after reassignment, and date every row. The reassignment history is also free training data: every move is a correction someone made by hand.
Fetch account context as an external lookup, and freeze it. Look up plan, entitlement and open tickets once, when the row is captured, and store the answers with the row. Train on those stored values; call the lookup live in production. A ticket was an escalation because of what was true that day, not what is true when you re-run the evaluation.
Treat the pipeline as a program and optimize one prompt at a time. Rewrite the summarizer or the router in a given round, never both, so that when the score moves you know which change moved it.
Score escalation separately. Exact match on the queue, plus recall on the escalation class as its own metric in the objective, weighted by what a missed escalation costs.
Ship a version, then close the loop. Serve a fixed version of the prompts, log every request, and record reassignments as outcomes. Those corrections are next month's dataset.

Pitfalls worth checking first

Pitfall	What it looks like	How to check, before spending anything
First-queue labels	The prompt learns the old router's habits	Compare first and final queue on a sample. If they differ often, label from the final queue.
Time travel	Account data fetched at evaluation time, not ticket time	Freeze the lookup onto the row when the ticket arrives.
Rare escalations	Accuracy is high, escalations are missed	Read escalation recall on its own. Put it in the objective.
Renamed queues	Old and new labels for the same work	Map old queue names to current ones before training, or drop rows from before the change.
Unused context	The lookup returns five fields and the prompt uses two	Check which fields actually change the answer. Stop pasting the rest into every call.

The pain points

How to do it

Pitfalls worth checking first

References