How to tag customer feedback against your own taxonomy with an LLM

The pain points

The taxonomy is custom and it overlaps. "Onboarding", "setup" and "first-week experience" are three tags because three teams asked for them. A model cannot guess where one ends and the next begins.

One comment, several tags. "Pricing is fine, but the import kept failing" is about pricing and about data import. Forcing one tag loses one of the signals.

The rare tags are the important ones. Security concerns and churn threats are a small share of comments and most of their value. A tagger tuned on overall accuracy learns to ignore them.

People disagree. Two analysts tagging the same comments disagree more often than anyone expects. The model is being graded against that disagreement.

The taxonomy moves. A new feature ships and a new tag appears. Last quarter's labels do not know about it.

How to do it

Write each tag as a definition with boundary examples: one clear case, one near-miss that belongs to a neighbour tag. That document is the first draft of the prompt.
Double-label a sample and measure agreement per tag. A tag your analysts cannot agree on needs a better definition before it needs a model. Fix the definitions first; it is the cheapest improvement you will make.
Output a list, score every tag. Precision and recall per tag, macro-averaged so a rare tag counts as much as a common one. Put that average in the objective, not exact match on the whole list.
Optimize, then read the per-tag table and the diff. The useful rewrites sharpen the definition of a tag that was being confused with a neighbour. Check that is what changed.
Version the taxonomy with the prompt. When a tag is added or split, relabel a sample under the new definitions and re-run against that, instead of mixing old and new labels.

Pitfalls worth checking first

Pitfall	What it looks like	How to check, before spending anything
Frequent tags dominate	Overall scores rise while rare tags go unfound	Macro-average per-tag recall; read the rare tags on their own.
Agreement ceiling	Scores plateau well short of perfect	Compare against how often your analysts agree. That is roughly the ceiling.
"Other" as a crutch	A growing share lands in a catch-all tag	Read a sample of "other". It is usually a missing tag or a vague definition.
Mixed taxonomies	The same comment carries old and new tags	Relabel under one version of the taxonomy before scoring.
Sentiment bleed	Negative comments get tagged as problems	Keep sentiment a separate field from topic.

The pain points

How to do it

Pitfalls worth checking first

References