How to qualify inbound leads with an LLM

The pain points

Your ICP is a soft score, but you wrote it as hard rules. "50 to 5,000 employees, SaaS or fintech, on Salesforce or HubSpot, manager or above" reads to a model as a checklist. One model applies every box and rejects good fits that miss one. Another reads the same sentence loosely and lets everyone through. Same words, opposite behaviour.

Intent lives in the free text. Firmographics tell you whether a company could buy. Whether it is buying now is in what the person wrote: a named competitor, a budget, a quarter. A prompt that treats an empty message as "no intent" throws away every demo request with a blank box.

Your labels disagree where it matters most. Two SDRs rarely argue about a student or a competitor. They argue about "good fit, buying now" versus "good fit, not yet", which is exactly the call you want the model to make.

Your history only covers the leads someone chose to work. If a router already filtered the queue, the labelled rows are the ones it liked. A prompt tuned on them is tuned on the wrong population.

The data changes after the decision. Firmographics re-fetched today know which accounts grew, raised and bought. They are not what the SDR saw.

How to do it

Build the dataset from dispositions, as of routing time. Store the enrichment snapshot with the lead when it arrives, and train on that snapshot. In production, fetch it live. The model should only ever see what it will see when it is actually routing.
Drop anything written after the decision. Note length, call counts, stage changes, owner. They predict the label very well because the person who assigned the label wrote them.
Pick a metric that prices the mistakes the way you do. Rejecting on fit is usually the biggest class, so plain accuracy rewards a prompt that rejects nearly everyone. Missing a real buyer costs far more than an SDR spending five minutes on a lead that isn't buying. Score balanced accuracy (the mean of per-class recall), or weight each class by what a miss costs you.
Hold rows out and read the per-class table. An aggregate that improved because the prompt now rejects more leads is not an improvement. The hold-out is the number; the per-class table says what moved it.
Let an optimizer rewrite the rules, and read the diff. If your prompt applies its rules in order and the first match wins, a rewrite that appends advice underneath cannot override the rule that is misfiring. The useful change edits that rule: raise the headcount floor, name the industries that are out, count the whole revenue stack instead of the CRM alone.
Measure your label noise first. Have two people label the same sample and compute agreement. No prompt beats the rate at which your own team agrees with itself, so that rate is your ceiling.

Pitfalls worth checking first

Pitfall	What it looks like	How to check, before spending anything
Leakage	One column predicts the label far better than anything else	Was it written before the decision? If not, drop it.
Live re-fetch	The prompt scores better on refreshed enrichment than on the snapshot	Score both versions of the same rows. The gap is how much the refresh knows about the outcome.
Selection bias	Labelled rows skew toward demo requests and senior titles	Keep a small random slice of unrouted leads labelled, and weight rows by how likely they were to be labelled.
Redundant columns	Company and email domain always agree	Check whether one column determines another. Keep one.
The wrong metric	Accuracy rises while recall of real buyers falls	Put balanced accuracy or a cost-weighted score in the objective, and read the per-class table.

The pain points

How to do it

Pitfalls worth checking first

References