How to extract key terms from contracts

The pain points

• "Right" is undefined until you define it. "Twelve (12) months", "one year" and "an initial term ending on the first anniversary" are the same answer. If the label is free text, you are grading spelling.
• Missing is an answer. Most contracts have no most-favoured-customer clause. A prompt that always answers "not present" scores well on that field and is useless.
• The wrong scorer is worse than none. Exact match fails correct answers that were phrased differently. A model acting as judge is lenient, costs a call per row, and its taste becomes the thing you optimize.
• Long documents. The clause that matters is on page thirty, and an amendment signed later overrides it.
• Fields are not equally important. Getting the auto-renewal notice date wrong costs money. Getting governing law wrong rarely does.

How to do it

1. Write the schema first. Every field gets a type (date, duration, amount with currency, enumerated value, short text), a normalized form, and an explicit "not present". The schema is the contract between the prompt and everything downstream.
2. Label from records you already keep, such as the contract-management system or the renewal calendar, and spot-check a sample against the documents.
3. Score each field on its own terms. Equality on normalized dates and enumerations, a numeric tolerance on amounts, word-overlap F1 on short text. Keep each field as its own metric so you can see which one a rewrite broke.
4. Weight fields by what they drive, and score precision and recall separately on "present". That keeps "always say missing" from looking good.
5. Hold out by counterparty or template. Contracts from one paper template are near-duplicates. If they sit on both sides of the split, the score is flattering.

Pitfalls worth checking first

References

• scikit-learn, Metrics and scoring: balanced accuracy, per-class precision and recall
• Kaufman, Rosset, Perlich and Stitelman, Leakage in Data Mining: Formulation, Detection, and Avoidance, ACM TKDD, 2012
• Khattab et al., DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, 2023
• Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, 2025
• Quante Carlo, How prompt optimization works: reflection, the gate, the hold-out

Tools that run this loop: DSPy and its GEPA optimizer, bpto (ours, open source), and Impromptune, the studio built on it.