Define the decision
Define the extraction task narrowly. A material parser might return product name, variant, dimensions, document date and stated limitations. Write an expected record for each field, including explicit unknowns. Decide how a wrong value, omitted limitation or unsupported assertion could affect the downstream campaign decision.
Build a usable record
Assemble representative permitted examples: clean tables, scanned pages, mixed units, old revisions, contradictory variants and documents that lack a requested value. Include embedded instructions that should be ignored. Keep a reserved set for checking later changes. Synthetic examples should be labelled, while real documents require suitable access and usage permissions.
Check the evidence boundary
Compare extracted values with independently reviewed expected records. Record field-level errors and their consequence rather than reporting one attractive overall percentage. Check whether evidence references actually support the extracted facts. A model that frequently abstains may still be safer for a high-consequence field than one that confidently fills every blank.
Worked example — illustrative
In a fictional evaluation, an extractor reads twenty varied product documents. It confuses a carton dimension with the graphic size in two cases and invents a coating in one missing-data case. The team adds targeted rules and review steps, then checks the reserved cases. The example’s counts are illustrative, not a claim about Leaf software performance.
Put the method into practice
Use the review-finding template to capture each failed case, expected answer, actual answer and corrective action. Repeat the relevant evaluation when the model, prompt, source format or extraction schema changes. Deployment requires the connected engine’s own tests and authority checks; a written evaluation plan is not proof of production accuracy.
- Define exact fields and unknowns.
- Use varied permitted examples.
- Measure consequential field errors.
- Retest after material changes.