Data & evaluation sets

What not to trust in these numbers

Each run below is a single measurement.

None of the headline numbers above have been reproduced by an independent repeat run of the same configuration — each task that produced a run explicitly ruled out re-running it just to see a different number. The variance between repeat runs of the same model and prompt is unknown, and differences between runs below conflate prompt changes and, for some runs, model changes — see each run's own provider note for which variable that run isolates.

Not the target model.

gemini-3.5-flash-lite was used because it has a free tier, not because it is the model this project plans to run in production (that is gemini-2.5-flash). These numbers are not presented anywhere as a measurement of the target configuration.

No escalation.

0% escalation reflects the absence of a Claude API key, not a measured share of documents that would not need escalation under the real two-tier architecture.

confidence_threshold was not calibrated.

The orchestrator default (0.75) was used as-is — it has never been checked against this model's real output before this run.

The dataset is not a typical product input.

23 clean, institutional/financial World Bank PDFs and text documents — not the scanned or photographed forms with real-world noise the pipeline is ultimately meant for. Real accuracy on messy user scans could differ in either direction.

(v1 only) Some of that run's accuracy is an artifact of the ground-truth convention, not model ability.

Most visibly plan_date at 0% in v1 — the model consistently read a real, valid date from the document's human-readable preamble while the ground truth used a different, also valid, date from a machine-readable table, and the v1 prompt never specified which one to prefer. This is accounted for where it was separately computed (v1's procurement_plan genre accuracy rises from 63.7% to 74.6% with plan_date excluded), but it is not backed out of the v1 headline figure, which includes this effect as measured. v2 redefined plan_date (see that run's provider note); whether the same convention gap applies to v2/v3 has not been separately re-checked here.

One scoring normalizer is incomplete.

Currency-code aliasing (e.g. "US$" vs "USD") is not applied to the currency field, so part of its measured accuracy reflects a notation mismatch in scoring, not a content error by the model.