How to Evaluate a Document Indexing Provider

Start with your requirements, not the product

Before comparing providers, write down what the project must deliver: which document types and date ranges, which index fields per document type, the target system and its import format, and the accuracy you will accept. Providers can then respond to the same specification, which makes proposals comparable.

If you have not defined scope yet, start with scoping and budgeting a backfile project.

Capabilities to confirm

• Backfile and day-forward coverage: whether the same workflow handles historical records and new recordings.
• OCR on difficult material: handwritten entries, stamps over text, faded typewriter copy, microfilm-derived images.
• Field extraction: which fields are extracted, how document types are classified, and how multi-page instruments are handled.
• Metadata expansion: whether the provider can add useful fields beyond the original index, such as normalized names or parsed legal descriptions.
• Confidence and exception review: how low-confidence values are flagged, who reviews them, and how corrections are recorded.
• Search and export: keyword and field search over the results, and export in a format your existing system accepts.

Questions to ask

1. How do you measure accuracy — per document, per field, or per character? Can you report it per field and per document type?
2. What share of records do you expect to route to human review on material like ours, and how will we see that before full production?
3. Who performs exception review — your staff, ours, or both? What tools do reviewers use?
4. How do you handle documents you cannot classify or read?
5. Can you deliver a test export into our target system before the main project begins?
6. Where are images and data stored and processed during the project, and how are they returned or deleted at the end?
7. Can corrected values feed back into later batches so the same errors are not repeated?
8. What happens to day-forward recordings once the backfile is complete?

What a pilot should prove

A pilot should use a sample that reflects your hardest material, not only clean recent documents. It should produce field-level accuracy figures measured against a verified answer key, an actual exception count, and a test import into the target system. See running a pilot batch for a step-by-step approach.

Red flags

• Claims of complete automation or perfect accuracy with no human review.
• Accuracy figures that cannot be reproduced on your documents.
• Reluctance to run a pilot on a representative sample.
• No clear answer about export format or how data reaches your existing system.
• Pricing that excludes exception review, rework, or import support without saying so.

A short evaluation checklist

• Written requirements shared with every provider
• Field-level accuracy measured on our sample
• Exception volume and review responsibility agreed
• Test export loaded into our target system
• Data handling and return/deletion terms documented
• Day-forward plan confirmed
• Procurement requirements of our jurisdiction followed