Document capture technology has come a long way in its roughly 3 decades of commercial application. But, much of that advancement has remained masked behind esoteric verbiage and abstruse pricing structures. Is there really anything new going on with today’s systems? It can be hard to tell from an outside perspective.
But, one truly substantial development in recent years has been the rise of Intelligent Document Capture (IDR) solutions. So, what exactly makes this technology different from what came before it? In this article we hope to de-clutter the scene, and demonstrate how IDR dramatically streamlines the capture of even highly unstructured, natural-language documents. We think lightweight modern systems that make use of IDR, such as Ephesoft, are creating huge new opportunities for firms to capture their most important data. Today we’ll discuss how this works, and what it means for your enterprise.
The Roots of Document Capture
In the early days of document capture technology, the only extraction method was Fixed Form processing. This approach was based on defining simple capture boxes within documents or pages, and the system would scan these zones for any content at all, and attempt to extract whatever was found. This approach was optimized for document types that were highly regular, where data would show up in predictable, describable geometric locations. Similar technology was devised for handling checkboxes, X’s, fill-in bubbles, and barcodes. This approach to capturing information defined the industry for a number of years.
An example of the kind of document for which Fixed Form processing is ideal.
But, Fixed Form processing had serious limitations. The majority of organized business documents are not rigid, but are semi-structured. For example, documents such as invoices, Explanations of Benefits, and so on; these kinds of records have generally predictable content, but there isn’t any way to predict where on the page key pieces of data would appear. For that reason, Fixed Form processing was never applicable to a huge portion of important business records. And the situation was even more difficult for totally unstructured data, such as free-written letters and email.
How IDR Fills the Gap
For these less-structured document types, the subfield of intelligent document recognition was developed. This new approach takes a content-based, rather than layout-based, approach to documents. Most modern capture solutions that utilize IDR depend on a pre-production learning phase, during which human operators provide example documents. The software then scans and analyzes all the words on every page in order to build a statistical model of word relationships and probabilities. For example, an operator may provide an example of both a mortgage document and a land usage document; the system will build a model that effectively notes the presence of terms like borrower, SSN, interest, and principal in the former document, while prioritizing words such as title, bounds, survey, easement, and so on for the latter. In actuality, this example is quite simplistic, whereas the extensive matrices that today’s systems can generate are quite nuanced and sophisticated.
Having created predictive models for these different types of documents, a modern capture system can then easily and correctly recognize other instances of the same document – e.g. two title surveys from the same company. But, much more usefully, it will also be able to correctly recognize and classify completely novel documents of the same type, like a title survey from a different surveyor, which might have an entirely different layout, and a handful of different terms too. How is this possible? Since IDR leverages probabilities rather than absolute relationships, it is flexible enough to tolerate slight differences in data. That novel set of title surveys might have somewhat different verbiage, but will likely retain > 90% of the same overall vocabulary because it is still a survey. This is the paradigm at the heart of IDR – document recognition in today’s solutions is no longer a rote and mechanical process, but is actually semantically-based, adaptable, and truly intelligent.
An IDR perspective on two documents from the same industry, but prepared by different firms — while they do not have identical geometric layout or keyword terms, there is enough semantic data in common that these two would be easily recognized as belonging to the same document class.
Today, state-of-the-art document capture solutions like Ephesoft leverage the power of IDR along with a mix of other different data extraction methods. Training a system like Ephesoft to recognize your documents can require as little as a couple of minutes, and this training and classification scheme can be modified on the fly whenever your needs or documents change. So, while it may seem that your data is too unpredictable or impractical to be scanned and extracted by automated systems, you might be surprised to see just how capable and agile today’s tools really are, and how valuable the results can be for your enterprise.