GigaOm Performance Benchmark for Intelligent Document Processing (IDP) Solutionsv1.0

1. Executive Summary

This benchmark report aims to compare the performance of two intelligent document processing (IDP) solutions offered by Automation Hero and ABBYY in terms of accuracy of data extraction and handwriting recognition. The first aspect of this performance benchmark test is extracting data from different invoices (in this case, hotel receipts). Both products were deployed in the cloud, and testing was done on the same set of invoices. The use case is IDP-enabled invoice processing for accounts payable (AP) automation. The other aspect is comparing performance regarding the accuracy of data extracted from handwritten documents (specifically, snippets of handwritten text from actual medical forms).

The document structures of these invoices were identical to transparently identify how each IDP solution would treat an unfamiliar invoice layout. Both IDP products processed the same document sets. This provision ensures a neutral collection of test documents that neither IDP product is pre-trained to recognize.

We found that the Automation Hero Hero Platform delivered 68% greater accuracy than ABBYY FlexiCapture in terms of global average (“headers” and “line items” combined) for invoice processing. In terms of headers (e.g., invoice number, invoice date, amount, customer name, customer address, etc.), the Hero Platform delivered 67% greater accuracy than ABBYY FlexiCapture IDP product. And for line items in invoices, the Hero Platform delivered 69% greater accuracy than ABBYY FlexiCapture.

For the other use case, handwriting recognition for snippets from actual medical forms, Automation Hero context-aware optical character recognition (OCR) delivered 281% greater full-field accuracy when compared to ABBYY FlexiCapture.

2. Drivers for IDP Data Extraction Accuracy and Handwriting Recognition Tests

Enterprises have long used OCR tools to extract data from documents containing structured and semi-structured data. However, the limitations of the OCR tools are hard to ignore. OCR tools, lacking intelligence of their own, can hardly deliver accuracy greater than 60% in data extraction. This is a best-case scenario using good-quality documents and good-quality scanners; in most cases, the accuracy can fall below 50%.

Given that the remainder of the document processing must be done manually, including for exceptions (like false positives, such as l/1, m/nr, and 5/S), the amount of human involvement and effort required makes using OCR tools for bulk document processing a cumbersome exercise.

Over the last decade or so, several vendors tried to experiment with a combination of Python code and OCR in what was termed “intelligent OCR.” While there was an increase in data extraction accuracy (+5-10%), the results were still not in line with the requirements of IT and automation center of excellence (CoE) leaders. IDP as a software product involves pre-processing and post-processing stages: the first primes the document in terms of shape, size and orientation, and the second ensures a greater degree of accuracy in processing unstructured data can be achieved.

The availability of artificial intelligence (AI), machine learning (ML), natural language processing (NLP), and deep learning capabilities with IDP ensures that the strike rate for processing data is significantly higher than what a mere OCR tool can deliver. IDP tools, in principle, are supposed to function with minimal training of minor template changes, whereas with OCR, this type of flexibility is nonexistent.

OCR tools have individual strengths and weaknesses; some excel in handwriting recognition, while others work best for image processing. So far, IDP solutions have struggled to tackle handwritten text other than block letters. Free-flowing text or cursive handwriting is not something an IDP solution can readily process with a significant level (>60%) of accuracy.

Human-in-the-loop (HITL) capability allows knowledge workers to manually correct values and text for certain information fields flagged by the IDP product for human intervention. Needless to say, the greater the proportion of straight-through processing (STP) delivered by an IDP product, the less the need for human intervention. This is particularly important for use cases where the volume of documents to be processed is significantly higher (e.g., an organization with 100 vendors and each vendor with multiple invoice formats).

Know Your Customer (KYC), invoice processing, insurance claims, patient onboarding, patient records, proof of delivery, bill of lading, and order forms are some of the key use cases for IDP solutions. IDP software is of good use in industry-specific processes, such as customer onboarding, mortgage processing, trade finance, and legal documents.

3. Comparing Vendor Offerings

The Hero Platform and ABBYY FlexiCapture were the two IDP products used for this test. The version numbers of the two IDP products used were:

• Automation Hero Hero Platform (version 6.9)
• ABBYY FlexiCapture (version 12.0.3.4164)

Automation Hero Hero Platform
Automation Hero provides an AI engine for document processing and automation. Its deep learning-based approach can turn structured or unstructured documents (e.g., contracts, invoices, receipts, prescriptions, and doctor notes) into highly accurate and actionable data.

Automation Hero’s context-aware OCR capabilities leverage a custom knowledge graph. This knowledge graph is created based on thousands of potential values for the tested form fields. The Hero Platform uses AI to directly use this data when making a prediction, leveraging the customer team’s stored knowledge and experience to automatically improve results.

ABBYY FlexiCapture
ABBYY FlexiCapture offers NLP, ML, and recognition capabilities via an enterprise-grade document capture platform. It captures, classifies, and transfers critical data from unstructured and structured documents. To orchestrate the process from acquisition to delivery, FlexiCapture can ingest data and integrate with content-driven business applications (e.g., RPA and BPM tools).

ABBYY FlexiCapture is available in the cloud, on premises, or as a software development kit (SDK). The recent release of FlexiCapture with NLP capabilities expanded capture capabilities to support unstructured documents, such as contracts, leases, articles, and agreements.

4. Test Results

We used 29 invoices (i.e., hotel receipts) for the first phase of this test. These invoices had different layouts and belonged to various vendors; neither of the two IDP products was pre-trained on any of the invoice layouts. Invoices were in different languages (predominantly English and a few Croatian) and Roman script. JPEG files of invoices were first converted to portable document format (PDF) and then ingested into the two IDP products for data extraction and processing.

Two information fields were extracted from the hotel receipts: headers and line items. Header fields include invoice number, invoice date, amount, customer name, customer address, and vendor name. Line items include other key information fields that need to be extracted, such as daily room rent, tax amount, tax rate, and so forth. Document quality varied from one hotel receipt to the other; however, a significant share of documents was of good quality.

The results from the first phase of this test are shown in Figure 1. Here, field-level accuracy implies extraction of the correct field (e.g., invoice number cannot be “text” or “name” or IBAN, it has to be numeric or alphanumeric in format) and the extracted value/field data matches the value in the invoice (e.g., customer name can be “Jens Walter” but not just “Mr.”).

Overall, the Hero Platform delivered 68% greater accuracy in terms of the global average (both headers and line items combined) compared to ABBYY FlexiCapture. For headers, Hero Platform delivered 67% greater accuracy compared to ABBYY FlexiCapture. Hero Platform’s data extraction accuracy was 69% greater than ABBYY FlexiCapture for line items.

Figure 1. Automation Hero Delivered 68% Greater Overall Accuracy in Invoice Processing

The second phase of the test involved handwriting recognition for snippets from actual medical forms. 958 snippets with handwritten text were processed using the two IDP products. Figure 2 presents some samples of the snippets from medical forms used for testing.

Figure 2. Handwriting Samples

Here, we assessed handwriting recognition accuracy in terms of whether the full extracted field information exactly matches the dictionary term for referring to specific ailments, health issues, and diseases (e.g., arthritis, seizure, and dermatitis) or not. Merely extracting information from a snippet with handwritten text would not qualify as a positive match. Table 1 summarizes the results of the handwriting recognition test, while Figure 3 charts the relative performance of the two products.

Table 1. Comparison of Handwriting Recognition Accuracy, Hero Platform and ABBYY FlexiCapture

Note that Hero Platform offers context-aware OCR capabilities leveraging a custom knowledge graph. This knowledge graph is created based on thousands of potential values for the form fields being tested.

Figure 3. Automation Hero Delivered 281% Greater Accuracy in Handwriting Recognition and Extraction

Hero Platform, with its context-aware OCR capabilities, delivered 281% greater accuracy in terms of handwriting recognition.

5. Conclusion

Data extraction accuracy is a key factor when selecting an IDP solution, which have struggled to meet expectations for processing handwritten text other than block letters. Free-flowing text or cursive handwriting is not something an IDP solution can process with a significant level (>60%) of accuracy.

The results from this performance benchmark test reveal that the Hero Platform delivered 68% greater accuracy (global average) for invoice processing when compared to ABBYY FlexiCapture. In terms of handwriting recognition, Hero Platform, with its context-aware OCR capabilities, delivered 281% greater accuracy compared to ABBYY FlexiCapture.

This test was conducted on actual invoices (i.e., hotel receipts) and handwritten text in snippets from real medical forms. It therefore provides a practical view of the performance of the two IDP products. Enterprise IT leaders, automation CoE leaders, and process leaders can refer to the results from this test to gain insight into the intricacies of two key IDP use cases—invoice processing and handwriting recognition—and select an IDP solution suitable for their specific business requirements.

6. Appendix

Environment Summary
The performance benchmark environment was designed to create as close to an apples-to-apples comparison as possible. The benchmark test involved two IDP products: ABBYY FlexiCapture (version 12.0.3.4164) and Hero Platform (Version 6.9). Both the IDP products were deployed in the cloud (AWS IaaS environment).

No additional tools or custom scripts were used for this test. Moreover, no customized configurations were used for either of the two IDP products. The benchmark tested the extraction of data from various invoices (i.e., hotel receipts) and snippets from medical forms having handwritten text; there was provision for using pre-trained AI/ML models with the IDP products.

Testing Configuration
The test involved invoice samples (29 hotel receipts in various languages with Roman script and different formats) and 958 handwritten snippets from actual medical forms. The document structures of these invoices were identical to transparently identify how each IDP solution would treat an unfamiliar invoice layout.

Both IDP products processed the same document sets. This provision ensures a neutral collection of test documents that both IDP products are not pre-trained to recognize.

For this performance test, the key attribute of comparison was the accuracy of data extraction at field level from various invoices, without any pre-training on the invoice layouts. The other evaluation parameter was the comparison of extraction accuracy for handwriting recognition based on tests conducted on medical form snippets with handwritten text.

GitHub
All results will be stored on GigaOm’s GitHub account and made publicly available after the completion of the test, excluding original document samples (as these contain personally identifiable information). This is to help users repeat the test if necessary and validate the test results independently. The GitHub repository includes only configuration files, scripts (none used in this test), and raw results.

7. Disclaimer

Data extraction accuracy is important, but it is only one criterion for selecting an IDP solution. This test is a point-in-time check into specific performance. There are numerous other factors to consider in selection across administration factors, features, functionality, user experience, scalability, vendor reliability, and many other criteria. It is also our experience that performance can change over time and be competitively different for multiple use cases and versions of the products. Also, a performance leader can hit against the point of diminishing returns, and viable contenders can quickly close the gap.

GigaOm runs all its performance tests to strict ethical standards. The report’s results are the objective findings of the data extraction accuracy derived from the application of the two products to two key use cases: invoice processing and handwriting recognition. The report clearly defines the selected criteria and process used to establish the actual test. It also clearly states the products and versions used for this test. The reader is left to determine how to qualify the information for their individual needs. The report does not make any claim regarding the third-party certification and presents the objective results received from the application of the products to the use cases as described in the report. The report strictly measures performance in terms of data extraction and handwriting recognition and does not purport to evaluate other factors that potential customers may find relevant when making a purchase decision.

This is a sponsored report. Automation Hero chose the competitor and the test configuration. GigaOm selected the most compatible configurations as-is out-of-the-box and ran the test with document samples (actual invoices and snippets from medical forms). Choosing compatible configurations is subject to judgment and not an indication that similar performance characteristics can be obtained for other configurations and/or versions of the products.

8. About Automation Hero

Automation Hero helps organizations process various types of documents faster with an IDP platform. It offers easy-to-use AI capabilities so organizations can focus on accelerating business processes to stay competitive. Automation Hero’s context-aware approach and AI-driven handwriting recognition aim to improve data processing accuracy and increase the degree of STP.

9. About GigaOm

GigaOm provides technical, operational, and business advice for IT’s strategic digital enterprise and business initiatives. Enterprise business leaders, CIOs, and technology organizations partner with GigaOm for practical, actionable, strategic, and visionary advice for modernizing and transforming their business. GigaOm’s advice empowers enterprises to successfully compete in an increasingly complicated business atmosphere that requires a solid understanding of constantly changing customer demands.

GigaOm works directly with enterprises both inside and outside of the IT organization to apply proven research and methodologies designed to avoid pitfalls and roadblocks while balancing risk and innovation. Research methodologies include but are not limited to adoption and benchmarking surveys, use cases, interviews, ROI/TCO, market landscapes, strategic trends, and technical benchmarks. Our analysts possess 20+ years of experience advising a spectrum of clients from early adopters to mainstream enterprises.

GigaOm’s perspective is that of the unbiased enterprise practitioner. Through this perspective, GigaOm connects with engaged and loyal subscribers on a deep and meaningful level.

10. Copyright

© Knowingly, Inc. 2022 "GigaOm Performance Benchmark for Intelligent Document Processing (IDP) Solutions" is a trademark of Knowingly, Inc. For permission to reproduce this report, please contact sales@gigaom.com.