DocsICP FilingConsole
官方文档
首页

Form and receipt information extraction

更新时间：
复制 MD 格式
产品详情
我的收藏

Feature overview

  • Form and receipt information extraction for fixed-layout documents is a task that uses a self-learning model based on deep learning. It performs well on documents with consistent layouts, such as forms, certificates, and vouchers. You can define custom extraction fields. The platform provides visual tools for data annotation and model training.

  • With good image quality and annotations for over 100 training samples, a fine-tuned model can achieve a recognition accuracy of over 95%.

  • The Toolbox also provides a classifier management tool and a field type management tool. These tools allow you to automatically classify, route, and recognize data from different layouts with high precision using a single API operation.

Benefits

  • High precision: Based on Alibaba Cloud's powerful pre-trained models, a fine-tuned multi-layout model can achieve an accuracy of over 95%.

  • Few-shot learning: You only need to annotate a small amount of data to optimize and iterate the model. The model also has good generalization capabilities.

  • Low barrier to entry: This feature is out-of-the-box and requires no code development. You can configure rules independently through a user-friendly and controllable interface.

  • High efficiency: The intelligent pre-annotation feature and support for collaborative annotation save time.

Scenarios

  • This feature is suitable for images or single-page PDF files where layouts are relatively fixed or enumerable (up to 50 types) and you need to extract key-value information in a structured way.

  • It is also ideal for scenarios involving forms and receipts where the model requires continuous optimization and many data samples are available for training. Examples include financial and tax reimbursement using business licenses and medical invoices, qualification verification, and structured recognition of internal business documents.

Related links

User guide

For a video tutorial on using form and receipt information extraction, see the following video:

The process of creating a form and receipt information extraction model is shown in the following figure. You need more than 20 images for training to create a model.

image

Step 1: Data preparation

In the Data Center, go to Datasets to upload and manage the data for your model task. Click Add Dataset to open the upload page. Enter a Dataset Name and upload your fixed-layout business data.

Note

A custom model for form and receipt information extraction requires at least 20 training images to achieve good recognition and extraction performance.

Tips:

  1. What are the specific requirements for data preparation?

    1. You can upload images, documents, or compressed packages to a dataset.

      1. Documents: Must be single-page PDF files no larger than 20 MB.

      2. Images: Must be JPG, JPEG, or PNG files no larger than 10 MB.

      3. Compressed packages: Must be ZIP files no larger than 20 MB.

    2. For a single image, the longest side cannot exceed 8,192 pixels, and the shortest side must be at least 15 pixels. If the longest side exceeds 1,024 pixels, the aspect ratio cannot exceed 50:1.

    3. Prepare at least 20 to 30 data samples of the same task type for model training and evaluation.

  2. How can I obtain better recognition results?

    1. For tasks within the product's scope, higher data quality leads to better recognition and extraction performance. Clear and clean data results in higher accuracy.

    2. Keep the size of a single character between 10 and 50 pixels for better recognition.

    3. The data is from real business scenarios and covers a comprehensive range of types and formats.

Step 2: Data annotation

Data annotation consists of three main steps: creating an annotation task, annotating the data, and performing a quality inspection.

Create an annotation task

image

On the Annotation Tasks page in the Data Center, click Create Annotation Task. On the creation page, enter a Task Name. In the Upload Data section, select a dataset or upload files from your local machine. Then, proceed to Field settings.

Layout analysis: If the forms or receipts in the images are rotated or distorted, or if the document area covers less than 70% of the image, select this option. You will then need to draw a bounding box around the document in the annotation step.

Pre-annotation: If you enable OCR pre-annotation, the text inside a box you draw will be automatically recognized. This improves annotation efficiency.

Question Bank: You can browse and select existing questions to create an annotation task.

Field Name: The name of the field that is exposed externally, which corresponds to the name in the API operation. The field name must be globally unique.

Field Type: The property definition for the field. Selecting an appropriate field type can improve the end-to-end recognition performance. You can choose from common fields or add your own custom fields. If no post-processing is needed, select the Regular field type.

Tips:

  1. How do I upload data for an annotation task?

    1. You can upload data from your local machine or select data from a dataset. If you have already uploaded data to a dataset, click Add Dataset above the list and select the dataset you want to annotate.

  2. How do I configure key-value information extraction?

    1. Enter all the field names you need to annotate into the content box. Select the appropriate field type for each to improve recognition accuracy.

    2. You can select a pre-created template from the Field Template Library to use its fields. For the same model, we recommend using the same template to ensure the field settings are identical.

Annotation

On the Annotation Tasks page in the Data Center, select the task you created and click Annotate to go to the Data Annotation interface. In the annotation tool, use the Bounding Box button to draw boxes around the fields you want to recognize. Select the corresponding field name and carefully check the automatically recognized text. After you have annotated all fields on all images, click Submit Task to complete this part of the annotation.

Important

The quality of the annotated data, including text and position, directly affects the model training performance and evaluation metrics.

image

If you encounter incorrect or unannotatable data, you can Skip that image.

Tips:

  1. How do I use the bounding box tools?

    1. For areas with straight text, select Rectangle to draw a box. For areas with tilted text, select Polygon Annotation. Make the box fit the field as closely as possible to improve recognition performance.

    2. The Polygon Annotation tool does not support pre-recognition. You must manually enter the recognition result into the box.

    3. OCR pre-annotation supports printed text but not handwritten text. Carefully check the text that is automatically recognized to ensure it is correct.

  2. What card or certificate type should I select?

    1. Target area: The area where the form or receipt is located. This corresponds to the outer border of the document and is used for positioning.

    2. KV area: The key-value fields on the form or receipt that need to be recognized.

    3. If the form or receipt occupies less than 70% of the image, we recommend selecting the target area to lock the recognition area.

  3. How should I draw boxes to obtain the best results?

    1. Make the annotation box fit the field as closely as possible. If the text is tilted or irregular, use the Polygon Annotation tool to make the border fit the text tightly. For dense text or text with texture interference, more detailed annotation is required to achieve better results after the model is trained.

    2. If a single field has multiple lines of content, draw a box for each line and assign the same field name to each line.

Quality inspection

Go to the Annotation Tasks page in the Data Center and select a completed annotation task to inspect its quality. The quality inspector can modify or reject annotations. After the inspection is complete, submit the task. Make sure that all fields have been annotated.

image

Tips:

  1. How do I perform a quality inspection?

    1. Quality inspectors can use the annotation tool to add, delete, or modify annotation information directly.

    2. The quality inspector completes the quality inspection for all images by selecting Submit Task.

  2. What should I do if I find that the annotation quality is low during inspection?

    1. Select the relevant field and edit it directly.

    2. Select Reject to return the task to the annotator for reprocessing.

Step 3: Model training and evaluation

Go to the Model Hub and click Create Model to open the model creation page. Select the annotated results for the training set and test set, and enter the basic information. After the model is created, training starts automatically.

image

Training set: The data source used to train the model. You can only select datasets that have been annotated and passed quality inspection. A dataset selected as a test set cannot be used as a training set. We recommend using more than 20 valid images for model training.

Test set: The data source used to test the model. You can only select datasets that have been annotated and passed quality inspection. A dataset selected as a training set cannot be used as a test set.

Automatically use 1/10 of the training set as the test set: If you enable this option, you do not need to manually select a test set. The system automatically partitions 1/10 of the training set to use as the test set. If you have already uploaded a test set before enabling this option, the system will ignore the manually uploaded data.

image.png

image.png

Model training fees and estimated duration. The cost and time vary based on the task type and data volume. For more information, see Billing of OCR Document AutoML.

Training duration: This depends on multiple factors, such as data volume, annotation quality, and machine resources. For example, on a V100 instance, training on 20 images takes about 1.5 hours, while 200 images take about 3 hours.

Tips:

  1. You need to verify the fields in the test set and training set. The field content of each dataset must be consistent.

  2. During model training configuration, you can select the field type for each field or delete fields as needed to improve training accuracy.

  3. The larger the data volume and the more precise the annotation, the better the model training and evaluation results will be.

  4. Configure appropriate field types to improve end-to-end accuracy.

image

Algorithm evaluation metrics: These include the model's overall accuracy, single-field accuracy, and detection F1-score.

  • Overall accuracy: The accuracy of the algorithm model before rule-based post-processing corrections. It is the ratio of correctly recognized fields to the total number of fields in the test set, including content and position.image

  • Single-field - Detection F1-score: A comprehensive evaluation metric (F1-Measure) before rule-based post-processing corrections. It is the weighted harmonic mean of the precision and recall rate of the field detection box. It is often used to evaluate the performance of a detection model.image

  • Single-field - Accuracy: The field recognition accuracy of the algorithm model before rule-based post-processing corrections. It is the ratio of the number of correctly recognized instances of a field to the total number of instances in the test set (the union of annotated boxes and predicted boxes, including content and position).image

Fine-tuning suggestions:

How do I fine-tune the model?

  1. Make sure the task type is within the feature's scope and that the data is as clear and clean as possible.

  2. During data annotation, annotate each recognition field completely. Make the annotation box fit the text as closely as possible. You can also adjust field types or advanced options to improve accuracy.

  3. The larger the data volume, the better the model training performance. A model generally performs well with a data volume of 100 or more samples.

  4. Ensure the data types and layout distributions match real business scenarios. The model can only achieve good recognition performance for data types and layouts it has been trained on.

How do I copy an annotation task?

  1. To fine-tune an existing annotation task, you can click Copy in the task's operations column. This creates a new annotation task with the original annotations, which you can then adjust.

Step 4: Model deployment

After model training is complete, go to the model details page in the Model Hub. Click the Deploy button at the bottom of the page to start model deployment. Deployment takes some time. After the model is successfully deployed, you can test its performance visually in the online experience or call the service directly using the API.

image.png

Note

After you publish and deploy the model service, refer to the API overview document to use the service. You can also go to the Online Test page to test the model's performance.

Important

The OCR self-learning feature was fully commercialized on August 23, 2023. Model training is billed based on duration, and model inference is billed based on the number of calls. For more information, see OCR Self-learning Billing.

Tools - Question Bank

A field template library is used in the Field settings step. You can pre-create fields for an annotation task and reuse the template multiple times. If multiple annotation tasks require the same fields, you can select a template from the library. This reduces the manual effort of editing fields and lowers the chance of errors.

Note

Modifications to the question library do not affect annotation tasks or models that are in progress.