PDF parsing

Prerequisites

• To use the workflow feature, you must first purchase the unstructured data feature for the instance.

• You have created a workflow. For more information, see Create a workflow or .

Background information

• The PDF parsing operator supports files in PDF format.

• By default, the PDF parsing operator uses 1 core and 4 GB of resources.

Permissions

In a Basic project, project administrators, developers, analysts, and custom project roles that have the Dataset - Use permission can use all datasets in the project as input and output datasets for the operator.

Procedure

1. On the Dataphin homepage, choose Develop > Data Development from the top menu bar.

2. On the Develop page, choose Project from the top menu bar.

3. In the left navigation bar, select Data Processing > Workflow. In the Workflow list on the left, click the target workflow to open the workflow canvas. Drag the PDF Parsing operator from the Operator Library on the left to the canvas and configure the following parameters in the PDF Parsing panel on the right.

   Parameter		Description
   Step name		Enter a name for the operator. Any characters are supported. The name cannot exceed 256 characters.
   Input settings	Dataset	Select the dataset to process. You can select a Hybrid Dataset or a Table Dataset. After you select a dataset, you must also select its Version. By default, the system selects the first version in ascending order. When this operator is used downstream of another operator, the dataset selected here is usually the same as the output dataset of the upstream operator.
   	Input field	The storage path of the PDF file. Select a source field from the input dataset. You can select only fields whose Is URL parameter is set to Yes.
   	Filter (Optional)	Use SQL syntax. Enter only the filter condition that follows the WHERE clause. You can use global system variables, such as the business date ${bizdate}.
   Operator settings	Model-assisted correction	If you disable this option, the operator uses only the built-in model for layout parsing and table extraction. If you enable this option and the built-in model performs poorly, such as causing layout misalignment, the operator automatically calls a large model to correct the output and improve parsing accuracy.
   	Image content understanding	If you enable this option, the operator calls a large model to parse image content in the PDF. In the output Markdown, the operator replaces images with text descriptions. You must also enter a Prompt for Image Understanding of up to 2,000 characters.
   	Model	You can configure this parameter only when you enable Model-assisted Correction or Image Content Understanding. Both features use the same model. You can select only models that are Enabled and whose model type in Intelligent Application Management > Model Configuration includes Text or Vision (Image and Video).
   Output settings	Dataset	You can select Source Dataset or Other Dataset. If the source dataset is a table dataset, the system selects Other Dataset by default. When you select other datasets, you must also specify a mixed dataset and its version as the output dataset. The default is the first version of this dataset in ascending order. You can also select other versions of the source dataset. When you select a source dataset, the metadata that the operator generates is written to the metadata table for the same version of that dataset.
   	Include image links in output	Select Yes to replace parsed images with links to their storage locations. Select No to remove image links.
   	File output path	Enter a directory name. If the directory does not exist, the operator creates it. The parsed text files are named in the format${PDF file name + auto-incrementing number starting from 1}.md. The operator generates a folder for the parsed images and stores them in it.
   	Load policy	If the metadata table of the selected output dataset has a primary key, update upon primary key conflict is selected by default. Otherwise, append data is selected by default. Append data: Appends new data directly to the target table. An error occurs if a primary key or constraint conflict occurs. update upon primary key conflict: If a primary key or constraint conflict occurs, the operator deletes the existing row with the duplicate primary key before inserting the new data. overwrite: Deletes all existing data from the target table before writing the new data. Note When Source Dataset is selected as the output dataset, the only supported load policy is update upon primary key conflict.
   	Field mapping	The PDF parsing operator outputs the following fields: markdown_url: The storage path of the Markdown text. markdown_content: The content of the Markdown file. This field is optional. If the content is very long, do not map this field to the output dataset. image_folder_url: The storage path of the image folder. By default, the operator's output fields are mapped to fields with the same names in the output dataset. You can also manually select target fields for custom mapping. In addition, you can map pass-through fields from the source dataset to the target dataset. You can manage field mappings in the following ways: Add output field: Click to add a new, empty field mapping row. This feature is available only when the output dataset is set to Other Dataset. Batch map: Click to choose batch manual mapping, same-name mapping, or same-row mapping. This feature is supported only when the output dataset is set to Other Dataset. Delete mapping: Click the Delete icon to delete the field mapping for the corresponding row. You cannot delete mappings for the operator's fixed output fields.

4. After you complete the configuration, click Save in the top menu bar of the canvas to save the configuration.

   After configuring a single operator, you can continue to build a complete workflow by selecting upstream or downstream operators.