Extract HTML body text

Prerequisites

• The workflow feature is available only if you have purchased the unstructured data feature for your instance.

• A workflow has been created. For more information, see Create a workflow.

Background information

• The HTML body extraction operator supports files in the following formats: .txt, .md, .csv, .json, .xml, and .html.

• By default, the HTML body extraction operator uses 1 Core and 2 GB of resources.

Permissions

In a Basic project, the following roles can use any dataset in the project as an input or output for the operator: project administrators, developers, analysts, and custom roles with the Dataset - Use permission.

Procedure

1. On the Dataphin home page, choose Development > Data Development from the top menu bar.

2. On the Development page, choose Project from the top menu bar.

3. In the navigation pane on the left, choose Data Processing > Workflow. In the Workflow list, click the target workflow to open its canvas. From the Operator Library, drag the HTML body extraction operator to the canvas. Then, in the HTML body extraction panel on the right, configure the parameters.

   Parameter		Description
   Step name		Enter a name for the operator. The name can contain any characters and be up to 256 characters long.
   Input configuration	Dataset	Select the dataset to process. You can select a Mixed dataset or a Table dataset. After you select a dataset, select a version. The default version is the first version of the dataset. When this operator is a downstream operator, the selected dataset is usually the same as the output dataset of the upstream operator.
   	Input field	The text or document URL to process. The system automatically detects the field type: If the source field is a URL type field, the operator extracts and processes the content of the document that the URL points to. The operator can read files in .md, .txt, .doc, and .html formats. If the source field is not a URL type field, the operator treats the field content as plain text to be processed. Select a source field from the input dataset. You can select any field from the metadata table of the input dataset.
   	Filter condition (Optional)	Use SQL syntax. Enter only the filter condition that follows the `WHERE` clause. You can use system global variables, such as the data timestamp `${bizdate}`.
   Output configuration	Dataset	You can select Source Dataset or Other Dataset. If the source dataset's metadata table does not have a primary key, the output dataset defaults to Other Dataset. If you select Other dataset, you must also specify a dataset and its version as the output dataset based on the input field type. The default version is the first version of the dataset. You can also select other versions of the source dataset. If the input field is a URL type field (a file is generated after processing), you can only select Mixed dataset for the output dataset. If the input field is not a URL type field (no file is generated after processing), you can select Mixed dataset or Table dataset for the output dataset. If you select Source dataset, the metadata generated by the operator is written to the metadata table of the same version of the source dataset.
   	File output path (Optional)	Enter the path for the output file. This parameter is available only when the input field is a URL type.
   	File output name	This parameter is displayed only when the input field is a URL type. The name of the generated file is in the format ${Source file name}_html_body_extraction.md. You cannot change this format.
   	Loading Policy	If the metadata table of the selected output dataset has a primary key, the default policy is to update data when a primary key conflict occurs. If no primary key exists, the default policy is to append data. Append data: Directly appends new data to the destination table. If a primary key conflict or constraint violation occurs, an error is reported. Update on primary key conflict: If a primary key conflict or constraint violation occurs, the operator first deletes the entire row of old data with the duplicate primary key, and then inserts the new data. Overwrite: Deletes the data in the destination table before writing the new data. Note If you select Source dataset for the output dataset, only the Update on primary key conflict loading policy is supported.
   	Field mapping	If the input field is a URL type, the HTML body extraction operator outputs only the fixed markdown_url field, which is the storage URL of the output file. If the input field is not a URL type, the HTML body extraction operator outputs only the fixed processed_text field, which is the extracted body content. By default, the operator's output fields are mapped to fields with the same names in the output dataset. You can also manually select destination fields for custom mapping. In addition, you can map pass-through fields from the source dataset to the destination dataset. You can manage field mappings in the following ways: Add output field: Click to add a new, empty field mapping row. This feature is available only if you select Other dataset for the output dataset. Batch mapping: Click to choose between batch manual mapping, same-name mapping, or same-row mapping. This feature is available only if you select Other dataset for the output dataset. Delete mapping: Click the Delete icon to delete the field mapping for the corresponding row. You cannot delete the mapping for the operator's fixed output fields.

4. After you complete the configuration, click Save in the top menu bar of the canvas.

   After you configure the operator, you can select upstream or downstream operators to complete the workflow.