DocsICP FilingConsole
Documentation
Home Page Dataphin Dataphin(Non-shared) User Guide Data Integration Offline integration Widget library development Input widget Configure the Hive input component

Configure the Hive input component

Updated at:
Copy as MD
Product
My Favorites

The Hive input component reads data from a Hive data source. To synchronize data from a Hive data source, configure the Hive input component as the source and then configure the target data source. This topic describes how to configure the Hive input component.

Limitations

The Hive input component supports Hive data tables in the following formats: orc, parquet, text, rc, seq, Iceberg, and Paimon. The Iceberg format is supported only for Hive computing engines or data sources on E-MapReduce 5.x. The Paimon format is supported on the same environments and also on Amazon EMR. Integration with ORC transactional tables and Kudu tables is not supported.

Note

To integrate data from a Kudu table, use the Impala input component. For more information, see Configure the Impala input component.

Prerequisites

  • A Hive data source has been created. For more information, see Create a Hive data source.

  • Your account must have synchronization read permissions on the data source to configure the Hive input component. If you do not have the required permissions, you must apply for them. For more information, see Apply for data source permissions.

Procedure

  1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

  2. In the top navigation bar of the integration page, select a Project. In Dev-Prod mode, you must also select an environment.

  3. In the left-side navigation pane, click Batch Integration. In the Batch Integration list, click the batch pipeline that you want to develop to open its configuration page.

  4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

  5. In the left-side navigation pane of the Component Library panel, select Input. Find the Hive component in the list on the right and drag it to the canvas.

  6. On the Hive input component card, click the image icon to open the Hive input configuration dialog box.

  7. In the Hive input configuration dialog box, configure the parameters.

    Parameter

    Description

    Step name

    The name of the Hive input component. Dataphin automatically generates a step name, which you can change as needed. The name must meet the following rules:

    • Can contain only Chinese characters, letters, underscores (_), and digits.

    • Cannot exceed 64 characters in length.

    Data source

    In the data source drop-down list, select a Hive data source. The list displays all Hive data sources, regardless of whether you have synchronization read permissions. You can click the image icon to copy the name of the current data source.

    • For a data source for which you do not have synchronization read permissions, click Apply next to the data source to apply for the permissions. For more information, see Apply for data source permissions.

    • If you do not have a Hive data source, click Create Data Source to create one. For more information, see Create a Hive data source.

    Table

    Select the source table for data synchronization. You can click the image icon to copy the name of the selected table.

    Note

    If the selected table is a Hudi table or Paimon table, you can configure only the partition parameter.

    Partition

    Supports reading static partitions or range partitions, with static partitions such as ds=20230101 or ds1=2023,ds2=01, and range partitions such as /*query*/ds >=20230101 and ds <= 20230107.

    Note

    If the selected table is a Hudi table or Paimon table, reading data from a range partition is not supported.

    If partition does not exist

    Select a policy for when the specified partition does not exist:

    • Fail the task: The task stops and is marked as failed.

    • Succeed with no data written: The task succeeds, but no data is written to the target table.

    File encoding

    Select the encoding of the source files stored in Hive. Supported file encodings include UTF-8 and GBK.

    String to replace with NULL

    Only source tables that use the textfile data storage format are supported. Enter the string to be replaced with NULL. For example, if you enter \N, the system replaces the string \N with NULL.

    Compression format

    This parameter is optional. If the files are compressed, select the corresponding compression format. Dataphin uses this format to decompress the files. For ORC tables, zlib is selected by default. If you need a different decompression format, you must specify it. No default format is selected for other table formats. The system supports the following compression formats: zlib, hadoop-snappy, lz4, and none.

    Field delimiter

    A field delimiter is usually specified when a table is created, for example, by using the ROW FORMAT DELIMITED FIELDS TERMINATED BY statement. Enter the field delimiter for reading the table. If you do not specify a delimiter, Dataphin uses \u0001 by default.

    Output fields

    This section lists all fields from the selected table that match your filter criteria. If you do not want to output certain fields to downstream components, you can remove them.

    Note

    When using the Hadoop computing engine, you can view the data classification and sensitivity level of the output fields. This feature is not supported for other computing engines.

    • To remove a single field: Click the sgaga icon in the Actions column for that field.

    • To remove multiple fields: Click Field management. In the Field management dialog box, select the fields that you want to remove, click the image left-arrow icon to move them to the unselected list, and then click OK.

      image..png

  8. Click OK to save the configuration of the Hive input component.

Previous：Configure HDFS Input ComponentNext：Configure HBase input component