Create a Flink compute source

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Plan > Compute Source.

2. On the Compute Source page, click Add Compute Source and select Flink Compute Source.

3. On the Create Compute Source page, configure the parameters.

   1. Basic information

      Parameter	Description
      Compute type	Select Flink.
      Compute source name	Enter a name for the compute source. The name must meet the following requirements: Can contain Chinese characters, letters, digits, underscores (_), and hyphens (-). Cannot exceed 64 characters.
      Compute source description	Enter a description for the compute source, up to 128 characters.

   2. Deployment mode

      Dataphin supports the Yarn and Kubernetes deployment modes. The required parameters differ for each mode.

      Yarn

      • Cluster basic information

        Parameter	Description
        Configuration file	Upload the configuration files for the cluster. The yarn-site.xml, core-site.xml, and hdfs-site.xml files are mandatory.
        Cluster Kerberos	Kerberos is an identity authentication protocol based on symmetric key cryptography. It provides authentication for other services and supports single sign-on (SSO), which allows an authenticated client to access multiple services such as HBase and HDFS. If your cluster uses Kerberos authentication, enable Cluster Kerberos and either upload a Krb5 authentication file or configure the KDC server address. Krb5 authentication file: Upload the krb5.conf file for Kerberos authentication. KDC server address: The address of the Key Distribution Center (KDC) server, which assists with Kerberos authentication. To configure multiple addresses, separate them with commas (,).
        Cluster type (Optional)	Select a cluster type for connection testing. Options include Aliyun E-MapReduce 5.x, CDH 5.x Hadoop, CDH 6.x Hadoop, Cloudera Data Platform 7.x, AsiaInfo DP5.3 Hadoop, and Transwarp TDH 6.x Hadoop. Important The connection test usually passes without a selected cluster type, but it can fail in some situations. Therefore, we recommend selecting one.

      • Flink computing resource

        Parameter	Description
        Computing resource type	You can select resource queue and session cluster.
        Resource queue	When you select resource queue for computing resource type, enter the name of the resource queue. This specifies the Yarn queue for submitting Flink tasks. The name must meet the following requirements: Length limit: The queue name cannot exceed 256 characters. Character limit: The queue name can contain only letters, digits, spaces, and the following special characters: -_.@'(). Case-sensitive: The queue name is case-sensitive. Uniqueness: The queue name must be unique within the compute source. To configure multiple resource queues, click + Add. Note You can add a maximum of 10 resource queues. To delete an unnecessary resource queue, click the icon. You must keep at least one resource queue. If you delete a queue, tasks that are configured to use it can no longer be submitted.
        Session cluster	When you select session cluster for computing resource type, you can select one or more session clusters. The drop-down list includes all clusters created on the Session Cluster page, regardless of their status. If you select only session cluster as the computing resource type, the system automatically obtains the necessary Kerberos authentication information from the selected session cluster. You do not need to configure this information in the compute source.

      • Flink Kerberos authentication

        Note

        You can configure Flink Kerberos authentication only if the computing resource type includes resource queue.

        • Flink Kerberos: If your Flink cluster uses Kerberos authentication, enable Flink Kerberos, upload a keytab file, and configure the principal.

          • Keytab file: Upload the keytab file. You can obtain this file from your Flink server.

          • Principal: Enter the Kerberos principal that corresponds to the Flink keytab file.

        • Username: If Flink Kerberos is disabled, you must enter the username for submitting Flink tasks to the cluster.

      • Checkpoint storage

        File system: Supported file systems include HDFS, OSS-HDFS, and Aliyun OSS (only for Flink 1.14 and 1.15). The required parameters vary by file system.

        Note

        Only the Aliyun E-MapReduce 5.x Hadoop compute engine supports the OSS-HDFS file system.

        • HDFS file system:

          Directory path: Enter the directory path on the cluster for checkpoint storage. Ensure that Flink has permission to access this path. Example: hdfs://cdh-cluster-00001:8020/openflink/savepoint/. If your HDFS cluster is configured for high availability (HA), you can use an HA path, such as hdfs://nameservice/path.

        • OSS-HDFS file system:

          • Directory path: Enter the directory path on the cluster for checkpoint storage. Ensure that Flink has permission to access this path. Example: hdfs://cdh-cluster-00001:8020/openflink/savepoint/. If your HDFS cluster is configured for high availability (HA), you can use an HA path, such as hdfs://nameservice/path.

          • AccessKey ID and AccessKey Secret: Enter the AccessKey ID and AccessKey Secret for accessing OSS on the cluster. You can use an existing AccessKey or create a new one. For more information, see Create an AccessKey.

            Note

            For security, the AccessKey Secret is displayed only during creation and cannot be retrieved later. Be sure to store it in a safe place.

        • Aliyun OSS file system:

          • Endpoint: Enter the connection address for the OSS service.

          • Directory path: Enter the path in the format oss://{Bucket}/{Object}.

          • AccessKey ID and AccessKey Secret: Enter the AccessKey ID and AccessKey Secret for accessing OSS on the cluster. You can use an existing AccessKey or create a new one. For more information, see Create an AccessKey.

            Note

            For security, the AccessKey Secret is displayed only during creation and cannot be retrieved later. Be sure to store it in a safe place.

        Important

        The settings you configure here take precedence over the AccessKey configured in the core-site.xml file.

      Kubernetes

      • Cluster basic information

        No basic cluster information is required for the Kubernetes deployment mode.

      • Flink engine configuration

        In Kubernetes deployment mode, the file system for the Flink compute engine can be NFS, S3, or Azure Blob Storage. The required parameters vary by file system.

        NFS

        Parameter	Description
        Server	Enter the domain name of the NFS server.
        Version	Select the NFS version. NFSv3 and NFSv4 are supported.
        Directory	Enter the directory path for checkpoint storage on NFS. For example, /data/checkpoint.
        Maximum capacity	Enter the maximum storage capacity supported by NFS. Checkpointing will fail if this limit is exceeded. Unit: Gi.

        S3

        Parameter	Description
        Endpoint (Optional)	Enter a valid address. Example: http://s3.us-east-2.amazonaws.com. Note If you are using Amazon S3, you do not need to enter an endpoint. This field is required for other S3-compatible storage.
        Directory path	Enter the storage path in the format s3://{YOUR-BUCKET}/{path}. We recommend that you use a dedicated directory for checkpoint storage and clean it regularly.
        Authentication method	Specifies the method for authenticating with S3. The default is AccessKey authentication. The IAM role authentication option is available only if it has been enabled for your tenant. AccessKey authentication: Authenticates with S3 using the provided access key and secret key. IAM role proxy: Authenticates with S3 using the IAM role attached to the Amazon EC2 instance where Dataphin is running. The system automatically retrieves the IAM role of the current instance. Ensure that the role has permission to access the target bucket.
        Access key and Secret key	Enter the access key and secret key for accessing Amazon S3. Click the icon to view the plaintext. This is required only when the authentication method is AccessKey authentication.

        Azure Blob Storage

        Parameter	Description
        Protocol	Currently, only ABFS is supported.
        Authentication method	Currently, only Shared key is supported.
        Directory path	Enter the storage path in the format abfs://{YOUR-CONTAINER}@${YOUR-AZURE-ACCOUNT}.dfs.core.windows.net/{object-path}.
        Access key	Enter the access key for the Azure Blob Storage account. Click the icon to view the plaintext.

4. Click Test Connection to verify connectivity between Dataphin and the cluster.

   The Kubernetes deployment mode does not support connection testing. You can click Submit directly.

5. After the connection test succeeds, click Submit.