Manage scheduling clusters

Background information

In Dataphin, each tenant has a default scheduling cluster, which is the Dataphin cluster. You can also register custom clusters for task scheduling under a Metastore tenant and specify which tenants can use them. This feature effectively addresses challenges of transmitting data over a public network, such as low security, high bandwidth costs, and low transfer efficiency.

For example, assume Dataphin is deployed in an on-premises data center, and you need to integrate data from a cloud database A to another cloud database B located in the same region. First, create a Kubernetes (K8s) cluster on the cloud by using a container management service and specify the machines for Dataphin tasks. Then, register this cluster in Dataphin and create a corresponding custom resource group. When you create an integration task, you can assign it to the custom resource group on the cluster. This ensures that data transfer occurs within the same region without data exchange with the Dataphin cluster.

Limitations

• Currently, the scheduling cluster management feature is available only for deployments based on the latest architecture. For more information, contact the product O&M team.

• You can register a maximum of 5 clusters in addition to the default cluster (Dataphin cluster).

Permissions

Only global roles under a Metastore tenant with the Manage Resource Configuration permission can manage scheduling clusters. Global roles with the View System Settings permission can view system settings.

Manage scheduling clusters

1. In the top menu bar of the Dataphin homepage, choose Management Center > System Settings.

2. In the left-side navigation pane, choose Tenant Settings > Resource settings.

3. On the Resource settings page, click the Scheduling cluster management tab. By default, the scheduling cluster list displays the clusters available to the current tenant, which includes the default cluster.

   The scheduling cluster list displays basic information about the clusters, including Scheduling cluster name/ID, Owner, Total resources, Status, Description, Associated tenants, Last Update, and the available management operations.

   • Total resources: The total resources of the cluster, not the amount available to the current tenant.

   • Status: The status of a scheduling cluster can be Waiting for resource report, Waiting timeout, Normal, or Abnormal. For more information, see Scheduling cluster status.

4. (Optional) You can filter scheduling clusters by entering a cluster name, or by selecting an owner or status.

5. Manage a scheduling cluster from the list.

   Note

   The default scheduling cluster is a system cluster and is view-only.

   Actions	Description
   Edit	Click the icon in the Actions column of the target scheduling cluster. In the Edit scheduling cluster basic information dialog box, modify the Cluster basic information, Metric collection configuration, and Real-time task O&M information of the cluster. For parameter descriptions, see Edit a registered scheduling cluster.
   Cluster Connection Configuration Guide	Click the icon in the Actions column of the target scheduling cluster. In the Cluster connection configuration guide dialog box, view instructions on how to configure the connection and authorization for a custom cluster. Only successfully connected clusters can be used to create custom resource groups. For more information, see Configure a cluster connection.
   Delete	Click the icon in the Actions column to delete a scheduling cluster that has no associated custom resource groups. Important After a cluster is deleted, the agent application deployed on it stops running and cannot be restored. We recommend that you contact the cluster owner to delete the corresponding container (Pod). Use the sh uninstall.sh command to delete the container.

Scheduling cluster status

The status of a scheduling cluster can be Waiting for resource report, Waiting timeout, Normal, or Abnormal. The following table describes these statuses.