DocsICP FilingConsole
Documentation
Home Page Dataphin Dataphin(Non-shared) Support FAQ Resource scheduling

Resource scheduling

Updated at:
Copy as MD
Product
My Favorites

Troubleshoot common resource scheduling issues in Dataphin.

1. The resource scheduling dashboard shows no data or the display time is not updated

Verify that the Prometheus component is installed and that the connection to Prometheus is working.

If Prometheus is not installed, contact the Dataphin deployment team. If it is installed but the displayed time is outdated, the connection to Prometheus may be abnormal. Contact the Dataphin O&M team for troubleshooting, or check the connection yourself by following these steps:

  1. Log on to the Dataphin cluster.

  2. Run the kubectl get pods -n dataphin | grep rs command to find the rs pod.

  3. Run the kubectl describe pod <pod_name> -n dataphin command to find the primary rs pod.

    Check if the status of the dataphin-rs-scheduler-rpc-service label is active. If the status is active, it is the primary pod. If the status is standby, it is the secondary pod.

  4. Run the kubectl exec -it <pod_name> -n dataphin -- bash command to log on to the primary rs pod.

  5. Run the cd /home/admin/logs/dataphin-rs command to go to the rs log directory.

  6. Check the log for messages about an abnormal Prometheus connection. You can search by running the grep "failed to connect to" <log_file> command. If the log contains this information, the connection is abnormal.image

2. A node is executed again after it has already run successfully

Check if the node has an upstream dependency. If it does, a rerun of the upstream node triggers a rerun of the downstream node.

3. The Resource Settings page is not displayed

You must import the license to view the Management Center > System Settings > Resource Settings page.

4. A "resource waiting" message appears when running a node

Check for the following conditions in order:

  • Many nodes are running in the current resource group.

  • Many large, long-running nodes are continuously occupying the resource group's resources.

  • The current resource group is not configured properly. You can go to Management Center > System Settings > Resource Settings to configure the resource group.

  • The resource configuration for the current tenant is not appropriate. When the resource value of the current tenant changes, the resource groups for that tenant are adjusted accordingly.

For example, if the runtime log ends with a message similar to Sending to agent e405****-****-****-****-****2fc2, the node was successfully sent to the scheduling cluster. However, a problem may have occurred when the scheduling cluster tried to create the node pod. Possible reasons include the following:

  • The scheduling cluster has insufficient remaining resources to create a new node pod.

  • The scheduling cluster may have a fault. Common issues include network plugin or clock exceptions.

4.1. Suggestions for insufficient tenant resources

  • Stop low-priority nodes in the resource group to release resources.

  • Increase the allocation ratio for the tenant's resource group.

  • Use off-peak scheduling to prevent nodes from competing for resources during peak hours.

  • Optimize the resource configuration of nodes based on resource utilization. You can view node resource utilization on the resource dashboard.

4.2. Suggestions for insufficient cluster resources

  • Stop low-priority nodes in the resource group to release resources.

  • Use off-peak scheduling to prevent nodes from competing for resources during peak hours.

  • Optimize the resource configuration of nodes based on resource utilization. You can view node resource utilization on the resource dashboard.

  • Scale out the cluster resources.

5. A "resource waiting" message appears when running a shared node, but resources are sufficient

If resources are sufficient but a "resource waiting" message still appears, the shared node concurrency limit may be the cause. When the number of concurrent shared nodes exceeds the configured limit, new nodes are queued.

The concurrency limit varies for different types of shared nodes.

  • SQL type default: By default, one shared container pod is created, which can run 200 nodes concurrently.

  • Python and Shell type default: By default, one shared container pod is created, which can run 15 nodes concurrently.

You can adjust the number of shared container pods and the concurrency per pod as needed.

6. Do the resource-consuming instances in runtime diagnostics belong to the current resource group or the current tenant?

The resource-consuming instances in runtime diagnostics belong to the current tenant.

7. Automatic rerun is not triggered after a node times out, even though the number of automatic reruns on failure is set

By default, a node timeout does not trigger an automatic rerun. To enable automatic reruns for timed-out nodes, go to O&M > Global Configurations > Runtime Configurations > Rerun Configurations and turn on the corresponding switch.image

8. Offline task runtime resources

For more information about offline task runtime resources, see Node runtime resources.

For example, to calculate the resources required for a Python shared node type with 20 pods, each processing 50 nodes:

The resources consumed by one pod consist of base resources and node resources. The base resources are 0.5 cores and 4,096 MB (4 GB). The default Python node resources are 0.1 cores and 256 MB. Therefore, the total resources required for one pod are 0.6 cores and 4,352 MB. If you deploy 20 pods, the required resources are 20 × 0.6 cores = 12 cores, and 20 × 4,352 MB = 87,040 MB (85 GB).

9. A "503 Service Temporarily Unavailable" error occurs when querying the instance list on the O&M page

First, check whether the request was blocked by a Web Application Firewall (WAF). A WAF detects and blocks malicious traffic such as SQL injections and cross-site scripting (XSS) attacks.

10. Timestamps in the runtime log appear out of order

image

The timestamps are not actually out of order. The runtime log combines logs from three stages, and the middle section contains logs periodically polled from the server. Logs from different stages are not sorted together by time.

11. Is there Python documentation for calling OpenAPI from a trigger-based node developed in Python?

Currently, only some APIs support Python development, including public cloud APIs and O&M-related APIs.

References:

12. A commented-out resource annotation in the code is still counted in the recommended optimization node

Commented-out resource annotations still take effect. To exclude the resource from being counted, delete the annotation entirely.

13. After an upgrade, a previously working OpenAPI call fails with a "tenantId cannot be empty" error

This error occurs if you upgrade from a version of Dataphin earlier than V3.0. Starting from Dataphin V3.x, the tenantId parameter is required.

14. A node is shown as paused on the auto triggered task page in O&M, even though it is configured for normal scheduling

Check whether the periodic scheduling for the node's project is set to Disabled, Skip Execution. If so, change it to Enabled, Auto Schedule.image

Previous：Data developmentNext：Approval management