Task runtime resources

Task runtime resources refer to the schedule resources, such as CPU and memory, that a task consumes when it runs in Dataphin. This topic describes how to customize these resources to meet your specific needs.

Task resource classifications

1. Dataphin tasks are divided into two main categories based on their runtime mode:

   • Shared schedule resource tasks: All tasks that use shared schedule resources run in the same container. Examples include SQL tasks, logical table tasks, and tasks created from computing templates that have the Shared Resource configuration enabled. For more information, see Create an offline computing template.

   • Exclusive schedule resource tasks: Each time an exclusive schedule resource task is launched, a new container is started to run it. Examples include Shell, Python, MapReduce, and integration tasks.

2. Real-time integration and real-time computing tasks do not consume Dataphin schedule cluster resources during runtime.

Runtime resource consumption

The resource consumption of a task during runtime consists of two parts: the container base and the task resources.

• Container base: The resources for the container base are determined by the system configuration and cannot be modified. For more information, see Container base.

• Task resources: You can customize task resources. Dataphin provides the following two configuration methods.

  • You can configure task resources by adding an annotation statement. For more information, see Task resource configuration.

  • You can configure runtime resources for a task using the custom resource group feature in Dataphin. For more information, see Overview of custom schedule resource groups.

    Important

    Currently, only customers with deployments based on the latest architecture can use the custom resource group feature. For more information, contact the product O&M team.

Container base

The resource consumption details for the container base are as follows:

Default task resources

The following table shows the default resource configuration and customization support for each task type:

Task resource configuration

The configurable range for CPU is from 0.001 CPU to 4.0 CPU, and for memory is from 32 MB to 16 GB. To customize the task resource size, add the following annotation statement at the beginning of the task code:

@required_resource{required_memory=memory_size;required_cpus=cpu_size}

Example:

#!/bin/bash
@required_resource{required_memory=2GB;required_cpus=1.0}
@resource_reference{"test-boot.jar"}

java -jar test-boot.jar -Xms 1024m -Xmx 1024m