Plan resources and select a resource package

An optimal resource planning solution helps you achieve the following objectives:

• Storage resources

  Storage resources are sufficient to store all inventory data and accommodate the incremental data that is generated during a specific period of time in the future.

• Computing resources

  • Sufficient resources are available to meet the demands of all computing jobs without being wasted.

  • Jobs with different priorities do not interfere with each other, and high-priority jobs are guaranteed to obtain sufficient computing resources.

  • Even when some jobs process large data volumes and consume many computing resources, other jobs can still obtain computing resources and run without being blocked.

  • Varying resource requirements at different times are met. Resources are allocated by time period based on the resource usage of different projects to improve resource utilization.

To plan computing resources based on the computing-to-storage ratio, follow these steps:

1. Estimate the storage capacity.

   Storage capacity = Current storage volume + (Estimated data volume increase per month × Number of months)

   You can obtain the current storage volume after your project is running. After the project is running for two or three months, you can estimate the data volume increase per month by dividing the total increased volume by the number of months. If more services need to be processed on Data Mid-end in the future or data volume increases every month, you can increase the estimated value to reserve sufficient resources.

2. Estimate the computing-to-storage ratio.

   Estimate the computing-to-storage ratio based on the data statistics in the project development and testing phases and the data statistics after a project is running for one or two months.

   The computing-to-storage ratio is typically planned to be between 2 and 10. You can purchase computing resources based on the planned ratio and the estimated storage capacity:

   • If the project runs many batch processing jobs daily and the SQL programs have high computational complexity, you can plan for a computing-to-storage ratio of 10.

   • If the project runs few batch processing jobs daily and the SQL programs have low computational complexity, you can plan for a computing-to-storage ratio of 2.

   • If the project runs a moderate number of batch processing jobs daily and the SQL programs have medium computational complexity, you can plan for a ratio between 2 and 10.

3. Estimate the amount of computing resources.

   You can estimate the amount of computing resources based on the estimated storage capacity and computing-to-storage ratio. The formula is: Amount of computing resources = Estimated storage capacity × Estimated computing-to-storage ratio.

   Note

   When you estimate the computing resources a project will consume based on the computing-to-storage ratio, both the storage capacity and the ratio are estimates and may not be precise. Therefore, this method requires a technical lead with extensive project implementation experience to ensure that each estimation step is as accurate as possible.

To plan computing resources based on the actual computing resource consumption of a project, follow these steps:

1. Check the project's CPU consumption.

   Information Schema provides the TASKS_HISTORY view, which records the computing resources consumed by all computing jobs in a MaxCompute project. You can query the cost_cpu field in the TASKS_HISTORY view using the MaxCompute client. You can use this field to calculate the amount of consumed resources. The following command is an example of how to query the TASKS_HISTORY view:

   select * from information_schema.tasks_history where ...;

   cost_cpu indicates the CPU utilization of a computing job in the project. For example, the value of cost_cpu is 5000, which is calculated using the following formula: 10 × 100 × 5. 10 indicates the number of CPU cores that are used. 5 indicates the duration (in seconds) in which the job runs.

2. Plan the amount of computing resources.

   1. Calculate the total sum of the daily average cost_cpu values for all jobs across all projects under your Alibaba Cloud account.

      For example, an average of 1,000 jobs run per day in a MaxCompute project, and the cost_cpu values for these jobs are W1 to W1000. Calculate the total sum of W1 to W1000 to obtain the total sum of cost_cpu of this project. The total sum is Wz in this example.

      Note

      Typically, a data mid-end is divided into the following six MaxCompute projects:

      • ods_dev: development and testing project for the operational data store (ODS) layer

      • ods_prod: production project for the ODS layer

      • cdm_dev: development and testing project for the common data model (CDM) layer

      • cdm_prod: production project for the CDM layer

      • ads_dev: development and testing project for the application layer

      • ads_prod: production project for the application layer

      In this case, you need to calculate the total sum of cost_cpu of all computing jobs that run in these six projects.

   2. Convert the calculated cost_cpu sum.

      Because the cost_cpu value is measured in seconds, it is too granular for project evaluation. Typically, you can divide the cost_cpu value by 100 and then by 3,600 to obtain the value in core×h (CPU cores consumed × hours). This makes it easier to estimate the minimum amount of computing resources required to run all jobs within a specified time frame. Assume that this value is W.

   3. Confirm the daily time window in which all jobs must be completed.

      For example, you must complete all batch processing jobs between 00:00:00 and 06:00:00. This means that the total running duration of all jobs in a day is 6 hours. For example, all jobs must be completed in N hours.

   4. Estimate the minimum amount of computing resources to purchase.

      Use the W and N values from the previous two steps to calculate W (CPU core consumption × hours) / N (job runtime in hours). The result is the minimum amount of computing resources that you need to purchase for the MaxCompute project.

      Before you estimate the minimum amount, make sure that the value of cost_cpu does not fluctuate for all batch processing jobs, all jobs are running in N hours, and no resources are wasted. However, the total running duration of jobs may be longer than N hours in some scenarios. For example, if the amount of data that you need to calculate increases, the value of W increases. In addition, latency may occur when you use DataWorks or Dataphin to run scheduled jobs or when jobs attempt to obtain computing resources. This latency increases the intervals between jobs. In this case, the actual period in which all jobs are running is less than N. As above mentioned, the value of W increases, and the value of N decreases in practical application. You need to add more computing resources to run jobs faster. Therefore, we recommend that you double the value of W/N.