Spot instances

Prerequisites

An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.

Spot instances

Spot instances are idle Spark executor resources from the AnalyticDB for MySQL resource pool that you can purchase at a significant discount.

Spot instance resources differ from ACU elastic resources in the following ways:

• Spot instance resources cost about 30% less than ACU elastic resources.

• When you submit a Spark job to a job resource group with spot instances enabled, the cluster first attempts to acquire idle, lower-priced spot instance resources. If the attempt is successful, the Spark job runs on spot instance resources and is billed at the spot rate. If the attempt fails, the Spark job uses ACU elastic resources and is billed at the standard rate for ACU elastic resources.

• By default, Alibaba Cloud provides a one-hour protection period for AnalyticDB for MySQL spot instances. This ensures that spot instance resources are not preempted within the first hour. After the protection period expires, the resources may be reclaimed, and the probability of preemption increases as the resource usage time increases.

• Spark jobs that run on spot instance resources have a lower priority. If other clusters submit Spark jobs that do not use spot instances, they may preempt your spot instance resources. This preemption releases the resources, which can cause these jobs to fail. The probability of job failure due to spot instance preemption is about 1% to 5%.

Use cases and recommendations

We recommend enabling spot instances for the following scenarios:

• Spark job development.

• Cost-sensitive workloads.

• Workloads that can tolerate Spark job failures.

• Workloads where most Spark jobs complete in less than one hour.

For mixed workloads that include both short-running and long-running jobs, we recommend one of the following strategies:

• Create two job resource groups. Enable spot instances on one for short-running jobs, and keep it disabled on the other for long-running jobs.

• Create one job resource group and enable spot instances. When you run a Spark job, you can configure the spark.adb.spotInstance.disabled parameter to prevent the Spark job from using spot instances.

Limitations

• Spot instances are available only in the following regions: China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hangzhou), China (Shanghai), China (Shenzhen), China (Guangzhou), China (Hong Kong), Japan (Tokyo), Singapore, US (Silicon Valley), US (Virginia), Germany (Frankfurt), and UK (London).

• Spot instances can be enabled only for job resource groups. Enabling this feature does not guarantee that all resources or all Spark jobs in the resource group will use spot instances. It only means that Spark jobs will attempt to use them.

• Only Spark jobs can use spot instances.

• Only specific Spark Executor resource specifications support spot instances: large and 2xlarge.

Billing

Spot instance resources are priced at an approximately 30% discount compared to ACU elastic resources. For more information about pricing, see Pricing of Enterprise Edition and Basic Edition and Pricing of Data Lakehouse Edition.

Enable spot instances

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the navigation pane on the left, choose Cluster Management>Resource Management, and click the Resource Groups tab.

   Existing resource group

   1. Find the target job resource group and click Actions in the Modify column.

   2. In the Modify Resource Group panel that appears, turn on the Preemptible instance switch.

   3. Click OK.

   4. In the Preemptible instance column for the resource group, check whether the feature is enabled.

      After you enable this feature, all Spark jobs that run in the job resource group will attempt to use spot instances.

   New resource group

   1. In the upper-right corner of the resource group list, click Create Resource Group and configure the parameters.

      Parameter	Description
      Resource Group Name	A custom name for the resource group. The name must be 2 to 30 characters long, start with a letter, and contain only letters, digits, and underscores (_).
      Job Type	Select Job from the drop-down list. job resource groups are designed for high-throughput batch processing and scale compute resources up and down on demand.
      Minimum Computing Resources	The minimum value is 0 ACU.
      Maximum Computing Resources	The maximum number of compute resources that you can set in the console is 1,024 ACUs, in increments of 8 ACUs. If you need a higher limit, submit a ticket.
      Spot Instance	Turn on the Spot Instance switch. After you enable spot instances, Spark jobs that run in the job resource group attempt to use spot instance resources.

   2. Click OK to create the resource group.

   3. After the resource group is created, check the Spot Instance column to confirm that the feature is enabled.

      After you enable this feature, all Spark jobs that run in the job resource group will attempt to use spot instances.

Disable for a specific job (Optional)

If you do not want a specific Spark job to use spot instances, set the following parameter in the CONF section of the Spark job:

After the job is complete, you can check the Spot Instance Resource Usage metric to confirm that the configuration has taken effect. If the value of Spot Instance Resource Usage is 0, the submitted Spark job did not use spot instance resources.

View spot instance resource usage

After a Spark job completes, go to the Monitoring page for the job resource group to view spot instance resource usage.

Disable spot instances

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the navigation pane on the left, choose Cluster Management>Resource Management, and click the Resource Groups tab.

3. Find the target job resource group and click Actions in the Modify column.

4. In the Modify Resource Group panel that appears, turn off the Spot Instance switch.

5. Click OK.

6. In the Spot Instance column for the resource group, check whether the feature is disabled.

Related APIs