Deploy a Spark cluster-Resource Orchestration Service(ROS)-阿里云帮助中心

You can use Resource Orchestration Service (ROS) to deploy a Spark cluster by creating a stack.

Background

Apache Spark is a general-purpose computing engine designed for large-scale data processing. It uses Scala as its application framework and supports in-memory distributed datasets, which allows for interactive queries and iterative workload optimization.

The Spark Cluster Edition - Existing VPC template creates multiple ECS instances based on your existing resources such as a VPC, vSwitches, and security groups. One ECS instance is assigned an EIP to serve as the management node, while Auto Scaling manages the other ECS instances. The template uses the following software versions:

Java Development Kit (JDK): 1.8.0
Hadoop (distributed system infrastructure): 2.7.7
Scala (programming language): 2.12.1
Spark (computing engine): 2.1.0

After the stack is created, you can obtain the SparkWebSiteURL to log on to the Spark management console. To access the SparkWebSiteURL from the internet, you must add an inbound rule to the security group to allow access on port 8080. For more information, see Add a security group rule.

Step 1: Create a stack

Log on to the ROS console.
In the left-side navigation pane, click Public Templates.
Find the Spark Cluster Edition (Existing VPC) template.
Click Create Stack.

On the Configure Parameters page, enter a Stack Name and configure the following parameters.

Parameter	Description	Example
Existing VPC Instance ID	The ID of the VPC. For information about how to create and query VPCs, see Create and manage a VPC.	vpc-bp1m6fww66xbntjyc****
VSwitch Zone	The zone where the vSwitch resides.	China (Hangzhou) Zone K
VSwitch ID	The ID of the vSwitch. For information about how to create and query vSwitches, see Create and manage vSwitches.	vsw-bp183p93qs667muql****
Business security group ID	The ID of the ECS security group. For information about how to query a security group ID, see View security groups.	sg-bp15ed6xe1yxeycg7o****
Instance specification	The instance type of the ECS instances. Select a valid instance type. For more information, see Instance families.	ecs.c5.large
Instance password	The password of the ECS instances.	Test_12****
Disk type	Valid values: `cloud_efficiency`: Ultra Cloud Disk. `cloud_ssd`: Standard SSD. For more information, see Block storage devices.	cloud_efficiency
System disk size	The size of the system disk. Valid values: 20 to 500. Unit: GB.	40
Number of instances	The number of instances in the Spark cluster. Valid values: 3 to 10.	3

Click Next:Check and Confirm. Then, Click Create.
On the Stack Information tab, view the status of the stack. After the stack is created, click the Output tab to obtain the SparkWebSiteURL.
Access the SparkWebSiteURL to log on to the Spark management console.

Step 2: View the resources

In the navigation pane on the left, click Stacks.
On the Stacks page, click the name of the target stack.

Click the Resources tab to view the resource information.

The following table describes the resources created in this example.

Resource	Quantity	Description	Specifications
ALIYUN::ECS::Instance	1	Creates one ECS instance to install the Spark Master service.	Total instances: 1 Instance type: ecs.c5.large Disk type: Ultra Cloud Disk System disk size: 40 GB Public IP address assigned: Yes
ALIYUN::ESS::ScalingGroup	2	Creates two Auto Scaling groups for the Spark Slave service. Auto Scaling automatically adjusts the number of ECS instances based on your scaling rules to meet changes in business demand.	Total instances: 2 Instance type: ecs.c5.large Disk type: Ultra Cloud Disk System disk size: 40 GB Public IP address assigned: Yes
ALIYUN::RAM::Role	1	Creates a RAM role to issue a short-lived Security Token Service (STS) token to grant access permissions more securely.	None
ALIYUN::VPC::EIP	1	Creates an EIP and associates it with an ECS instance to provide the instance with internet access.	None
ALIYUN::OOS::Template	2	Creates two CloudOps Orchestration Service (OOS) templates to create lifecycle hooks. For more information about lifecycle hooks, see Lifecycle hooks.	None

Note

For pricing details, see the official pricing page or the pricing documentation for each product.