Access cloud resources with an ECS application role

Prerequisites

Create an OSS bucket in the OSS console. The bucket must be in the same region as your E-MapReduce cluster. For more information, see Create buckets.

Procedure

1. Step 1: Create a policy

2. Step 2: Create a RAM role

3. Step 3: Create a cluster and access external resources

Step 1: Create a policy

1. Go to the custom policy creation page.

   1. Log on to the RAM console with your Alibaba Cloud account.

   2. In the left-side navigation pane, choose Permission Settings > Policies.

   3. On the Policies page, click Create Policy.

2. On the Create Policy page, click the JSON tab.

3. Enter the policy content and click OK.

   Add the following policy.

   {
       "Version": "1",
       "Statement": [
           {
               "Action": [
                   "oss:GetObject",
                   "oss:ListObjects"
               ],
               "Resource": [
                   "acs:oss:*:*:emr-logs2",
                   "acs:oss:*:*:emr-logs2/*"
               ],
               "Effect": "Allow"
           }
       ]
   }

   Note

   The policy includes the following elements:

   • Action: the operation on a resource. In this example, permissions to read objects and list directories in OSS.

   • Resource: the authorized object. In this example, the OSS bucket named emr-logs2 and all objects within it.

   For more information about the basic elements of a policy, see policy elements.

4. Enter a Name for the policy.

   This example uses test-emr.

5. Click OK.

Step 2: Create a RAM role

1. In the left-side navigation pane of the RAM console, choose Identities > Role.

2. On the Role page, click Create Role.

3. Create the RAM role.

   1. Select Alibaba Cloud Service.

   2. Click Next.

   3. In the Configure Role panel, configure the following parameters.

      Parameter	Description
      Role Name	This example uses test-emr.
      Select Trusted Service	Select Elastic Compute Service.

   4. Click Complete.

4. Optional: Modify the trusted service.

   Important

   You can skip this step if your cluster runs EMR 3.32 or later, EMR 4.5 or later, or EMR 5.x or later.

   1. On the Role page, click the Role Name that you just created.

   2. Click the trust policy tab.

   3. Click Edit trust policy.

   4. Change ecs.aliyuncs.com to emr.aliyuncs.com.

      

   5. Click Save trust policy.

5. Attach the custom policy.

   1. On the Role page, click Add Authorization in the Actions column for the Role Name that you just created.

   2. On the Add Authorization page, select Custom Policies and attach the policy you created.

   3. Click OK.

   4. Click Close.

Step 3: Create a cluster and access external resources

1. Log on to the EMR on ECS console.

2. In the top navigation bar, select a region and a resource group as needed.

3. Click CREATE_CLUSTER. On the Basic Configuration page, in the Advanced Settings section, add the name of the RAM role you created in Step 2: Create a RAM role. For more information, see Create a cluster.

   

4. After the cluster is created, log on to the cluster via SSH. For more information, see Log on to a cluster.

   Run the following command to verify authorization.

   hdfs dfs -ls oss://<yourBucketName>/

   Note

   Replace <yourBucketName> with the name of your OSS bucket.

   • If the role does not have the required permissions, access to the OSS path fails.

   • If the role has the required permissions, you can access the OSS path.

FAQ

• Q: What do I do if the NoPermission message appears when I create a cluster?
  A: Perform the following operations in sequence to resolve the issue:

  1. Check whether the RAM user that you used to create the cluster is authorized to create clusters and change ECS application roles. If the RAM user is granted the AliyunEMRDevelopAccess permission, you must change the permission to AliyunEMRFullAccess.
  2. Check whether the name of the ECS application role that you specified when you create the cluster is valid.
  3. Check whether the trust policy is changed to emr.aliyuncs.com.

• Q: Why can't HDFS access an OSS path?

  A: Troubleshoot as follows:

  1. Verify that the OSS bucket and the cluster are in the same region. If they are in different regions, you must add the corresponding endpoint to the access path.

  2. Verify that the OSS bucket is included as a resource in the policy you created. If not, modify the policy to include it.

  3. Check whether permissions for the bucket are configured in the OSS console. If so, remove them and manage permissions through the Action element in your RAM policy instead.