E-MapReduce Logs

Asset Details

• Custom project and dedicated Logstore

  When you create a custom project, the system automatically creates a dedicated Logstore named emr_EMR service name_log in the project. For example, if you deliver logs from the host service, the Logstore is named emr_host_log.

  Important

  • Do not delete the related Simple Log Service project or Logstore before you disable the log feature. Otherwise, stored logs are purged and new logs cannot be delivered to Simple Log Service.

  • If you enable the pay-by-ingested-data billing mode, the system creates a dedicated Logstore that uses this mode by default. To switch to the pay-by-feature mode, you can modify the Logstore configuration. For more information, see Modify Logstore configurations.

• Dedicated dashboard

  No dedicated dashboard is provided. You can create a custom dashboard to display query results. For more information, see Create a dashboard.

Billing

• Currently, EMR does not charge for the log management feature.

• If the Logstore uses the pay-by-feature billing mode, Simple Log Service charges for storage space, read traffic, requests, data transformation, and data shipping. For more information, see Billable items of the pay-by-feature mode.

• If the Logstore uses the pay-by-ingested-data billing mode, Simple Log Service charges based on factors such as the amount of raw data ingested. For more information, see Billable items of the pay-by-ingested-data mode.

Limits

• The log management feature is available only for clusters in new DataLake, DataFlow, Online Analytical Processing (OLAP), DataServing, and Hadoop scenarios.

• Log delivery is supported for the following services: HDFS, YARN, yarn_application, Hive, Spark, JindoData, Tez, Flink, HBase, ZooKeeper, Kafka, Presto, Kudu, Impala, Flume, StarRocks, ClickHouse, Kyuubi, Remote Shuffle Service (RSS), and host.

Enable log delivery

1. Go to the Log Management page.

   1. Log on to the E-MapReduce console.

   2. In the top navigation bar, select a region and a resource group as needed.

   3. On the EMR on ECS page, click the Cluster ID of the target cluster.

   4. Click the Logs tab.

2. Configure the log shipping scope.

   1. On the Logs page, click Set Log Shipping Scope in the upper-right corner.

   2. In the dialog box that appears, configure the Project for log storage.

      You can select an existing Project or create a new one.

      Important

      You cannot change the Project once it is selected.

   3. Select the services whose logs you want to ship, and then click OK.

3. View service logs.

   1. From the Please select EMR service drop-down list, select the service whose logs you want to view.

   2. Analyze logs on the log page.

      You can specify a time range and a query statement to perform real-time queries. For example, you can quickly analyze the distribution of a specific field over a period, or filter and aggregate access records based on specified conditions.

Related operations

What to do next

After you deliver EMR logs to Simple Log Service, you can query and analyze logs, download data, ship and transform data, and configure alerts. For more information, see General operations for cloud service logs.