DocsICP FilingConsole
官方文档
首页

View Jobs in the Console

更新时间：
复制 MD 格式
产品详情
我的收藏

View Compute Engine job status, logs, and history via the Spark web UI, HDFS shell, or History Server.

Prerequisites

  • The Compute Engine service for your Lindorm instance must be activated. For more information, see Activate the service.

  • Your client's IP address must be in the whitelist for your Lindorm instance. For more information, see Configure a whitelist.

View Spark job information

Access the Spark web UI

Note

To access the Spark web UI, paste its address into your browser's address bar. The interface opens to the Jobs page by default. The header displays basic application information, such as User, Total Uptime, and Scheduling Mode (for example, FIFO). Below the header is an expandable Event Timeline section.

Table 1. Spark web UI tabs

Tab

Description

Jobs

Displays detailed information about all jobs.

Stages

Displays status information for all stages of a Spark job.

Storage

Displays storage information for persisted resilient distributed datasets (RDDs) and DataFrames (DFs).

Environment

Displays the environment configuration for the Spark job, compiled at startup from the runtime environment, configuration files, and user-specified parameters.

Executors

Displays the status of the executors for the Spark job.

SQL

Displays details for SQL queries executed by the Lindorm Distributed Processing System (LDPS).

Kyuubi Query Engine

Displays information about Java Database Connectivity (JDBC) sessions.

View job status

View job status using the Spark web UI

You can use the Spark web UI to view a Spark job's status.

Note

  • You can view a job's status on the Spark web UI only while it is Running.

  • To view completed Spark jobs, you must enable the History Server. For more information, see View job history.

  1. Log on to the Lindorm console.

  2. On the Instances page, click the ID of the target instance.

  3. In the left-side navigation pane, choose Compute Engine > Jobs. This page lists the running Compute Engine jobs.

  4. Click the WebUI Address of the desired Compute Engine job, and log in to the Spark web UI with the Wide Table Engine username and password.

    Note

    You can find the default username and password for the Wide Table Engine on the Database Connections page, under the Wide Table Engine tab.

  5. In the top navigation bar, click Executors to view details about the running job and its executors.

    The Executors page has two sections. The Summary section at the top displays the number of executors and aggregated metrics, categorized as Active, Dead, and Total. The Executors table below lists details for each executor, including its ID, Address, Status, resource usage (Storage Memory, Disk Used, Cores), and task metrics (Active Tasks, Failed Tasks, Complete Tasks, Total Tasks, Task Time (GC Time)).

  6. In the Executors list, click stdout or stderr in the Logs column to view the runtime logs. You can also click Thread Dump to view thread stack information or System Status Dump to view the executor's system status.

    Note

    • Click stdout to view the standard output log.

    • Click stderr to view the standard error log.

View job logs using the HDFS shell

Runtime logs for Compute Engine jobs are automatically stored in the underlying LindormDFS. To review logs from an interrupted job, activate Lindorm and use the HDFS shell.

Important

  • When you submit a JAR job, initialize the SparkSession object at the beginning of the Main function. This ensures that logs from abnormal runs are also synchronized to LindormDFS.

  • If your Compute Engine service runs a large number of jobs, the volume of logs might put significant pressure on LindormDFS. To mitigate this, you can set the spark.dfsLog.executor.enabled=false parameter when you start a job. This parameter prevents executor logs from being saved to LindormDFS. The collection of driver logs is not affected.

  1. Log on to the Lindorm console.

  2. On the Instances page, click the ID of the target instance.

  3. In the left-side navigation pane, choose Compute Engine > Jobs, and find the JobId of your job. For example: 562f7c98-2a66-****.

  4. Use the HDFS shell to view the logs. For information about how to configure the HDFS shell, see Connection guide.

  5. The logs for a Compute Engine job are stored in the /ldspark/ldspark-logs/${JobId} directory. The driver logs are stored in the __driver_logs__ subdirectory, and the executor logs are stored in the __executor_logs__/${EXECUTOR_ID} subdirectory. For example, to view the stderr log for the driver, run the following command:

    $HADOOP_HOME/bin/hadoop fs -cat  /ldspark/ldspark-logs/${JobId}/__driver_logs__/stderr | less
    Note

    You can also mount the LindormDFS directory to an ECS instance by using a FUSE client. For specific steps, see Connect to and use LindormDFS through HDFS FUSE.

View job run history

Billing

You are charged based on the resources that your Lindorm instance consumes and its duration of use. The Lindorm instance consumes 1 CU (1 CU = 1 vCPU and 4 GB of memory). The price for 1 CU is CNY 0.203 per hour. Billing starts as soon as you enable the History Server, regardless of whether you use it.

Enable the History Server

  1. Log on to the Lindorm console.

  2. On the Instances page, click the ID of the target instance.

  3. In the left-side navigation pane, click compute engine.

  4. Click the service management tab.

  5. Click the History Server tab and turn on the Spark History switch.

View job history

  1. Log on to the Lindorm console.

  2. On the Instances page, click the ID of the target instance.

  3. In the left-side navigation pane, click compute engine.

  4. Click the service management tab.

  5. Click the History Server tab.

  6. Click the WebUI URL. Log on to the History Server page with the username and password for the wide-column engine to view job run history.

    Note

    • To get the default username and password for the wide-column engine, go to the database connection page, click the wide-column engine tab, and then copy them.

    • If the History Server page displays the message No completed applications found!, it means no historical jobs are available or the job history has expired and been cleared.

    The History Server page displays historical job information in a table. The table includes the following columns: Version, App ID, App name, Started, Completed, Duration, Spark user, Last updated, and Event log. The Event log column contains a Download button.

    Task

    Actions

    Filter historical jobs

    In the Search text box in the upper-right corner of the History Server page, enter an App name, Started time, or Completed time to filter the jobs.

    View running jobs

    In the bottom-left corner of the History Server page, click Show incomplete applications.

  7. Click an App ID to open the Spark UI for the corresponding job.