DocsICP FilingConsole
Documentation
Home Page Dataphin Dataphin(Non-shared) User Guide Dataphin Manager Application O&M Log query

Log query

Updated at:
Copy as MD
Product
My Favorites

Use Log Query to view, filter, and download application logs, end-to-end trace logs, and Pod logs.

Application logs

Filter and download logs

  1. In the top menu bar of Dataphin Manager, click Application O&M.

  2. In the left-side navigation pane, choose Application O&M > Log Query.

  3. On the Application Log Search tab, you can apply the following filters.

    • Keyword: Enter a keyword of at least five characters and click Search to find relevant logs. The search progress for each application subdomain is displayed below.

    • Application subdomain name: Select one or more application subdomains from the drop-down list. You can also enter a keyword to search for a specific application subdomain.

    • Log type: Select one or more log types. Options include application log, RPC log, Nginx log, JVM log, audit log, and Mesos log. The Mesos log option appears only if the scheduling cluster is Mesos. By default, application log, RPC log, Nginx log, and JVM log are selected.

    • File time range: Select Last 1 day, Last 2 days, Last 3 days, or Last 5 days. By default, Last 1 day is selected.

    • Display context: Specify the number of lines (n) to display before and after the keyword. The value must be an integer from 1 to 500.

  4. Click Filter and Download Logs to download the filtered logs.

Download task list

  1. Click Full Log Download. In the Download Selection dialog box, select an application and click OK to create a download task.

  2. Click Download Task List. The list displays the Application name, Pod name, Download time, and Download progress for each task. In the Actions column for a specific task, click Download to save the application log to your computer.

    Note

    • By default, you can only download directories that are smaller than 1 GB before compression.

    • The full log download provides a compressed file from the /home/admin/logs directory. You can download logs for only one application at a time.

    • Logs in the download task list are stored in object storage and are kept for up to 3 days.

End-to-end trace logs

On the End-to-end Trace Log Search tab, enter an instance ID that starts with 't' and click Search. You can find the instance ID in the instance O&M list. The search results show basic information about the instance, including a sequence number, creation time, and last update time. The sequence number indicates the number of retries. The system automatically retries failed instances based on your settings, while a manual rerun runs only once.

Click Expand next to a sequence number to view details. Then, click execution log, task scheduling log, resource scheduling log, or executor log to view the corresponding log details in the code pane on the right. To download a specific log, click the image icon next to its type. To download all log types for that instance, click Download next to the sequence number.

  • execution log: The execution log of the task instance.

  • task scheduling log: Logs generated by the task scheduling system for the task instance.

  • resource scheduling log: Logs generated by the resource scheduling system for the task instance.

  • executor log: The log from the executor while it runs the task instance.

Pod logs

On the Pod Log Search tab, select a namespace (required) and a Pod (required), then specify the Log start time and Line limit to search for the logs of a specific Pod. The code block below displays the details of the retrieved Pod log.

  • The system automatically retrieves the namespaces from K8s. This list includes namespaces required for Dataphin deployment. If you are using a shared K8s cluster, it also includes namespaces used by other services. The namespaces typically used by Dataphin include the following:

    • Dataphin: The namespace for the Dataphin application.

    • Dataphin-schedule-pool: The namespace for Dataphin task scheduling.

    • Stolon: The self-hosted PostgreSQL metadata database for Dataphin.

    • Redis: The self-hosted Redis for Dataphin.

    • RocketMQ: The self-hosted RocketMQ for Dataphin.

    • Rook-ceph: The self-hosted Ceph object storage for Dataphin.

  • Pod: Select a Pod from the specified namespace.

  • Log start time: Select a time in YYYY-MM-DD hh:mm:ss format. The search results will only include log entries from after the selected time. Dataphin application logs support time filtering. For other logs, this feature works only if the log entries begin with a timestamp in YYYY-MM-DD hh:mm:ss format.

  • Line limit: Specifies the maximum number of lines to retrieve. Enter an integer from 1 to 1,000.

Note

  • A backend toggle controls whether Dataphin application logs are sent to the Pod's standard output. If enabled, you can view the application log for a selected Pod on the Pod Log Search tab. You can also use a log collection tool, such as Logtail, to collect application logs from the standard output. To enable this toggle, contact the Dataphin product O&M team.

  • The Pod's standard output only contains the application log. To view the RPC log, Nginx log, JVM log, and audit log, use the Application Log Search tab.

Previous：Instance O&MNext：Version upgrade