Operator profiling (public preview)

Background

While managing Flink deployments, you might encounter unexpected results, a common problem known as a data correctness issue. The causes of data correctness issues are complex and difficult to pinpoint. A common troubleshooting method is to logically deconstruct the SQL deployment and use the Print connector to print the results of each step. However, this process is time-consuming. You might also fail to identify the root cause due to discrepancies between test data, production data, or state. As a result, resolving these issues often takes a long time and requires multiple stop-and-start cycles. Realtime Compute for Apache Flink provides the Operator profiling feature to address these challenges. This feature lets you inspect the input and output of selected operators without modifying the deployment, which helps you troubleshoot data correctness issues more efficiently.

Limitations

• This feature supports only running SQL deployments.

• Only deployments that use Ververica Runtime (VVR) 8.0.4 or later are supported.

• Deployments that use CREATE TABLE AS ... (CTAS) or CREATE DATABASE AS ... (CDAS) statements are not supported.

• Deployments in a session cluster are not supported.

• You cannot start a new profiling session until the current one stops.

Procedure

1. Log on to the Realtime Compute for Apache Flink console and click Console in the Actions column for the target workspace.

2. In the left-side navigation pane, choose O&M > Deployments.

3. Click the name of the target deployment, and then click the Status tab.

4. Enable Operator profiling.

   1. Turn on the Operators Observing switch.

      The Operators Observing section appears at the bottom of the page. It displays the Observing status (such as "Sampling complete") and a list of operators. The list includes the Name, Status, Bytes received, Records received, Bytes sent, and Tasks columns.

   2. Select one or more operators to profile.

      In the Operators Observing section at the bottom of the page, turn on the Expand switch. A message appears, such as 1 operator selected, to confirm the number of selected operators. Set the Max sampling duration and then click Start. The profiling results are available in the inspect-taskmanager_0.out log file on the TaskManager Log List page.

   3. Set the Max Sampling Duration.

      Set a duration between 1 and 30 minutes. Note that sampling may stop early if the storage limit is reached before the duration elapses.

   4. Click Start. The Observing status changes to Sampling.

      Note

      You cannot start a new profiling session until the current one stops.

      After profiling starts, a notification at the bottom of the panel indicates that you can view the results in a log file, such as inspect-taskmanager_0.out, on the TaskManager Log List page. To stop sampling, click Stop.

5. View the profiling results.

   Click TaskManager Log List in the prompt to go to the runtime logs page and view the log with the Log name inspect-taskmanager_0.out.

   Copy the operator name from the DAG on the Status tab, and then search in the log file to view the output of a specific operator.

   Each line in the log is a JSON record containing fields such as op_name (the operator name) and source (the data direction). A value of OUT indicates outbound data, while IN indicates inbound data. For example, a source operator only produces output, so its records have a source value of OUT. A sink operator only consumes input, so its records have a source value of IN. In the filter in the upper-right corner of the logs page, you can enter an operator name, such as Sink: blackhole_sink, to filter the logs. Matching operator names are highlighted.

How profiling stops

• Profiling stops automatically if the JobManager (JM) or a TaskManager (TM) fails over.

• Sampling stops early if the maximum storage limit is reached.

• Sampling stops when the configured sampling duration is reached.

• You can also stop profiling manually.

References

• What do the colors of the legends in the Status Durations column of a vertex subtask indicate?

• Data is written as expected (for example, Kafka receives data from Flink), but the deployment status overview in Flink shows that no data is processed. What can I do?