Flink metrics

更新时间:
复制 MD 格式

Managed Service for Prometheus collects metrics from Realtime Compute for Apache Flink, covering job health, throughput, latency, checkpoints, and JVM resources.

Flink metrics use a hierarchical naming pattern:

  • flink_jobmanager_* -- Cluster-level metrics reported by the JobManager.

  • flink_taskmanager_job_task_* -- Task-level metrics reported by individual TaskManagers.

  • flink_taskmanager_job_task_operator_* -- Operator-level metrics scoped to a specific operator within a task.

Use these prefixes to scope PromQL queries and alert rules.

Important

Managed Service for Prometheus charges based on the volume of data written or the number of data points reported. Metrics fall into two categories:

  • Basic metrics -- Reporting basic metrics from Alibaba Cloud Realtime Compute for Apache Flink to Managed Service for Prometheus is free of charge. This benefit does not apply to other Flink services, such as self-managed clusters.

  • Custom metrics -- All other metrics are custom metrics. Custom metrics have been billed since January 6, 2020.

Job health

Metric Description Details Unit Type
flink_jobmanager_job_numRestarts Number of job restarts caused by errors Counts error-triggered restarts only. Restarts from JobManager failovers are excluded. A steadily increasing value indicates recurring failures that require investigation. Count Custom
flink_jobmanager_job_uptime Job uptime How long the current job has been running without interruption. ms Custom
flink_jobmanager_numRunningJobs Number of running jobs Total jobs currently in a running state on this JobManager. N/A Custom
flink_jobmanager_taskSlotsAvailable Available task slots Slots not currently assigned to any task. A value of 0 means no capacity remains for new tasks. N/A Custom
flink_jobmanager_taskSlotsTotal Total task slots Total slot capacity across all registered TaskManagers. N/A Custom
flink_jobmanager_numRegisteredTaskManagers Registered TaskManagers Number of TaskManagers currently registered with the JobManager. A drop may indicate TaskManager failures. N/A Custom

Latency

Metric Description Details Unit Type
flink_taskmanager_job_task_operator_currentEmitEventTimeLag Business latency Time difference between the event timestamp and the wall clock at the point of emission. A high value indicates delays in data pulling or processing. ms Custom
flink_taskmanager_job_task_operator_currentFetchEventTimeLag Transmission latency Time difference between the event timestamp and the wall clock at the point of fetching. A high value indicates delays in data pulling from the source. ms Custom
flink_taskmanager_job_task_operator_watermarkLag Watermark latency Latency at the subtask level, measured as the gap between the current watermark and the wall clock. ms Custom
flink_taskmanager_job_task_currentInputWatermark Last received watermark Timestamp of the most recent watermark received by each task. Useful for detecting event-time skew across subtasks. N/A Custom

Throughput -- records

Metric Description Details Unit Type
flink_taskmanager_job_task_numRecordsIn Total input records (task) If this value stops increasing for an extended period, the upstream source may have stopped sending data. Count Custom
flink_taskmanager_job_task_numRecordsOut Total output records (task) If this value stops increasing while input records continue to grow, a logic error in the job code may be dropping data. Count Custom
flink_taskmanager_job_task_operator_numRecordsIn Total input records (operator) If this value stops increasing for a specific operator, the upstream operator may have dropped data. Count Custom
flink_taskmanager_job_task_operator_numRecordsOut Total output records (operator) If this value stops increasing for a specific operator, a logic error may be filtering out all data. Count Custom
flink_taskmanager_job_task_operator_source_numRecordsIn Input records (source operator) Records received by the source operator only. Compare against the total task input to isolate source-level throughput. Count Custom
flink_taskmanager_job_task_operator_sink_numRecordsOut Output records (sink operator) Records written by the sink operator. Compare against operator-level output to verify data reaches the sink. Count Custom
flink_taskmanager_job_task_numRecordsInPerSecond Input records per second (task) Monitor the processing speed of the data stream. Check whether throughput meets expectations under different input loads. Count/s Custom
flink_taskmanager_job_task_numRecordsOutPerSecond Output records per second (task) Monitor the output speed of the data stream. Compare against input rate to detect processing bottlenecks. Count/s Custom
flink_taskmanager_job_task_operator_numRecordsInPerSecond Input records per second (operator) Per-operator input rate. Useful for identifying which operator is the throughput bottleneck. Count/s Custom
flink_taskmanager_job_task_operator_numRecordsOutPerSecond Output records per second (operator) Per-operator output rate. Compare against the input rate of the same operator to measure processing efficiency. Count/s Custom
flink_taskmanager_job_task_operator_source_numRecordsInPerSecond Input records per second (source operator) Generation rate of each data source. If this value drops to 0, the upstream source may have stopped producing data. Count/s Custom
flink_taskmanager_job_task_operator_sink_numRecordsOutPerSecond Output records per second (sink operator) Output rate of each sink. If this value drops to 0, a logic error may be filtering out all data before it reaches the sink. Count/s Custom

Throughput -- bytes

Metric Description Details Unit Type
flink_taskmanager_job_task_operator_numBytesIn Total input bytes (operator) Cumulative bytes received from the upstream source. Use this alongside the per-second rate to assess throughput trends. Bytes Custom
flink_taskmanager_job_task_operator_numBytesOut Total output bytes (operator) Cumulative bytes sent to the downstream sink. Bytes Custom
flink_taskmanager_job_task_operator_numBytesInPerSecond Input bytes per second (operator) Byte-level input rate from the upstream source. Bytes/s Custom
flink_taskmanager_job_task_operator_numBytesOutPerSecond Output bytes per second (operator) Byte-level output rate to the downstream sink. Bytes/s Custom
flink_taskmanager_job_task_numBytesInLocalPerSecond Input bytes per second (local) Byte rate for data received from tasks on the same TaskManager. Bytes/s Custom
flink_taskmanager_job_task_numBytesOutPerSecond Output bytes per second (task) Byte-level output rate across all downstream channels. Bytes/s Custom
flink_taskmanager_job_task_numBytesInRemotePerSecond Input bytes per second (remote) Byte rate for data received from tasks on other TaskManagers. Bytes/s Custom

Network buffers

Metric Description Details Unit Type
flink_taskmanager_job_task_numBuffersInLocalPerSecond Local buffers consumed per second A high value indicates frequent communication between tasks on the same node. Count/s Custom
flink_taskmanager_job_task_numBuffersInRemotePerSecond Remote buffers received per second Reflects the frequency of cross-TaskManager communication. A high value combined with high latency may indicate network congestion. Count/s Custom
flink_taskmanager_job_task_numBuffersOutPerSecond Buffers sent per second Measures the output pressure of tasks and network bandwidth usage. Count/s Custom

Source and sink

Metric Description Details Unit Type
flink_taskmanager_job_task_operator_pendingRecords Pending records at source Number of records in the external system that the source operator has not yet pulled. A growing value indicates the source is falling behind. Count Custom
flink_taskmanager_job_task_operator_sourceIdleTime Source idle time Duration since the source operator last processed a record. A large value indicates a low data generation rate in the external system. ms Custom
flink_taskmanager_job_task_operator_currentSendTime Send time for the latest record Time taken to send the most recent record. ms Custom

Checkpoints

Metric Description Details Unit Type
flink_jobmanager_job_totalNumberOfCheckpoints Total checkpoints Cumulative number of checkpoints triggered since the job started. Unit Custom
flink_jobmanager_job_numberOfCompletedCheckpoints Completed checkpoints Number of checkpoints that finished successfully. Unit Custom
flink_jobmanager_job_numberOfFailedCheckpoints Failed checkpoints Number of checkpoints that failed. A non-zero value warrants investigation. unit Custom
flink_jobmanager_job_numberOfInProgressCheckpoints In-progress checkpoints Number of checkpoints currently in progress. Unit Custom
flink_jobmanager_job_lastCheckpointDuration Last checkpoint duration Time taken to complete the most recent checkpoint. Possible causes of long durations: large state size, transient network issues, unaligned barriers, or data backpressure. ms Custom
flink_jobmanager_job_lastCheckpointSize Last checkpoint size Size of the most recent checkpoint that was actually uploaded. Compare over time to track state growth. Bytes Custom

State access latency

All state access latency metrics report the maximum latency of a single operation in nanoseconds (ns). Use them to identify slow state backend operations.

General

Metric Description Unit Type
flink_taskmanager_job_task_operator_state_name_stateClearLatency Max latency of a single state clear operation ns Custom

Value state

Metric Description Unit Type
flink_taskmanager_job_task_operator_state_name_valueStateGetLatency Max latency of a single value state read ns Custom
flink_taskmanager_job_task_operator_state_name_valueStateUpdateLatency Max latency of a single value state update ns Custom

Aggregating state

Metric Description Unit Type
flink_taskmanager_job_task_operator_state_name_aggregatingStateGetLatency Max latency of a single aggregating state read ns Custom
flink_taskmanager_job_task_operator_state_name_aggregatingStateAddLatency Max latency of a single aggregating state add ns Custom
flink_taskmanager_job_task_operator_state_name_aggregatingStateMergeNamespacesLatency Max latency of a single aggregating state merge-namespaces operation ns Custom

Reducing state

Metric Description Unit Type
flink_taskmanager_job_task_operator_state_name_reducingStateGetLatency Max latency of a single reducing state read ns Custom
flink_taskmanager_job_task_operator_state_name_reducingStateAddLatency Max latency of a single reducing state add ns Custom
flink_taskmanager_job_task_operator_state_name_reducingStateMergeNamespacesLatency Max latency of a single reducing state merge-namespaces operation ns Custom

Map state

Metric Description Unit Type
flink_taskmanager_job_task_operator_state_name_mapStateGetLatency Max latency of a single map state read ns Custom
flink_taskmanager_job_task_operator_state_name_mapStatePutLatency Max latency of a single map state put ns Custom
flink_taskmanager_job_task_operator_state_name_mapStatePutAllLatency Max latency of a single map state putAll ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateRemoveLatency Max latency of a single map state remove ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateContainsLatency Max latency of a single map state contains check ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateEntriesInitLatency Max latency of a single map state entries iterator init ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateKeysInitLatency Max latency of a single map state keys iterator init ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateValuesInitLatency Max latency of a single map state values iterator init ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorInitLatency Max latency of a single map state iterator init ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateIsEmptyLatency Max latency of a single map state isEmpty check ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorHasNextLatency Max latency of a single map state iterator hasNext ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorNextLatency Max latency of a single map state iterator next ns Custom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorRemoveLatency Max latency of a single map state iterator remove ns Custom

List state

Metric Description Unit Type
flink_taskmanager_job_task_operator_state_name_listStateGetLatency Max latency of a single list state read ns Custom
flink_taskmanager_job_task_operator_state_name_listStateAddLatency Max latency of a single list state add ns Custom
flink_taskmanager_job_task_operator_state_name_listStateAddAllLatency Max latency of a single list state addAll ns Custom
flink_taskmanager_job_task_operator_state_name_listStateUpdateLatency Max latency of a single list state update ns Custom
flink_taskmanager_job_task_operator_state_name_listStateMergeNamespacesLatency Max latency of a single list state merge-namespaces operation ns Custom

Sorted map state

Metric Description Unit Type
flink_taskmanager_job_task_operator_state_name_sortedMapStateFirstEntryLatency Max latency of accessing the first entry of a sorted map state ns Custom
flink_taskmanager_job_task_operator_state_name_sortedMapStateLastEntryLatency Max latency of accessing the last entry of a sorted map state ns Custom

GeminiDB state storage

Metric Description Details Unit Type
flink_taskmanager_job_task_operator_geminiDB_total_size State data size Total size of state data managed by GeminiDB. Use this metric to locate nodes where state bottlenecks may occur and to verify whether TTL policies are taking effect. Bytes Custom
flink_taskmanager_job_task_operator_geminiDB_total_filesize State data file size Size of the state data files on local disk. Monitor this metric to detect disk space pressure caused by large state and to take action before disk runs out. Bytes Custom

Window metrics

Metric Description Details Unit Type
flink_taskmanager_job_task_operator_numLateRecordsDropped Late records dropped Number of records dropped because they arrived after the window closed. Unit Custom
flink_taskmanager_job_task_operator_lateRecordsDroppedRate Late records drop rate Ratio of late records dropped to total records processed. N/A Custom

CDC connector metrics

These metrics apply only to Flink CDC (Change Data Capture) connector jobs. They track progress through the snapshot (full data) and binlog (incremental data) phases.

Phase indicators

Metric Description Details Unit Type
flink_jobmanager_job_operator_coordinator_enumerator_isSnapshotting Snapshot phase active Whether the job is in the full-data (snapshot) processing phase. N/A Custom
flink_jobmanager_job_operator_coordinator_enumerator_isBinlogReading Binlog phase active Whether the job is in the incremental-data (binlog) processing phase. N/A Custom

Snapshot progress

Metric Description Details Unit Type
flink_jobmanager_job_operator_coordinator_enumerator_numTablesRemaining Tables remaining (snapshot) Number of tables not yet processed during the snapshot phase. Unit Custom
flink_jobmanager_job_operator_coordinator_enumerator_numTablesSnapshotted Tables pending processing (snapshot) Number of tables waiting to be processed during the snapshot phase. Item Custom
flink_jobmanager_job_operator_coordinator_enumerator_numSnapshotSplitsProcessed Tables processed (snapshot) Number of tables that have completed snapshot processing. Item Custom
flink_jobmanager_job_operator_coordinator_enumerator_cdcns_schema_table_numSnapshotSplitsProcessed Shards processed (snapshot) Number of shards that have completed snapshot processing. Unit Custom
flink_jobmanager_job_operator_coordinator_enumerator_numSnapshotSplitsRemaining Shards remaining (snapshot) Number of shards waiting to be processed during the snapshot phase. Unit Custom
flink_jobmanager_job_operator_coordinator_enumerator_cdcns_schema_table_numSnapshotSplitsRemaining Shards remaining per table (snapshot) Number of shards waiting to be processed for a specific table. Unit Custom

Data volume

Metric Description Details Unit Type
flink_taskmanager_job_task_operator_currentReadTimestampMs Latest read timestamp Timestamp of the most recent data record read. Use this to track binlog read progress. ms Custom
flink_taskmanager_job_task_operator_numSnapshotRecords Records processed (snapshot) Total number of records processed during the snapshot phase. Count Custom
flink_taskmanager_job_task_operator_cdcns_schema_table_numRecordsIn Records read per table Total records read from each table across all phases. Count Custom
flink_taskmanager_job_task_operator_cdcns_schema_table_numSnapshotRecords Records processed per table (snapshot) Records processed for each table during the snapshot phase. Count Custom

Incremental phase DML and DDL

Metric Description Unit Type
flink_taskmanager_job_task_operator_numInsertDMLRecords INSERT statements processed (total) Count Custom
flink_taskmanager_job_task_operator_numUpdateDMLRecords UPDATE statements processed (total) Items Custom
flink_taskmanager_job_task_operator_numDeleteDMLRecords DELETE statements processed (total) Item Custom
flink_taskmanager_job_task_operator_numDDLRecords DDL statements processed (total) Item Custom
flink_taskmanager_job_task_operator_cdcns_schema_table_numInsertDMLRecords INSERT statements processed (per table) Count Custom
flink_taskmanager_job_task_operator_cdcns_schema_table_numUpdateDMLRecords UPDATE statements processed (per table) Items Custom
flink_taskmanager_job_task_operator_cdcns_schema_table_numDeleteDMLRecords DELETE statements processed (per table) Count Custom
flink_taskmanager_job_task_operator_cdcns_schema_table_numDDLRecords DDL statements processed (per table) Item Custom

JobManager JVM and system resources

Basic metrics from Realtime Compute for Apache Flink are free of charge. G1 garbage collector metrics are custom metrics.

CPU

Metric Description Details Unit Type
flink_jobmanager_Status_JVM_CPU_Load CPU load Recent CPU load of the JobManager JVM. A value consistently above 100% indicates the CPU is saturated, which may cause high response times or system stuttering. N/A Basic
flink_jobmanager_Status_ProcessTree_CPU_Usage CPU utilization CPU time consumed by the JobManager process. 100% = one core fully used, 400% = four cores fully used. If load is high but utilization is low, frequent I/O may be causing processes to enter an uninterruptible sleep state. N/A Basic

Memory

Metric Description Unit Type
flink_jobmanager_Status_JVM_Memory_Heap_Used Heap memory used Bytes Basic
flink_jobmanager_Status_JVM_Memory_Heap_Committed Heap memory committed Bytes Basic
flink_jobmanager_Status_JVM_Memory_Heap_Max Heap memory max Bytes Basic
flink_jobmanager_Status_JVM_Memory_NonHeap_Used Non-heap memory used Bytes Basic
flink_jobmanager_Status_JVM_Memory_NonHeap_Committed Non-heap memory committed Bytes Basic
flink_jobmanager_Status_JVM_Memory_NonHeap_Max Non-heap memory max Bytes Basic

Threads

Metric Description Details Unit Type
flink_jobmanager_Status_JVM_Threads_Count Thread count Excessive threads consume memory and reduce job stability. Unit Basic

Garbage collection

Metric Description Details Unit Type
flink_jobmanager_Status_JVM_GarbageCollector_ParNew_Count ParNew GC count Frequent GCs increase memory overhead and degrade job performance. Use this metric for diagnostics and failure troubleshooting. Count Basic
flink_jobmanager_Status_JVM_GarbageCollector_ParNew_Time ParNew GC time Long GC pauses degrade job performance. Use this metric alongside GC count for root-cause analysis. ms Basic
flink_jobmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_Count CMS GC count Cumulative number of CMS garbage collections. Count Basic
flink_jobmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_Time CMS GC time Cumulative time spent in CMS garbage collection. ms Basic
flink_jobmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Count G1 young generation GC count Cumulative number of G1 young generation garbage collections. Count Custom
flink_jobmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Time G1 young generation GC time Cumulative time spent in G1 young generation garbage collection. ms Custom
flink_jobmanager_Status_JVM_GarbageCollector_G1_Old_Generation_Count G1 old generation GC count Cumulative number of G1 old generation garbage collections. Count Custom
flink_jobmanager_Status_JVM_GarbageCollector_G1_Old_Generation_Time G1 old generation GC time Cumulative time spent in G1 old generation garbage collection. ms Custom

Class loading

Metric Description Details Unit Type
flink_jobmanager_Status_JVM_ClassLoader_ClassesLoaded Classes loaded Total classes loaded since JVM creation. Excessive class loading increases memory consumption and may degrade performance. N/A Basic
flink_jobmanager_Status_JVM_ClassLoader_ClassesUnloaded Classes unloaded Total classes unloaded since JVM creation. Excessive class unloading increases memory overhead and may degrade performance. N/A Basic

TaskManager JVM and system resources

Basic metrics from Realtime Compute for Apache Flink are free of charge. G1 garbage collector metrics are custom metrics.

CPU

Metric Description Details Unit Type
flink_taskmanager_Status_JVM_CPU_Load CPU load CPU load of a single TaskManager. In Flink, CPU load is calculated as CPU Usage / Number of CPU cores. If flink_taskmanager_Status_JVM_CPU_Load exceeds this value, CPU processing may be blocked. N/A Basic
flink_taskmanager_Status_ProcessTree_CPU_Usage CPU utilization CPU time consumed by the TaskManager process. 100% = one core fully used, 400% = four cores fully used. If load is high but utilization is low, frequent I/O may be causing processes to enter an uninterruptible sleep state. N/A Basic

Memory

Metric Description Unit Type
flink_taskmanager_Status_JVM_Memory_Heap_Used Heap memory used Bytes Basic
flink_taskmanager_Status_JVM_Memory_Heap_Committed Heap memory committed Bytes Basic
flink_taskmanager_Status_JVM_Memory_Heap_Max Heap memory max Bytes Basic
flink_taskmanager_Status_JVM_Memory_NonHeap_Used Non-heap memory used Bytes Basic
flink_taskmanager_Status_JVM_Memory_NonHeap_Committed Non-heap memory committed Bytes Basic
flink_taskmanager_Status_JVM_Memory_NonHeap_Max Non-heap memory max Bytes Basic
flink_taskmanager_Status_ProcessTree_Memory_RSS Resident set size (RSS) Bytes Basic

Threads

Metric Description Details Unit Type
flink_taskmanager_Status_JVM_Threads_Count Thread count Excessive threads consume memory and reduce job stability. Unit Basic

Garbage collection

Metric Description Details Unit Type
flink_taskmanager_Status_JVM_GarbageCollector_ParNew_Count ParNew GC count Frequent GCs increase memory overhead and degrade job performance. Use this metric for diagnostics and task-level failure troubleshooting. Count Basic
flink_taskmanager_Status_JVM_GarbageCollector_ParNew_Time ParNew GC time Long GC pauses degrade job performance. Use this metric alongside GC count for root-cause analysis. ms Basic
flink_taskmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_Count CMS GC count Cumulative number of CMS garbage collections. Count Basic
flink_taskmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_Time CMS GC time Cumulative time spent in CMS garbage collection. ms Basic
flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Count G1 young generation GC count Cumulative number of G1 young generation garbage collections. Count Custom
flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Time G1 young generation GC time Cumulative time spent in G1 young generation garbage collection. ms Custom
flink_taskmanager_Status_JVM_GarbageCollector_G1_Old_Generation_Count G1 old generation GC count Cumulative number of G1 old generation garbage collections. Count Custom
flink_taskmanager_Status_JVM_GarbageCollector_G1_Old_Generation_Time G1 old generation GC time Cumulative time spent in G1 old generation garbage collection. ms Custom

Class loading

Metric Description Details Unit Type
flink_taskmanager_Status_JVM_ClassLoader_ClassesLoaded Classes loaded Total classes loaded since JVM creation. Excessive class loading increases memory consumption and may degrade performance. N/A Basic
flink_taskmanager_Status_JVM_ClassLoader_ClassesUnloaded Classes unloaded Total classes unloaded since JVM creation. Excessive class unloading increases memory overhead and may degrade performance. N/A Basic

Common metric labels

Every Flink metric includes the following Prometheus labels for filtering and aggregation in PromQL:

Label Description
vvpNamespace Project namespace
deploymentName Deployment name
deploymentId Deployment ID
jobId Job ID

References