Flink metrics

更新时间:
复制 MD 格式

Managed Service for Prometheus collects metrics from Alibaba Cloud Realtime Compute for Apache Flink to help you monitor job health, throughput, latency, checkpoint performance, and JVM resource usage.

Flink metrics follow a hierarchical naming pattern that reflects where each metric is collected:

  • flink_jobmanager_* -- Cluster-level metrics reported by the JobManager.

  • flink_taskmanager_job_task_* -- Task-level metrics reported by individual TaskManagers.

  • flink_taskmanager_job_task_operator_* -- Operator-level metrics scoped to a specific operator within a task.

Use these prefixes to target the right scope when writing PromQL queries or configuring alert rules.

Important

Managed Service for Prometheus charges based on the volume of data written or the number of data points reported. Metrics fall into two categories:

  • Basic metrics -- Reporting basic metrics from Alibaba Cloud Realtime Compute for Apache Flink to Managed Service for Prometheus is free of charge. This benefit does not apply to other Flink services, such as self-managed clusters.

  • Custom metrics -- All other metrics are custom metrics. Custom metrics have been billed since January 6, 2020.

Job health

MetricDescriptionDetailsUnitType
flink_jobmanager_job_numRestartsNumber of job restarts caused by errorsCounts error-triggered restarts only. Restarts from JobManager failovers are excluded. A steadily increasing value indicates recurring failures that require investigation.CountCustom
flink_jobmanager_job_uptimeJob uptimeHow long the current job has been running without interruption.msCustom
flink_jobmanager_numRunningJobsNumber of running jobsTotal jobs currently in a running state on this JobManager.N/ACustom
flink_jobmanager_taskSlotsAvailableAvailable task slotsSlots not currently assigned to any task. A value of 0 means no capacity remains for new tasks.N/ACustom
flink_jobmanager_taskSlotsTotalTotal task slotsTotal slot capacity across all registered TaskManagers.N/ACustom
flink_jobmanager_numRegisteredTaskManagersRegistered TaskManagersNumber of TaskManagers currently registered with the JobManager. A drop may indicate TaskManager failures.N/ACustom

Latency

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_operator_currentEmitEventTimeLagBusiness latencyTime difference between the event timestamp and the wall clock at the point of emission. A high value indicates delays in data pulling or processing.msCustom
flink_taskmanager_job_task_operator_currentFetchEventTimeLagTransmission latencyTime difference between the event timestamp and the wall clock at the point of fetching. A high value indicates delays in data pulling from the source.msCustom
flink_taskmanager_job_task_operator_watermarkLagWatermark latencyLatency at the subtask level, measured as the gap between the current watermark and the wall clock.msCustom
flink_taskmanager_job_task_currentInputWatermarkLast received watermarkTimestamp of the most recent watermark received by each task. Useful for detecting event-time skew across subtasks.N/ACustom

Throughput -- records

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_numRecordsInTotal input records (task)If this value stops increasing for an extended period, the upstream source may have stopped sending data.CountCustom
flink_taskmanager_job_task_numRecordsOutTotal output records (task)If this value stops increasing while input records continue to grow, a logic error in the job code may be dropping data.CountCustom
flink_taskmanager_job_task_operator_numRecordsInTotal input records (operator)If this value stops increasing for a specific operator, the upstream operator may have dropped data.CountCustom
flink_taskmanager_job_task_operator_numRecordsOutTotal output records (operator)If this value stops increasing for a specific operator, a logic error may be filtering out all data.CountCustom
flink_taskmanager_job_task_operator_source_numRecordsInInput records (source operator)Records received by the source operator only. Compare against the total task input to isolate source-level throughput.CountCustom
flink_taskmanager_job_task_operator_sink_numRecordsOutOutput records (sink operator)Records written by the sink operator. Compare against operator-level output to verify data reaches the sink.CountCustom
flink_taskmanager_job_task_numRecordsInPerSecondInput records per second (task)Monitor the processing speed of the data stream. Check whether throughput meets expectations under different input loads.Count/sCustom
flink_taskmanager_job_task_numRecordsOutPerSecondOutput records per second (task)Monitor the output speed of the data stream. Compare against input rate to detect processing bottlenecks.Count/sCustom
flink_taskmanager_job_task_operator_numRecordsInPerSecondInput records per second (operator)Per-operator input rate. Useful for identifying which operator is the throughput bottleneck.Count/sCustom
flink_taskmanager_job_task_operator_numRecordsOutPerSecondOutput records per second (operator)Per-operator output rate. Compare against the input rate of the same operator to measure processing efficiency.Count/sCustom
flink_taskmanager_job_task_operator_source_numRecordsInPerSecondInput records per second (source operator)Generation rate of each data source. If this value drops to 0, the upstream source may have stopped producing data.Count/sCustom
flink_taskmanager_job_task_operator_sink_numRecordsOutPerSecondOutput records per second (sink operator)Output rate of each sink. If this value drops to 0, a logic error may be filtering out all data before it reaches the sink.Count/sCustom

Throughput -- bytes

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_operator_numBytesInTotal input bytes (operator)Cumulative bytes received from the upstream source. Use this alongside the per-second rate to assess throughput trends.BytesCustom
flink_taskmanager_job_task_operator_numBytesOutTotal output bytes (operator)Cumulative bytes sent to the downstream sink.BytesCustom
flink_taskmanager_job_task_operator_numBytesInPerSecondInput bytes per second (operator)Byte-level input rate from the upstream source.Bytes/sCustom
flink_taskmanager_job_task_operator_numBytesOutPerSecondOutput bytes per second (operator)Byte-level output rate to the downstream sink.Bytes/sCustom
flink_taskmanager_job_task_numBytesInLocalPerSecondInput bytes per second (local)Byte rate for data received from tasks on the same TaskManager.Bytes/sCustom
flink_taskmanager_job_task_numBytesOutPerSecondOutput bytes per second (task)Byte-level output rate across all downstream channels.Bytes/sCustom
flink_taskmanager_job_task_numBytesInRemotePerSecondInput bytes per second (remote)Byte rate for data received from tasks on other TaskManagers.Bytes/sCustom

Network buffers

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_numBuffersInLocalPerSecondLocal buffers consumed per secondA high value indicates frequent communication between tasks on the same node.Count/sCustom
flink_taskmanager_job_task_numBuffersInRemotePerSecondRemote buffers received per secondReflects the frequency of cross-TaskManager communication. A high value combined with high latency may indicate network congestion.Count/sCustom
flink_taskmanager_job_task_numBuffersOutPerSecondBuffers sent per secondMeasures the output pressure of tasks and network bandwidth usage.Count/sCustom

Source and sink

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_operator_pendingRecordsPending records at sourceNumber of records in the external system that the source operator has not yet pulled. A growing value indicates the source is falling behind.CountCustom
flink_taskmanager_job_task_operator_sourceIdleTimeSource idle timeDuration since the source operator last processed a record. A large value indicates a low data generation rate in the external system.msCustom
flink_taskmanager_job_task_operator_currentSendTimeSend time for the latest recordTime taken to send the most recent record.msCustom

Checkpoints

MetricDescriptionDetailsUnitType
flink_jobmanager_job_totalNumberOfCheckpointsTotal checkpointsCumulative number of checkpoints triggered since the job started.UnitCustom
flink_jobmanager_job_numberOfCompletedCheckpointsCompleted checkpointsNumber of checkpoints that finished successfully.UnitCustom
flink_jobmanager_job_numberOfFailedCheckpointsFailed checkpointsNumber of checkpoints that failed. A non-zero value warrants investigation.unitCustom
flink_jobmanager_job_numberOfInProgressCheckpointsIn-progress checkpointsNumber of checkpoints currently in progress.UnitCustom
flink_jobmanager_job_lastCheckpointDurationLast checkpoint durationTime taken to complete the most recent checkpoint. Possible causes of long durations: large state size, transient network issues, unaligned barriers, or data backpressure.msCustom
flink_jobmanager_job_lastCheckpointSizeLast checkpoint sizeSize of the most recent checkpoint that was actually uploaded. Compare over time to track state growth.BytesCustom

State access latency

All state access latency metrics report the maximum latency of a single operation, measured in nanoseconds (ns). Use these metrics to identify slow state backend operations that may affect overall job performance.

General

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_state_name_stateClearLatencyMax latency of a single state clear operationnsCustom

Value state

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_state_name_valueStateGetLatencyMax latency of a single value state readnsCustom
flink_taskmanager_job_task_operator_state_name_valueStateUpdateLatencyMax latency of a single value state updatenanosecond (ns)Custom

Aggregating state

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_state_name_aggregatingStateGetLatencyMax latency of a single aggregating state readnsCustom
flink_taskmanager_job_task_operator_state_name_aggregatingStateAddLatencyMax latency of a single aggregating state addnsCustom
flink_taskmanager_job_task_operator_state_name_aggregatingStateMergeNamespacesLatencyMax latency of a single aggregating state merge-namespaces operationnsCustom

Reducing state

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_state_name_reducingStateGetLatencyMax latency of a single reducing state readnsCustom
flink_taskmanager_job_task_operator_state_name_reducingStateAddLatencyMax latency of a single reducing state addnsCustom
flink_taskmanager_job_task_operator_state_name_reducingStateMergeNamespacesLatencyMax latency of a single reducing state merge-namespaces operationnsCustom

Map state

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_state_name_mapStateGetLatencyMax latency of a single map state readnsCustom
flink_taskmanager_job_task_operator_state_name_mapStatePutLatencyMax latency of a single map state putnsCustom
flink_taskmanager_job_task_operator_state_name_mapStatePutAllLatencyMax latency of a single map state putAllnsCustom
flink_taskmanager_job_task_operator_state_name_mapStateRemoveLatencyMax latency of a single map state removensCustom
flink_taskmanager_job_task_operator_state_name_mapStateContainsLatencyMax latency of a single map state contains checknsCustom
flink_taskmanager_job_task_operator_state_name_mapStateEntriesInitLatencyMax latency of a single map state entries iterator initnsCustom
flink_taskmanager_job_task_operator_state_name_mapStateKeysInitLatencyMax latency of a single map state keys iterator initnsCustom
flink_taskmanager_job_task_operator_state_name_mapStateValuesInitLatencyMax latency of a single map state values iterator initnsCustom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorInitLatencyMax latency of a single map state iterator initnsCustom
flink_taskmanager_job_task_operator_state_name_mapStateIsEmptyLatencyMax latency of a single map state isEmpty checknsCustom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorHasNextLatencyMax latency of a single map state iterator hasNextnsCustom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorNextLatencyMax latency of a single map state iterator nextnsCustom
flink_taskmanager_job_task_operator_state_name_mapStateIteratorRemoveLatencyMax latency of a single map state iterator removensCustom

List state

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_state_name_listStateGetLatencyMax latency of a single list state readnsCustom
flink_taskmanager_job_task_operator_state_name_listStateAddLatencyMax latency of a single list state addnsCustom
flink_taskmanager_job_task_operator_state_name_listStateAddAllLatencyMax latency of a single list state addAllnsCustom
flink_taskmanager_job_task_operator_state_name_listStateUpdateLatencyMax latency of a single list state updatensCustom
flink_taskmanager_job_task_operator_state_name_listStateMergeNamespacesLatencyMax latency of a single list state merge-namespaces operationnsCustom

Sorted map state

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_state_name_sortedMapStateFirstEntryLatencyMax latency of accessing the first entry of a sorted map statensCustom
flink_taskmanager_job_task_operator_state_name_sortedMapStateLastEntryLatencyMax latency of accessing the last entry of a sorted map statensCustom

GeminiDB state storage

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_operator_geminiDB_total_sizeState data sizeTotal size of state data managed by GeminiDB. Use this metric to locate nodes where state bottlenecks may occur and to verify whether TTL policies are taking effect.ByteCustom
flink_taskmanager_job_task_operator_geminiDB_total_filesizeState data file sizeSize of the state data files on local disk. Monitor this metric to detect disk space pressure caused by large state and to take action before disk runs out.ByteCustom

Window metrics

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_operator_numLateRecordsDroppedLate records droppedNumber of records dropped because they arrived after the window closed.UnitCustom
flink_taskmanager_job_task_operator_lateRecordsDroppedRateLate records drop rateRatio of late records dropped to total records processed.N/ACustom

CDC connector metrics

These metrics apply only to Flink CDC (Change Data Capture) connector jobs. They track progress through the snapshot (full data) and binlog (incremental data) phases.

Phase indicators

MetricDescriptionDetailsUnitType
flink_jobmanager_job_operator_coordinator_enumerator_isSnapshottingSnapshot phase activeWhether the job is in the full-data (snapshot) processing phase.N/ACustom
flink_jobmanager_job_operator_coordinator_enumerator_isBinlogReadingBinlog phase activeWhether the job is in the incremental-data (binlog) processing phase.N/ACustom

Snapshot progress

MetricDescriptionDetailsUnitType
flink_jobmanager_job_operator_coordinator_enumerator_numTablesRemainingTables remaining (snapshot)Number of tables not yet processed during the snapshot phase.UnitCustom
flink_jobmanager_job_operator_coordinator_enumerator_numTablesSnapshottedTables pending processing (snapshot)Number of tables waiting to be processed during the snapshot phase.ItemCustom
flink_jobmanager_job_operator_coordinator_enumerator_numSnapshotSplitsProcessedTables processed (snapshot)Number of tables that have completed snapshot processing.ItemCustom
flink_jobmanager_job_operator_coordinator_enumerator_cdcns_schema_table_numSnapshotSplitsProcessedShards processed (snapshot)Number of shards that have completed snapshot processing.UnitCustom
flink_jobmanager_job_operator_coordinator_enumerator_numSnapshotSplitsRemainingShards remaining (snapshot)Number of shards waiting to be processed during the snapshot phase.UnitCustom
flink_jobmanager_job_operator_coordinator_enumerator_cdcns_schema_table_numSnapshotSplitsRemainingShards remaining per table (snapshot)Number of shards waiting to be processed for a specific table.UnitCustom

Data volume

MetricDescriptionDetailsUnitType
flink_taskmanager_job_task_operator_currentReadTimestampMsLatest read timestampTimestamp of the most recent data record read. Use this to track binlog read progress.msCustom
flink_taskmanager_job_task_operator_numSnapshotRecordsRecords processed (snapshot)Total number of records processed during the snapshot phase.CountCustom
flink_taskmanager_job_task_operator_cdcns_schema_table_numRecordsInRecords read per tableTotal records read from each table across all phases.CountCustom
flink_taskmanager_job_task_operator_cdcns_schema_table_numSnapshotRecordsRecords processed per table (snapshot)Records processed for each table during the snapshot phase.CountCustom

Incremental phase DML and DDL

MetricDescriptionUnitType
flink_taskmanager_job_task_operator_numInsertDMLRecordsINSERT statements processed (total)CountCustom
flink_taskmanager_job_task_operator_numUpdateDMLRecordsUPDATE statements processed (total)ItemsCustom
flink_taskmanager_job_task_operator_numDeleteDMLRecordsDELETE statements processed (total)ItemCustom
flink_taskmanager_job_task_operator_numDDLRecordsDDL statements processed (total)ItemCustom
flink_taskmanager_job_task_operator_cdcns_schema_table_numInsertDMLRecordsINSERT statements processed (per table)CountCustom
flink_taskmanager_job_task_operator_cdcns_schema_table_numUpdateDMLRecordsUPDATE statements processed (per table)ItemsCustom
flink_taskmanager_job_task_operator_cdcns_schema_table_numDeleteDMLRecordsDELETE statements processed (per table)CountCustom
flink_taskmanager_job_task_operator_cdcns_schema_table_numDDLRecordsDDL statements processed (per table)ItemCustom

JobManager JVM and system resources

These basic metrics are free of charge when reported from Alibaba Cloud Realtime Compute for Apache Flink. G1 garbage collector metrics are custom metrics.

CPU

MetricDescriptionDetailsUnitType
flink_jobmanager_Status_JVM_CPU_LoadCPU loadRecent CPU load of the JobManager JVM. A value consistently above 100% indicates the CPU is saturated, which may cause high response times or system stuttering.N/ABasic
flink_jobmanager_Status_ProcessTree_CPU_UsageCPU utilizationCPU time consumed by the JobManager process. 100% = one core fully used, 400% = four cores fully used. If load is high but utilization is low, frequent I/O may be causing processes to enter an uninterruptible sleep state.N/ABasic

Memory

MetricDescriptionUnitType
flink_jobmanager_Status_JVM_Memory_Heap_UsedHeap memory usedBytesBasic
flink_jobmanager_Status_JVM_Memory_Heap_CommittedHeap memory committedByteBasic
flink_jobmanager_Status_JVM_Memory_Heap_MaxHeap memory maxBytesBasic
flink_jobmanager_Status_JVM_Memory_NonHeap_UsedNon-heap memory usedByteBasic
flink_jobmanager_Status_JVM_Memory_NonHeap_CommittedNon-heap memory committedByteBasic
flink_jobmanager_Status_JVM_Memory_NonHeap_MaxNon-heap memory maxBytesBasic

Threads

MetricDescriptionDetailsUnitType
flink_jobmanager_Status_JVM_Threads_CountThread countExcessive threads consume memory and reduce job stability.UnitBasic

Garbage collection

MetricDescriptionDetailsUnitType
flink_jobmanager_Status_JVM_GarbageCollector_ParNew_CountParNew GC countFrequent GCs increase memory overhead and degrade job performance. Use this metric for diagnostics and failure troubleshooting.CountBasic
flink_jobmanager_Status_JVM_GarbageCollector_ParNew_TimeParNew GC timeLong GC pauses degrade job performance. Use this metric alongside GC count for root-cause analysis.msBasic
flink_jobmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_CountCMS GC countCumulative number of CMS garbage collections.CountBasic
flink_jobmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_TimeCMS GC timeCumulative time spent in CMS garbage collection.msBasic
flink_jobmanager_Status_JVM_GarbageCollector_G1_Young_Generation_CountG1 young generation GC countCumulative number of G1 young generation garbage collections.CountCustom
flink_jobmanager_Status_JVM_GarbageCollector_G1_Young_Generation_TimeG1 young generation GC timeCumulative time spent in G1 young generation garbage collection.msCustom
flink_jobmanager_Status_JVM_GarbageCollector_G1_Old_Generation_CountG1 old generation GC countCumulative number of G1 old generation garbage collections.CountCustom
flink_jobmanager_Status_JVM_GarbageCollector_G1_Old_Generation_TimeG1 old generation GC timeCumulative time spent in G1 old generation garbage collection.msCustom

Class loading

MetricDescriptionDetailsUnitType
flink_jobmanager_Status_JVM_ClassLoader_ClassesLoadedClasses loadedTotal classes loaded since JVM creation. Excessive class loading increases memory consumption and may degrade performance.N/ABasic
flink_jobmanager_Status_JVM_ClassLoader_ClassesUnloadedClasses unloadedTotal classes unloaded since JVM creation. Excessive class unloading increases memory overhead and may degrade performance.N/ABasic

TaskManager JVM and system resources

These basic metrics are free of charge when reported from Alibaba Cloud Realtime Compute for Apache Flink. G1 garbage collector metrics are custom metrics.

CPU

MetricDescriptionDetailsUnitType
flink_taskmanager_Status_JVM_CPU_LoadCPU loadCPU load of a single TaskManager. In Flink, CPU load is calculated as CPU Usage / Number of CPU cores. If flink_taskmanager_Status_JVM_CPU_Load exceeds this value, CPU processing may be blocked.N/ABasic
flink_taskmanager_Status_ProcessTree_CPU_UsageCPU utilizationCPU time consumed by the TaskManager process. 100% = one core fully used, 400% = four cores fully used. If load is high but utilization is low, frequent I/O may be causing processes to enter an uninterruptible sleep state.N/ABasic

Memory

MetricDescriptionUnitType
flink_taskmanager_Status_JVM_Memory_Heap_UsedHeap memory usedBytesBasic
flink_taskmanager_Status_JVM_Memory_Heap_CommittedHeap memory committedByteBasic
flink_taskmanager_Status_JVM_Memory_Heap_MaxHeap memory maxBytesBasic
flink_taskmanager_Status_JVM_Memory_NonHeap_UsedNon-heap memory usedByteBasic
flink_taskmanager_Status_JVM_Memory_NonHeap_CommittedNon-heap memory committedBytesBasic
flink_taskmanager_Status_JVM_Memory_NonHeap_MaxNon-heap memory maxBytesBasic
flink_taskmanager_Status_ProcessTree_Memory_RSSResident set size (RSS)BytesBasic

Threads

MetricDescriptionDetailsUnitType
flink_taskmanager_Status_JVM_Threads_CountThread countExcessive threads consume memory and reduce job stability.UnitBasic

Garbage collection

MetricDescriptionDetailsUnitType
flink_taskmanager_Status_JVM_GarbageCollector_ParNew_CountParNew GC countFrequent GCs increase memory overhead and degrade job performance. Use this metric for diagnostics and task-level failure troubleshooting.CountBasic
flink_taskmanager_Status_JVM_GarbageCollector_ParNew_TimeParNew GC timeLong GC pauses degrade job performance. Use this metric alongside GC count for root-cause analysis.msBasic
flink_taskmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_CountCMS GC countCumulative number of CMS garbage collections.CountBasic
flink_taskmanager_Status_JVM_GarbageCollector_ConcurrentMarkSweep_TimeCMS GC timeCumulative time spent in CMS garbage collection.msBasic
flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_CountG1 young generation GC countCumulative number of G1 young generation garbage collections.CountCustom
flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_TimeG1 young generation GC timeCumulative time spent in G1 young generation garbage collection.msCustom
flink_taskmanager_Status_JVM_GarbageCollector_G1_Old_Generation_CountG1 old generation GC countCumulative number of G1 old generation garbage collections.CountCustom
flink_taskmanager_Status_JVM_GarbageCollector_G1_Old_Generation_TimeG1 old generation GC timeCumulative time spent in G1 old generation garbage collection.msCustom

Class loading

MetricDescriptionDetailsUnitType
flink_taskmanager_Status_JVM_ClassLoader_ClassesLoadedClasses loadedTotal classes loaded since JVM creation. Excessive class loading increases memory consumption and may degrade performance.N/ABasic
flink_taskmanager_Status_JVM_ClassLoader_ClassesUnloadedClasses unloadedTotal classes unloaded since JVM creation. Excessive class unloading increases memory overhead and may degrade performance.N/ABasic

Common metric labels

Every Flink metric includes the following Prometheus labels, which you can use to filter and aggregate metrics in PromQL queries:

LabelDescription
vvpNamespaceProject namespace
deploymentNameDeployment name
deploymentIdDeployment ID
jobIdJob ID

See also