Runtime OOM issues

View memory usage

LogView 2.0 includes the Sensor feature to view the runtime memory and CPU usage of the Master and each Worker. To troubleshoot OOM issues, first, use Sensor to confirm the memory usage trend and determine whether the memory shortage is on-heap or off-heap.

Cannot allocate memory

This error is usually caused by insufficient off-heap memory.

• On the Executor: Increase the spark.executor.memoryOverhead parameter.

• On the Driver: Increase the spark.driver.memoryOverhead parameter.

  20/09/01 11:05:27 WARN BlockManager: Persisting block rdd_230_291 to disk instead.
  20/09/01 11:05:29 INFO TableReaderUtil: read table close, blockId: 20200901030346786g5j464pr2_4
  20/09/01 11:05:29 INFO MemoryStore: Block rdd_230_51 stored as values in memory (estimated size 2.7 GB, free 8.4 GB)
  20/09/01 11:05:29 INFO Executor: Finished task 51.0 in stage 52.0 (TID 2260). 2210 bytes result sent to driver
  20/09/01 11:05:37 INFO TableReaderUtil: read table close, blockId: 20200901030346406gdkbyssa_9
  20/09/01 11:05:43 INFO MemoryStore: Block rdd_230_171 stored as values in memory (estimated size 2.7 GB, free 5.7 GB)
  20/09/01 11:05:43 INFO Executor: Finished task 171.0 in stage 52.0 (TID 2380). 2253 bytes result sent to driver
  20/09/01 11:05:46 INFO TableReaderUtil: read table close, blockId: 20200901030346386fnh0tsa_11
  20/09/01 11:05:48 INFO TableReaderUtil: read table close, blockId: 20200901030345422g9kbyssa_18
  OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000004ce300000, 3699376128, 0) failed; error='Cannot allocate memory' (errno=12)

java.lang.OutOfMemoryError: Java heap space

This error is usually caused by insufficient on-heap memory. Resolve this issue as follows:

• Increase Executor heap memory: Increase spark.executor.memory.

• Limit the Executor degree of parallelism: Decrease spark.executor.cores. Multiple concurrent tasks share the Executor memory. Decreasing the degree of parallelism provides more memory to each task.

• Increase the number of partitions: This reduces the payload for each Executor.

• Troubleshoot data skew: If only some tasks encounter OOM errors while other tasks have sufficient memory, the issue is usually caused by data skew. Check the data distribution and optimize it.

No route to host / Could not find CoarseGrainedScheduler

The No route to host: workerd********* or Could not find CoarseGrainedScheduler error likely means that some Executor processes were stopped due to an OOM error. This type of fault does not directly indicate an OOM issue. Use LogView Sensor to confirm the memory usage, and then adjust the memory configuration as follows:

• Increase Executor heap memory: Increase spark.executor.memory.

• Limit the Executor degree of parallelism: Decrease spark.executor.cores. Multiple concurrent tasks share the Executor memory. Decreasing the degree of parallelism provides more memory to each task.

• Increase the number of partitions: This reduces the payload for each Executor.

• Troubleshoot data skew: If only some tasks encounter OOM errors while other tasks have sufficient memory, the issue is usually caused by data skew. Check the data distribution and optimize it.