Co and recursive calls

Introduction

Synchronous recursive functions can quickly cause a process to crash from a stack overflow if the exit condition fails. Asynchronous recursion is sometimes used to avoid this problem:

async function recursive() {
  if( active ) return;
  // do something
  await recursive();
}

The function call after the `await` keyword can span multiple event loops. This approach prevents stack overflow errors. However, this method is not foolproof. This topic describes a production failure that illustrates this point.

Identifying the problem

A customer started using the Node.js Performance Platform, and monitoring revealed frequent out-of-memory (OOM) errors caused by memory growth. The customer then added an alerting rule: @heap_used / @heap_limit > 0.5. The goal of this rule was to generate a heapsnapshot file for analysis when a memory leak occurred, even if the heap was small.

After receiving authorization, we accessed the customer's project and viewed the generated heapsnapshot file. The process trend graph also showed side effects from the high memory usage, such as longer garbage collection (GC) times, which reduced the process's efficiency:

Locating the problem

The heap snapshot file provided a general idea of where the memory leak was located, but finding the exact source proved difficult.

Heap snapshot analysis

The first piece of information was the memory leak report:

The file was nearly 1 GB. The term `(context)` indicates that this is not a normal object. It is a context object, such as a closure, created during function execution. This context object may not be destroyed after the function finishes execution.

This context object was also related to the `co` module. This suggests that `co` scheduled a long-running Generator. Otherwise, this type of context object would be garbage collected after execution.

However, this information alone was not enough to draw a conclusion. We continued the analysis.

We tried to inspect the object content at `@22621725` and trace its reference from the GC root, but this was unsuccessful.

The next useful information came from the object cluster view:

This view shows a chain of a `context` object referencing another `context` object, with a `Promise` in between. If you are familiar with `co`, you know that it wraps non-Promise calls into a Promise. The `Promise` here signifies a new Generator call.

The reference chain is very long. After expanding 20 layers, the `Percent` value had barely decreased. This approach proved inconclusive.

The next useful piece of information was in the class view:

The graph showed an uncommon object: `scheduleUpdatingTask`.

There were 390,285 `scheduleUpdatingTask` objects in this heap snapshot. Click the class to view its details:

This class is located in the `/home/xxx/app/schedule/updateDeviceInfo.js` file.

These were all the clues available from the snapshot. The next step was to analyze the code.

Code analysis

After receiving authorization from the customer, we obtained the relevant code and found `scheduleUpdatingTask` in the `app/schedule/updateDeviceInfo.js` file.

// Execute business logic, wait for a moment after success, and then continue.
// If acquiring the lock fails, stop.
const scheduleUpdatingTask = function* (ctx) {
  if (!taskActive) return;
  try {
    yield doSomething(ctx);
  } catch (e) {
    // Catch business exceptions so that the next schedule can run even if this one fails.
    ctx.logger.error(e);
  }
  yield scheduleUpdatingTask(ctx);
};

In the entire project, the only place that repeatedly called `scheduleUpdatingTask` was the function itself. This is a recursive call.

Of course, this is not a standard recursive call. A standard recursive call would have caused a stack overflow.

The stack did not overflow because in the Co/Generator system, the execution before and after the `yield` keyword spans multiple event loop cycles.

Although a stack overflow was avoided, the context object attached to a Generator is destroyed only after the entire generator finishes execution. Therefore, this recursion created a chain of context objects that referenced each other, which prevented the memory from being reclaimed.

In this code, the termination condition `if (!taskActive) return;` clearly failed.

This implementation explains the observed behavior. To confirm this, we wrote some code to reproduce the problem:

const co = require('co');

function sleep(ms) {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve();
    }, ms);
  });
}

function* task() {
  yield sleep(2);
  console.log(process.memoryUsage());
  yield task();
}

co(function* () {
  yield task();
});

After this code is executed, the application does not crash immediately. Instead, its memory usage gradually increases and exhibits the same behavior as hpmweb.

Of course, we wondered if async functions would also cause this problem:

function sleep(ms) {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve();
    }, ms);
  });
}

async function task() {
  await sleep(2);
  console.log(process.memoryUsage());
  await task();
}

task();

The memory will continue to grow.

Solving the problem

Although analyzing the heap snapshot in the Node.js Performance Platform was not straightforward, we successfully identified the root cause of the problem. Now that the cause has been identified, we can explore the solution.

The examples show that using recursive calls in `co` or `async` functions delays garbage collection. This delay leads to memory accumulation and pressure. This does not mean that you cannot use recursion in this scenario.

However, you must evaluate your application to determine the length of the reference chain that this recursion will create. In the example in this topic, the failed exit condition results in an infinite recursion.

It is possible to continue execution without creating a long context reference chain:

async function task() {
  while (true) {
    await sleep(2);
    console.log(process.memoryUsage());
  }
}

By replacing the recursive call with a `while (true)` loop, you eliminate the context reference chain problem. Because the `await` inside the loop triggers event loop scheduling, the `while (true)` loop does not block the main thread.

A side note

Tail call optimization for normal functions is still not widely supported, let alone for Generator or `async` functions.