Custom instrumentation of Node.js LLM applications

Use the ARMS Node.js probe for automatic collection

If you only need automatic collection of model SDK calls, use the ARMS Node.js probe. The following combination has been verified with Node.js 20, real DashScope requests, and ARMS server-side data:

npm install \
  @loongsuite/cms_node_sdk@1.0.4 \
  openai@5.23.2

export ARMS_LICENSE="<ARMS License>"
export CMS_SERVICE_NAME="weather-agent"
export ARMS_REGION_ID="cn-hongkong"
# No configuration required for the default workspace; set this only for non-default workspaces
# export ARMS_WORKSPACE="<workspace>"

node -r @loongsuite/cms_node_sdk/register app.js

ARMS_LICENSE is a credential and must not be written into source code, images, logs, or documentation. The service name must be unique and stable within the same environment.

@loongsuite/cms_node_sdk@1.0.4 declares OpenAI automatic instrumentation support for openai >=4 <6. OpenAI 6 is not within this supported range. Even if a model request succeeds, you cannot assume LLM Spans have been generated. After upgrading OpenAI or the probe, you must redo server-side verification.

This probe's preload entry point is designed for long-running services. When a new service starts for the first time, the probe must complete its configuration handshake. CLI scripts, one-shot scripts, and extremely short Serverless processes may exit before the handshake or batch export finishes. For short tasks that need complete custom hierarchy or deterministic flush, use the manual OTLP mode in the following section.

When the probe already creates LLM Spans automatically, do not call startLlm() again for the same request. If you also need business Spans such as Entry, Agent, Step, and Tool, choose the manual mode below and complete all instrumentation with a single standard OpenTelemetry Provider.

Use the util for full manual instrumentation

The application initializes a standard OpenTelemetry JS TracerProvider, SpanProcessor, and OTLP Exporter, and explicitly passes the provider to ExtendedTelemetryHandler.

In this mode, the application manually creates the required GenAI Spans and waits for forceFlush() and shutdown() before exiting. The rest of this document uses this mode. You will complete the following steps:

1. Install dependencies

2. Configure Resource and content collection policy

3. Initialize OpenTelemetry

4. Create GenAI Spans (Entry, Agent, Step, LLM, Tool, and others)

5. Handle errors and close Spans

6. Verify instrumentation results in the ARMS console

Resource

Set at least a stable service name and mark the GenAI application and instrumentation source for ARMS:

export OTEL_SERVICE_NAME="weather-agent"
export OTEL_RESOURCE_ATTRIBUTES="acs.arms.service.feature=genai_app,gen_ai.instrumentation.sdk.name=loongsuite-genai-utils"

acs.arms.service.feature and gen_ai.instrumentation.sdk.name are Resource Attributes, not regular Span Attributes. If OTEL_RESOURCE_ATTRIBUTES is already set, append with a comma. Do not overwrite existing attributes such as service.namespace and deployment.environment.name.

Message content collection

The environment variables in this section only control message recording by the util. When the application uses the util directly without setting environment variables, messages are processed in NO_CONTENT mode, which does not collect complete message content. Explicitly set the desired mode before starting the process:

export OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="SPAN_ONLY"

Available values:

Message content may contain personal information, business secrets, prompts, and tool parameters. Before enabling this in production environments, complete sanitization, permission, and retention period assessments.

@loongsuite/cms_node_sdk@1.0.4 OpenAI automatic instrumentation uses the probe's own content collection configuration. You cannot use the util environment variables above to infer its behavior. This version records OpenAI request and response content by default. If the business does not allow content to enter traces and you cannot confirm that content collection is turned off on the probe side, switch to manual mode and keep NO_CONTENT.

Context is not activated automatically

startXxx() saves the OTel Context corresponding to the new Span into invocation.contextToken, but does not automatically set it as context.active().

When creating manual child Spans, you must pass the parent invocation's contextToken as the second argument:

handler.startInvokeAgent(agentInv, entryInv.contextToken);
handler.startReactStep(stepInv, agentInv.contextToken);
handler.startLlm(llmInv, stepInv.contextToken);
handler.startExecuteTool(toolInv, stepInv.contextToken);

When calling a model SDK, HTTP client, or database that may have automatic instrumentation, you must also use context.with() to activate the corresponding context:

import { context } from "@opentelemetry/api";

const response = await context.with(
  llmInv.contextToken,
  () => modelClient.chat.completions.create(request),
);

Otherwise, automatically generated SDK/HTTP Spans may become sibling nodes of the LLM Span, or even generate a separate trace.

Public attributes are inherited through Baggage

sessionId, userId on Entry and agentName on Agent are written to OTel Baggage. Subsequent GenAI Spans created with their contextToken as the parent context can inherit:

• gen_ai.session.id

• gen_ai.user.id

• gen_ai.agent.name

Inheritance depends on correct parent Context. When the parent-child relationship is broken, these public attributes may also be missing.

Why this document uses explicit start/stop

The Handler also provides callback-style interfaces such as entry(), invokeAgent(), and llm(), but these callback-style interfaces do not automatically execute callbacks with the new invocation's Context.

Whenever the callback calls auto-instrumented SDKs or needs to create multi-level GenAI Spans, use the explicit startXxx(), context.with(), stopXxx()/failXxx() pattern so that context boundaries are clear.

Model SDK collected by the same OpenTelemetry Provider

This section applies only when automatic instrumentation and the util explicitly share the same @opentelemetry/api TracerProvider. It does not apply to the @loongsuite/cms_node_sdk@1.0.4 scenario described in the probe section.

Do not create a manual LLM Span. Only execute the model call within the Step's Context:

const response = await context.with(
  stepInv.contextToken,
  () => modelClient.chat.completions.create(request),
);

Before release, confirm that a single model request produces only one LLM Span, and confirm that the automatic LLM Span's parentSpanId points to the corresponding Step.

Create an LLM Span manually

import {
  createLLMInvocation,
} from "@loongsuite/otel-util-genai";

const llmInv = createLLMInvocation({
  provider: "dashscope",
  operationName: "chat",
  requestModel: "qwen-plus",
  inputMessages: toGenAIInputMessages(messages),
  toolDefinitions: toGenAIToolDefinitions(tools),
});
handler.startLlm(llmInv, stepInv.contextToken);

try {
  const response = await context.with(
    llmInv.contextToken,
    () => modelClient.chat.completions.create({
      model: "qwen-plus",
      messages,
      tools,
    }),
  );

  const choice = response.choices?.[0];
  if (!choice?.message) {
    throw new Error("The model response has no first choice");
  }

  llmInv.responseId = response.id ?? null;
  llmInv.responseModelName = response.model ?? "qwen-plus";
  llmInv.finishReasons = [
    choice.finish_reason ?? "stop",
  ];
  llmInv.outputMessages = [
    toGenAIOutputMessage(
      choice.message,
      choice.finish_reason,
    ),
  ];

  if (response.usage) {
    llmInv.inputTokens =
      response.usage.prompt_tokens ?? null;
    llmInv.outputTokens =
      response.usage.completion_tokens ?? null;
    llmInv.totalTokens =
      response.usage.total_tokens ?? null;
  }

  handler.stopLlm(llmInv);
} catch (error) {
  handler.failLlm(
    llmInv,
    toSafeGenAIError(
      error,
      "LLMError",
      "LLM request failed",
    ),
  );
  throw error;
}

Do not stop the LLM Span before sending the request. Otherwise, latency, automatic child Spans, and error status will be inaccurate.

Image URL input

The request object sent by the application to the model and the message object written to GenAI Spans use two different schemas. Construct them separately. The following example uses the image URL actually received by the model, so it maps to a Uri Part in the GenAI messages:

const prompt = "Describe this image in one sentence.";
const imageUrl = "https://example.com/image.jpg";

const inputMessages = [{
  role: "user",
  parts: [
    { type: "text", content: prompt },
    {
      type: "uri",
      mimeType: "image/jpeg",
      modality: "image",
      uri: imageUrl,
    },
  ],
}];

const llmInv = createLLMInvocation({
  provider: "dashscope",
  operationName: "chat",
  requestModel: "qwen3-vl-plus",
  inputMessages,
  outputType: "text",
});
handler.startLlm(llmInv, entryInv.contextToken);

try {
  const response = await context.with(
    llmInv.contextToken,
    () => modelClient.chat.completions.create({
      model: "qwen3-vl-plus",
      messages: [{
        role: "user",
        content: [
          { type: "text", text: prompt },
          {
            type: "image_url",
            image_url: { url: imageUrl },
          },
        ],
      }],
    }),
  );

  const choice = response.choices?.[0];
  if (!choice?.message) {
    throw new Error("The model response has no first choice");
  }
  llmInv.responseId = response.id ?? null;
  llmInv.responseModelName =
    response.model ?? "qwen3-vl-plus";
  llmInv.finishReasons = [
    choice.finish_reason ?? "stop",
  ];
  llmInv.inputTokens =
    response.usage?.prompt_tokens ?? null;
  llmInv.outputTokens =
    response.usage?.completion_tokens ?? null;
  llmInv.totalTokens =
    response.usage?.total_tokens ?? null;
  llmInv.outputMessages = [{
    role: "assistant",
    parts: [{
      type: "text",
      content: choice.message.content ?? "",
    }],
    finishReason: choice.finish_reason ?? "stop",
  }];
  handler.stopLlm(llmInv);
} catch (error) {
  handler.failLlm(
    llmInv,
    toSafeGenAIError(
      error,
      "LLMError",
      "Multimodal LLM request failed",
    ),
  );
  throw error;
}

The TypeScript public API uses mimeType. In SPAN_ONLY or SPAN_AND_EVENT mode, 0.1.0 automatically generates:

gen_ai.input.messages
gen_ai.input.multimodal_metadata

Both attributes use the Schema field mime_type for URI data:

[
  {
    "type": "uri",
    "mime_type": "image/jpeg",
    "uri": "https://example.com/image.jpg",
    "modality": "image"
  }
]

Multi-modal metadata only summarizes Uri Parts in the final messages. It does not include Text, BLOB, Base64Blob, or File. The tool only records telemetry data and is not responsible for uploading media, calling models, or converting provider file objects into URIs.

File Part support boundaries

File Part in TypeScript uses:

{
  type: "file",
  mimeType: "application/pdf",
  modality: "document",
  fileId: "file-123",
}

When written to message JSON, fileId is converted to file_id and mimeType is converted to mime_type. However, different model providers have significantly different interfaces for file upload and file ID referencing. This document does not treat File Part as an example that has completed real model end-to-end verification. Only map File Part when the application actually uses a file ID returned by the provider to send a model request. Do not construct fake file IDs just to demonstrate the field.

Offline unit test

Tests should not only assert that Spans are generated, but also check:

• All Spans share the same traceId.

• The Agent's parentSpanId points to Entry.

• The Step's parentSpanId points to Agent.

• LLM and Tool parentSpanId values point to the corresponding Step.

• Automatically generated SDK/HTTP Spans are under the manual LLM or Tool Span.

• gen_ai.session.id, gen_ai.user.id, and gen_ai.agent.name are inherited.

• LLM and Agent token values are correct.

• Both assistant tool call and tool response are retained.

• Multi-modal fields use mime_type / file_id, with no residual camelCase.

• Multi-modal metadata only contains URI Parts.

• All Spans on the error chain are marked as Error.

After downloading the companion Demo, run:

cd nodejs-genai-util-demo
npm ci
npm test
npm run demo

The Demo's package.json should pin the npm versions verified in this document. Customer-facing Demos must not depend on file:../... local source code paths.

Real model verification

export DASHSCOPE_API_KEY="<your-api-key>"
npm run demo:dashscope

This step uses an in-memory Exporter to separately confirm that the model, tool calling, and message conversion are correct. Check that the model actually completed tool calling, and record the traceId from the program output. Do not print API keys in logs.

Tool calling to OTLP combined verification

export OTEL_SERVICE_NAME="weather-agent-validation"
export OTEL_RESOURCE_ATTRIBUTES="service.name=weather-agent-validation"
export OTEL_EXPORTER_OTLP_ENDPOINT="<ARMS endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="<ARMS auth header>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export GENAI_DEMO_ALLOW_CONTENT_EXPORT="true"
npm run demo:e2e

GENAI_DEMO_ALLOW_CONTENT_EXPORT=true is the companion Demo's safety confirmation switch, indicating that the operator confirms the complete input and output in the example can be sent to the configured OTLP backend. It does not automatically sanitize data. Use only public, fictitious, or already sanitized data.

Only after both forceFlush() and shutdown() succeed does the Demo output:

export completed traceId=<trace-id>

A successful export request from the exporter only proves that the client completed the export request. It cannot replace ARMS console verification.

If the console provides OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, use that variable as shown in the console example. Do not rewrite the URL yourself.

Image URL to OTLP combined verification

export DASHSCOPE_API_KEY="<your-api-key>"
export OTEL_SERVICE_NAME="multimodal-agent-validation"
export OTEL_RESOURCE_ATTRIBUTES="service.name=multimodal-agent-validation"
export OTEL_EXPORTER_OTLP_ENDPOINT="<ARMS endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="<ARMS auth header>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export GENAI_DEMO_ALLOW_CONTENT_EXPORT="true"
npm run demo:multimodal-e2e

The image must be a public HTTPS URL without sensitive content and must not contain usernames, passwords, query parameters, fragments, or temporary signing parameters. The complete URI is written to both gen_ai.input.messages and gen_ai.input.multimodal_metadata.

The program should output the actual model response, util.version, model name, response ID, finish reason, tokens, and traceId. The model response must reflect the image content. Receiving only HTTP 200 does not prove that the model processed the image.

Event Log to OTLP combined verification

This use case does not call a model. It only verifies Event Log conversion, input/output multi-modal fields, and OTLP reporting:

export OTEL_SERVICE_NAME="event-log-validation"
export OTEL_RESOURCE_ATTRIBUTES="service.name=event-log-validation"
export OTEL_EXPORTER_OTLP_ENDPOINT="<ARMS endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="<ARMS auth header>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export GENAI_DEMO_ALLOW_CONTENT_EXPORT="true"
npm run demo:event-log-e2e

The program should output span.count=4, input.mime_type=image/png, output.mime_type=image/webp, and traceId. The server should receive ENTRY -> AGENT -> STEP -> LLM, with both gen_ai.input.multimodal_metadata and gen_ai.output.multimodal_metadata present.

ARMS Node.js probe automatic collection verification

This use case is installed separately from the manual util Demo to avoid mixing in a second Provider:

cd nodejs-genai-util-demo/arms-probe-demo
npm ci

export DASHSCOPE_API_KEY="<your-api-key>"
export ARMS_LICENSE="<ARMS License>"
export CMS_SERVICE_NAME="probe-openai-validation"
export ARMS_REGION_ID="cn-hongkong"
# No configuration required for the default workspace
# export ARMS_WORKSPACE="<workspace>"

node -r @loongsuite/cms_node_sdk/register app.js

The companion verification program pins @loongsuite/cms_node_sdk@1.0.4 and openai@5.23.2. By default, it waits 65 seconds for the first configuration handshake of a new service before making real DashScope calls. For services that are already registered, set PROBE_WARMUP_MS=0 to shorten verification.

The server should at minimum show the parent-child relationship of HTTP SERVER -> openai.chat LLM -> model HTTP CLIENT. The openai.chat Span should include:

• gen_ai.span.kind=LLM and gen_ai.operation.name=chat

• Request/response model, response ID, and finish reason

• Input/output/total tokens

• Input and output messages (only when the business allows content recording)

• otel.scope.name=openai, otel.scope.version=1.0.4

• Resource with telemetry.sdk.name=cms_node_sdk and correct service.name

If logs show BatchSpanProcessor: span export failed, HTTP 404/403, or the service cannot be found in ARMS, do not assume instrumentation is successful just because the model returned 200. Confirm that License, region, workspace, and service configuration are consistent. Allow first-time configuration handshake time for new services. For short tasks, switch to manual OTLP mode.

Verify instrumentation results

Use the output traceId to verify each item in the LLM Application Monitoring page under Trace Explorer in the ARMS console:

1. The trace can be retrieved under the correct region, correct application, and correct service.name.

2. The call chain tree matches the business execution order.

3. A single model request has only one LLM Span.

4. Every GenAI Span has the correct gen_ai.span.kind and gen_ai.operation.name.

5. LLM includes gen_ai.request.model, input/output/total tokens.

6. Agent tokens equal the aggregate of its contained model calls.

7. Tool includes name, call ID, arguments, and result.

8. With content collection enabled, the second-round LLM input includes both tool call and tool response.

9. Entry, Agent, Step, LLM, and Tool all have session/user/agent public attributes.

10. All open Spans in the failure case are marked as Error.

11. The image verification gen_ai.input.messages contains type=uri, mime_type, modality, and a real URL.

12. gen_ai.input.multimodal_metadata is a JSON array and only summarizes URI Parts.

13. Event Log verification gen_ai.output.messages and gen_ai.output.multimodal_metadata contain output URIs and use mime_type.

14. No residual mimeType / fileId in message JSON and metadata.

15. otel.scope.version matches the actually installed npm version.

16. Resource contains service.name, acs.arms.service.feature=genai_app, and gen_ai.instrumentation.sdk.name=loongsuite-genai-utils.

The website example maintainer should also verify separately in clean directories on both Node.js 20 and 22 with the same npm versions. Examples must not be marked as verified when server-side querying is not completed, local source code is still used, or versions are inconsistent.

No data in ARMS at all

• Verify that the endpoint, region, and authentication header are from the same application.

• Verify service.name.

• Confirm you are using an OTLP Exporter, not ConsoleSpanExporter.

• Confirm that forceFlush() and shutdown() are called before the process exits.

• Check the Exporter error log and network egress.

When using the ARMS Node.js probe, also check ARMS_LICENSE, ARMS_REGION_ID, ARMS_WORKSPACE, and CMS_SERVICE_NAME. Allow configuration handshake time for the first startup of a new service. If the model call succeeds but exporter 404/403 appears, it is still a reporting failure.

No LLM Span under ARMS probe

• Confirm that Node.js startup arguments include -r @loongsuite/cms_node_sdk/register and that the probe loads before openai.

• @loongsuite/cms_node_sdk@1.0.4 only declares support for openai >=4 <6. OpenAI 6 is not within the verified range.

• Check whether the startup log lists openai instrumentation.

• Confirm with ARMS server-side data, not just by looking at the model response.

util does not export Spans

@loongsuite/cms_node_sdk@1.0.4 does not register the standard @opentelemetry/api Provider and cannot be directly reused by the default Handler. Do not mix the two approaches. When you need the util, initialize a standard OpenTelemetry Provider and OTLP Exporter according to the initialization section.

Two LLM Spans for the same model call

This usually means the model SDK is already collected by automatic instrumentation while the business also calls startLlm(). Keep only one type of LLM instrumentation within the same Provider. @loongsuite/cms_node_sdk@1.0.4 and the util should be fully separated according to the instrumentation mode section.

Spans are in the same trace, but the hierarchy is wrong

• Check whether manual child Spans explicitly pass the parent contextToken.

• Check whether auto-instrumented operations execute within context.with().

• Check whether the code crosses custom async boundaries that do not correctly propagate the AsyncLocalStorage Context.

Child Spans are missing session, user, or agent

Check whether Entry/Agent has the corresponding fields set, and confirm that child Spans use the parent invocation's contextToken.

Input and output messages are not visible

Confirm that environment variables are set before the Node.js process starts:

OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY

Then check whether the message conversion result conforms to the { role, parts } structure.

Image messages present but no multi-modal metadata

• Confirm the installed version is the verified release 0.1.0 from this document. When using subsequent versions, re-run the complete verification.

• Confirm the content collection mode is SPAN_ONLY or SPAN_AND_EVENT.

• Metadata only summarizes type: "uri" Parts, not BLOB, Base64Blob, or File.

• When calling the TypeScript API directly, check mimeType, modality, and uri.

• For Event Log input, check the Schema fields mime_type, modality, and uri.

• Check whether invocation.attributes explicitly overwrites automatically generated attributes.

File Part does not appear in multi-modal metadata

This is expected behavior. File Part is retained in message JSON and serializes fileId as file_id, but URI metadata only summarizes Uri Parts. Only record File Part when the provider's actual request uses a file ID.

Peer dependency errors during installation

Check whether the project mixes OpenTelemetry SDK 1.x and 2.x. The verified combination in this document is API 1.9.1, trace/resources 1.30.1, and OTLP exporter 0.57.2.

Intermittent Span loss in short tasks

In manual OTLP mode, do not rely on natural process exit. CLI scripts, Serverless handlers, and test programs must all explicitly wait for forceFlush()/shutdown().

@loongsuite/cms_node_sdk/register signal handling executes shutdown on SIGINT/SIGTERM, but new services still need the first configuration handshake. For extremely short tasks, prefer the manual OTLP mode where you can explicitly control the lifecycle.