Best practices for custom instrumentation of Go LLM applications

Event emission (Events)

util-genai supports sending GenAI events via the OTel Logs API. When StopLLM / StopInvokeAgent is called, the Handler automatically emits an event to the LoggerProvider (event name gen_ai.client.inference.operation.details or gen_ai.client.agent.invoke.operation.details) if all of the following conditions are met:

• Environment variable OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental is set.

• Environment variable OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=true is set.

• A LoggerProvider was injected via WithLoggerProvider(lp) when creating the Handler.

Message content is included in the event body only when OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT includes EVENT (i.e., EVENT_ONLY or SPAN_AND_EVENT).

export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
export OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=true
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY

import (
    logapi "go.opentelemetry.io/otel/log"
    sdklog "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/otel/exporters/stdout/stdoutlog"
)

logExporter, _ := stdoutlog.New()
lp := sdklog.NewLoggerProvider(sdklog.WithProcessor(
    sdklog.NewBatchProcessor(logExporter),
))
defer lp.Shutdown(ctx)

handler := utilgenai.NewTelemetryHandler(
    utilgenai.WithTracerProvider(tp),
    utilgenai.WithLoggerProvider(lp),
)

After setup, use StartLLM / StopLLM and StartInvokeAgent / StopInvokeAgent as usual. Events are emitted automatically without additional code.

Content offloading (CompletionHook)

When LLM prompt/response payloads are large (e.g., tens of KBs of multi-turn conversation context), writing them directly to Span Attributes can make Spans excessively large. util-genai provides a CompletionHook interface that, during StopLLM, automatically offloads large payloads asynchronously to external storage (file system or custom backend). Only a content-addressed reference URI is written to the Span (gen_ai.input.messages_ref / gen_ai.output.messages_ref, etc.).

The built-in UploadCompletionHook uses SHA-256 for file name computation to achieve deduplication (identical payloads are not written twice). It supports JSON and JSONL serialization formats, and internally uses a worker pool with a bounded FIFO set for asynchronous concurrent uploads.

# Configure external storage path and format
export OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH=/data/genai-traces
export OTEL_INSTRUMENTATION_GENAI_UPLOAD_FORMAT=json
export OTEL_INSTRUMENTATION_GENAI_UPLOAD_MAX_QUEUE_SIZE=128

// Use built-in FS Uploader (you can also implement a custom Uploader interface to write to OSS, etc.)
hook := utilgenai.NewUploadCompletionHook(
    utilgenai.WithUploader(utilgenai.NewFSUploader("/data/genai-traces")),
    utilgenai.WithUploadFormat(utilgenai.UploadFormatJSON),
    utilgenai.WithUploadQueueSize(128),
)

handler := utilgenai.NewTelemetryHandler(
    utilgenai.WithTracerProvider(tp),
    utilgenai.WithCompletionHook(hook),
)
defer handler.Shutdown(ctx) // Gracefully drain pending uploads in the queue

After StopLLM completes, the Handler internally calls offloadLLMContent to serialize InputMessages, OutputMessages, SystemInstruction, and ToolDefinitions and submit them to the CompletionHook. The Hook computes SHA-256, writes to the file system (e.g., /data/genai-traces/<sha256>.json), and writes the reference attribute gen_ai.input.messages_ref=sha256://<hash> on the Span.

Custom Uploader interface:

type Uploader interface {
    Upload(ctx context.Context, ref string, data []byte) error
}

Implement this interface to write payloads to OSS, S3, or other object storage services.

Reasoning message type

util-genai includes a Reasoning message part type for carrying the model's reasoning/thinking process (e.g., Chain-of-Thought). Use it within OutputMessage.Parts:

invocation.OutputMessages = []utilgenai.OutputMessage{{
    Role: "assistant",
    Parts: []utilgenai.MessagePart{
        utilgenai.Reasoning{Content: "Let me analyze this problem... First..."},
        utilgenai.Text{Content: "The final answer is..."},
    },
    FinishReason: utilgenai.FinishReasonStop,
}}
// Reasoning tokens can be tracked separately via ReasoningOutputTokens
reasoningTokens := 150
invocation.ReasoningOutputTokens = &reasoningTokens

The corresponding Span attribute is gen_ai.usage.reasoning.output_tokens, allowing you to distinguish between reasoning tokens and regular output tokens in the console.

Entry Span details

In the Entry Span details panel, key attributes include gen_ai.session.id (unique session identifier) and gen_ai.user.id (user identifier). Once set at the function entry point, these propagate automatically to downstream child Spans (provided they are created from the same Context). You can also view gen_ai.input.messages (complete user input message content) and gen_ai.output.messages (complete model output message content).

Agent Span details

The Agent Span panel displays the Agent name and description, along with Agent-level token usage aggregation. Example: gen_ai.agent.name=TechContentAgent, gen_ai.provider.name=dashscope, gen_ai.request.model=qwen-plus, gen_ai.usage.input_tokens=3982, gen_ai.usage.output_tokens=884.

Tool Span details

The Tool Span displays detailed tool call information. For example, for execute_tool generate_seo_keywords, the details panel shows the tool name (gen_ai.tool.name=generate_seo_keywords), tool type (gen_ai.tool.type=function), tool call arguments (gen_ai.tool.call.arguments), and tool return result (gen_ai.tool.call.result), making it easy to verify whether tool input/output matches expectations.

LLM Span details

LLM Spans can be automatically collected by ARMS probes for supported Go components (such as sashabaranov/go-openai, dashscope, etc.), or manually created using LLMInvocation as shown in this topic. Auto-collection is suitable for common text conversation model calls. For ASR, TTS, or other model calls not yet automatically covered by probes or that require additional business attributes, manually create LLM Spans and write gen_ai.asr.* / gen_ai.tts.* custom attributes.

The LLM Span details panel displays basic information such as application name, interface name, IP, start and end times, SpanID, ParentSpanID, and status code. The Additional Info tab shows the response output content, custom Attributes, and token usage for that call.

Custom Span details

Custom Spans (such as duplicate_tool_detection, response_loop_detection) must be viewed in the full view of the console trace view. The panel shows the gen_ai.loop_detection.* attributes, useful for analyzing business-side loops, retries, and other abnormal behaviors.

main.go

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "os"
    "strings"
    "time"

    utilgenai "github.com/alibaba/loongsuite-go/util-genai"
    openai "github.com/sashabaranov/go-openai"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
    "go.opentelemetry.io/otel/trace"
)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
    exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
    if err != nil {
        return nil, err
    }
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceNameKey.String("techcontent-agent"),
            semconv.ServiceVersionKey.String("0.1.0"),
            // Critical Resource attributes: do not write via Span Attribute
            attribute.String("acs.arms.service.feature", "genai_app"),
            attribute.String("gen_ai.instrumentation.sdk.name", "loongsuite-genai-utils"),
        ),
    )
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

func main() {
    ctx := context.Background()
    tp, err := initTracer(ctx)
    if err != nil {
        log.Fatalf("init tracer: %v", err)
    }
    defer tp.Shutdown(ctx)

    apiKey := os.Getenv("DASHSCOPE_API_KEY")
    if apiKey == "" {
        log.Fatal("DASHSCOPE_API_KEY is required")
    }

    cfg := openai.DefaultConfig(apiKey)
    cfg.BaseURL = "https://dashscope.aliyuncs.com/compatible-mode/v1"
    client := openai.NewClientWithConfig(cfg)

    handler := utilgenai.NewTelemetryHandler(utilgenai.WithTracerProvider(tp))
    tracer := otel.Tracer("techcontent-agent")

    if err := runAgent(ctx, client, handler, tracer, "CMS 2.0 AI alert noise reduction"); err != nil {
        log.Fatalf("run agent: %v", err)
    }
}

func runAgent(
    ctx context.Context,
    client *openai.Client,
    handler *utilgenai.TelemetryHandler,
    tracer trace.Tracer,
    topic string,
) error {
    // ---------- Entry Span ----------
    sessionID := fmt.Sprintf("sess-%d", time.Now().UnixNano())
    ctx, entrySpan := tracer.Start(ctx, "enter",
        trace.WithSpanKind(trace.SpanKindServer),
        trace.WithAttributes(
            attribute.String("gen_ai.span.kind", "ENTRY"),
            attribute.String("gen_ai.operation.name", "enter"),
            attribute.String("gen_ai.session.id", sessionID),
            attribute.String("gen_ai.user.id", "anonymous"),
        ),
    )
    defer entrySpan.End()

    inputMsgs := []utilgenai.InputMessage{
        {Role: "user", Parts: []utilgenai.MessagePart{utilgenai.Text{Content: topic}}},
    }
    entrySpan.SetAttributes(
        attribute.String("gen_ai.input.messages", utilgenai.InputMessagesToJSON(inputMsgs)),
    )

    // ---------- Agent Span ----------
    agentInv := utilgenai.NewInvokeAgentInvocation()
    agentInv.Provider = "dashscope"
    agentInv.AgentName = "TechContentAgent"
    agentInv.AgentDescription = "Technical content generation assistant"
    agentInv.RequestModel = "qwen-plus"
    agentInv.ConversationID = sessionID

    ctx = handler.StartInvokeAgent(ctx, agentInv)

    messages := []openai.ChatCompletionMessage{
        {Role: openai.ChatMessageRoleSystem, Content: "You are an experienced cloud-native technical editor who uses tools to gather material before writing."},
        {Role: openai.ChatMessageRoleUser, Content: topic},
    }

    var (
        totalIn, totalOut int
        finalContent     string
        toolCounter      = map[string]int{}
        prevContent      string
    )

    // ---------- ReAct Loop ----------
    for round := 0; round < 5; round++ {
        checkDuplicateTools(ctx, tracer, toolCounter, &messages)

        stepCtx, stepSpan := tracer.Start(ctx, "react",
            trace.WithSpanKind(trace.SpanKindInternal),
            trace.WithAttributes(
                attribute.String("gen_ai.span.kind", "STEP"),
                attribute.String("gen_ai.operation.name", "react"),
                attribute.Int("gen_ai.step.round", round+1),
            ),
        )

        // Manual LLM Span (can be omitted if ARMS Go auto-instrumentation is enabled)
        llmInv := utilgenai.NewLLMInvocation("qwen-plus")
        llmInv.Provider = "dashscope"
        llmInv.OperationName = utilgenai.OperationChat
        llmInv.ConversationID = sessionID
        llmInv.InputMessages = openAIToUtilMessages(messages)

        stepCtx = handler.StartLLM(stepCtx, llmInv)
        resp, err := client.CreateChatCompletion(stepCtx, openai.ChatCompletionRequest{
            Model:    "qwen-plus",
            Messages: messages,
            Tools:    toolDefinitions(),
        })
        if err != nil {
            handler.FailLLM(llmInv, &utilgenai.Error{Message: err.Error(), Type: "APIError"})
            stepSpan.RecordError(err)
            stepSpan.End()
            handler.FailInvokeAgent(agentInv, &utilgenai.Error{Message: err.Error(), Type: "APIError"})
            return err
        }

        choice := resp.Choices[0]
        llmInv.ResponseID = resp.ID
        llmInv.ResponseModelName = resp.Model
        pt := resp.Usage.PromptTokens
        ct := resp.Usage.CompletionTokens
        llmInv.InputTokens = &pt
        llmInv.OutputTokens = &ct
        totalIn += pt
        totalOut += ct

        llmInv.OutputMessages = []utilgenai.OutputMessage{{
            Role:         "assistant",
            Parts:        []utilgenai.MessagePart{utilgenai.Text{Content: choice.Message.Content}},
            FinishReason: utilgenai.FinishReason(choice.FinishReason),
        }}
        handler.StopLLM(llmInv)

        messages = append(messages, choice.Message)

        // ---------- Tool Loop ----------
        for _, tc := range choice.Message.ToolCalls {
            toolCounter[tc.Function.Name]++
            toolInv := utilgenai.NewExecuteToolInvocation(tc.Function.Name)
            toolInv.ToolCallID = tc.ID
            toolInv.ToolType = "function"

            _ = json.Unmarshal([]byte(tc.Function.Arguments), &toolInv.Input)

            stepCtx = handler.StartExecuteTool(stepCtx, toolInv)
            result, err := dispatchTool(tc.Function.Name, tc.Function.Arguments)
            if err != nil {
                handler.FailExecuteTool(toolInv, &utilgenai.Error{Message: err.Error(), Type: "ToolError"})
                stepSpan.End()
                handler.FailInvokeAgent(agentInv, &utilgenai.Error{Message: err.Error(), Type: "ToolError"})
                return err
            }
            toolInv.Output = result
            handler.StopExecuteTool(toolInv)

            messages = append(messages, openai.ChatCompletionMessage{
                Role:       openai.ChatMessageRoleTool,
                Content:    result,
                ToolCallID: tc.ID,
                Name:       tc.Function.Name,
            })
        }

        // ReAct termination check
        finishReason := "continue"
        if len(choice.Message.ToolCalls) == 0 {
            finishReason = "stop"
            finalContent = choice.Message.Content
            stepSpan.SetAttributes(attribute.String("gen_ai.response.finish_reason", finishReason))
            stepSpan.End()

            if checkResponseLoop(ctx, tracer, finalContent, prevContent) {
                agentInv.Attributes = map[string]any{
                    "gen_ai.response.finish_reason": "loop_detected",
                }
                break
            }
            break
        }
        stepSpan.SetAttributes(attribute.String("gen_ai.response.finish_reason", finishReason))
        stepSpan.End()
        prevContent = choice.Message.Content
    }

    // Aggregate token usage and end Agent Span
    agentInv.InputTokens = &totalIn
    agentInv.OutputTokens = &totalOut
    handler.StopInvokeAgent(agentInv)

    // Entry output
    outputMsgs := []utilgenai.OutputMessage{{
        Role:         "assistant",
        Parts:        []utilgenai.MessagePart{utilgenai.Text{Content: finalContent}},
        FinishReason: utilgenai.FinishReasonStop,
    }}
    entrySpan.SetAttributes(
        attribute.String("gen_ai.output.messages", utilgenai.OutputMessagesToJSON(outputMsgs)),
    )

    fmt.Println(strings.Repeat("-", 60))
    fmt.Println(finalContent)
    return nil
}

tools.go

package main

import (
    "encoding/json"
    "fmt"

    openai "github.com/sashabaranov/go-openai"
)

func toolDefinitions() []openai.Tool {
    return []openai.Tool{
        {
            Type: openai.ToolTypeFunction,
            Function: &openai.FunctionDefinition{
                Name:        "generate_seo_keywords",
                Description: "Generate SEO keywords based on topic",
                Parameters: map[string]any{
                    "type": "object",
                    "properties": map[string]any{
                        "topic": map[string]any{"type": "string"},
                    },
                    "required": []string{"topic"},
                },
            },
        },
        {
            Type: openai.ToolTypeFunction,
            Function: &openai.FunctionDefinition{
                Name:        "outline_article",
                Description: "Generate article outline",
                Parameters: map[string]any{
                    "type": "object",
                    "properties": map[string]any{
                        "topic":    map[string]any{"type": "string"},
                        "keywords": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
                    },
                    "required": []string{"topic"},
                },
            },
        },
    }
}

func dispatchTool(name, argJSON string) (string, error) {
    var args map[string]any
    _ = json.Unmarshal([]byte(argJSON), &args)
    switch name {
    case "generate_seo_keywords":
        topic, _ := args["topic"].(string)
        return fmt.Sprintf(`["%s best practices", "%s case study", "%s implementation"]`, topic, topic, topic), nil
    case "outline_article":
        topic, _ := args["topic"].(string)
        return fmt.Sprintf(`## %s Outline\n1. Background\n2. Technical Solution\n3. Results`, topic), nil
    default:
        return "", fmt.Errorf("unknown tool: %s", name)
    }
}

messages.go

package main

import (
    utilgenai "github.com/alibaba/loongsuite-go/util-genai"
    openai "github.com/sashabaranov/go-openai"
)

func openAIToUtilMessages(msgs []openai.ChatCompletionMessage) []utilgenai.InputMessage {
    out := make([]utilgenai.InputMessage, 0, len(msgs))
    for _, m := range msgs {
        out = append(out, utilgenai.InputMessage{
            Role:  m.Role,
            Parts: []utilgenai.MessagePart{utilgenai.Text{Content: m.Content}},
        })
    }
    return out
}

go.mod key dependencies

module github.com/example/techcontent-agent

go 1.24.0

require (
    github.com/alibaba/loongsuite-go/util-genai v0.1.0
    github.com/sashabaranov/go-openai v1.36.1
    go.opentelemetry.io/otel v1.40.0
    go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.40.0
    go.opentelemetry.io/otel/log v0.16.0
    go.opentelemetry.io/otel/sdk v1.40.0
    go.opentelemetry.io/otel/trace v1.40.0
)