Code Evaluator

Unlike the Agent evaluator and the LLM evaluator, which rely on model inference, a Code evaluator has no reasoning overhead and consumes no model calls. When a quality standard can be expressed in code, a Code evaluator is often a better fit, and it is well suited to the pass-or-fail checks in a regression safety net.

Use cases

Typical scenarios for a Code evaluator:

• Structured output validation — The most common use. Require the agent to return valid JSON that contains specified fields, and score 0 whenever a field is missing.

• Numeric answer comparison — Compare amounts, quantities, or calculation results. Define a tolerance range in code instead of asking a model to sense whether a value is close enough.

• Denylist scanning — Scan the output for sensitive words and prohibited phrasing.

• Output length and format checks — Check the output length, whether the output contains a Markdown table, or whether it starts with a specified prefix.

• Tool calling sequence verification — Verify that the required tools were called in the expected order.

• Citation completeness checks — Verify that every citation number in the answer can be found in the retrieved passages.

If your quality standard depends on subjective or semantic judgment that cannot be expressed in code, use the Agent evaluator or the LLM evaluator, which rely on model inference.

Prerequisites

Code Evaluator is currently available in the following regions:

Create an evaluator

1. Log on to the AgentLoop console.

2. In the left-side navigation pane, click Evaluation.

3. In the tab navigation bar at the top of the page, click the Evaluator tab.

4. In the upper-right corner, click Create Evaluator.

On the creation form, fill in the basic information. Set Evaluator Type to Code. Display name, description, and tags are optional. Tags let you retrieve evaluators by dimension in the evaluator list, so give your Code evaluators a consistent tag that is easy to filter on.

After you complete the form, click Create Evaluator to submit it. The edit page opens with two tabs: Basic Information and Evaluation Code. On the Basic Information tab, the evaluator name and evaluator type are read-only, and the remaining fields stay editable. The core configuration happens on the Evaluation Code tab.

Write the evaluation code

In the upper-right corner of the Evaluation Code tab, switch between Python and JavaScript. The two languages are mutually exclusive, and the file name in the editor changes to code.py or code.js accordingly. Click Reset to Template to load the official default template as a starting point.

Your evaluation code runs in a sandbox. The following constraints apply:

• Dependencies — Implement the evaluation logic with the language standard library only.

• Determinism — Keep the code in the form of a pure function: the same input produces the same result.

• Isolation — The code cannot access external services or depend on the state of the execution environment.

This is both a practical constraint of the sandbox and the precondition for evaluation results that are reproducible and trustworthy.

The only contract for your code is this: define a function named evaluate that takes data fields as input parameters and returns an object that contains score and explanation.

The Python default template is as follows.

def evaluate(input, output, context=None):
    # Evaluate one record and return score and explanation.
    return {
        "score": 0,
        "explanation": "Describe why this score was assigned",
    }

The JavaScript default template is as follows.

function evaluate(input, output, context) {
  // Evaluate one record and return score and explanation.
  return {
    score: 0,
    explanation: 'Describe why this score was assigned',
  }
}
module.exports = evaluate

Code samples

def evaluate(input, output, context=None):
    import json

    text = output if isinstance(output, str) else json.dumps(output, ensure_ascii=False)

    try:
        data = json.loads(text)
    except Exception as e:
        return {"score": 0, "explanation": "Output is not valid JSON: %s" % e}

    if not isinstance(data, dict):
        return {"score": 0, "explanation": "The top level of the output JSON is not an object"}

    missing = [k for k in ("title", "items") if k not in data]
    if missing:
        return {"score": 0, "explanation": "Missing required fields: %s" % ", ".join(missing)}

    if not isinstance(data["items"], list) or len(data["items"]) == 0:
        return {"score": 0, "explanation": "items is not an array, or it is an empty array"}

    return {"score": 1, "explanation": "JSON is valid and contains title and %d items" % len(data["items"])}

def evaluate(output, expected, tolerance=0.001):
    import re

    def to_number(v):
        if isinstance(v, (int, float)):
            return float(v)
        m = re.search(r"-?\d+(?:\.\d+)?", str(v).replace(",", ""))
        return float(m.group()) if m else None

    got, want = to_number(output), to_number(expected)

    if want is None:
        return {"score": 0, "explanation": "No number found in the ground truth"}
    if got is None:
        return {"score": 0, "explanation": "No number found in the model output"}

    denominator = abs(want) if want != 0 else 1.0
    diff = abs(got - want) / denominator

    if diff <= tolerance:
        return {"score": 1, "explanation": "Numbers match: expected %g, actual %g" % (want, got)}
    return {"score": 0, "explanation": "Deviation exceeds the limit: expected %g, actual %g, relative error %.2f%%" % (want, got, diff * 100)}

def evaluate(output, context=None, required_tools="search,summarize"):
    import json

    expected = [t.strip() for t in str(required_tools).split(",") if t.strip()]
    if not expected:
        return {"score": 1, "explanation": "No required tools declared, so the record passes by default"}

    raw = context if isinstance(context, str) else json.dumps(context or {}, ensure_ascii=False)

    called = set()
    for name in expected:
        if name in raw:
            called.add(name)

    missing = [t for t in expected if t not in called]
    hit_rate = len(called) / len(expected)

    if not missing:
        return {"score": 1, "explanation": "All required tools were called: %s" % ", ".join(expected)}
    return {
        "score": round(hit_rate, 2),
        "explanation": "Missing tool calls: %s (%d of %d matched)" % (", ".join(missing), len(called), len(expected)),
    }

function evaluate(input, output, context = null) {
  const text = typeof output === 'string' ? output : JSON.stringify(output)
  const banned = ['guaranteed profit', 'guaranteed returns', 'one hundred percent', 'absolutely safe']

  let score = 100
  const reasons = []

  const hits = banned.filter((w) => text.includes(w))
  if (hits.length > 0) {
    score -= 40 * hits.length
    reasons.push(`Prohibited phrasing matched: ${hits.join(', ')}`)
  }

  if (text.trim().length < 20) {
    score -= 30
    reasons.push(`Response too short: only ${text.trim().length} characters`)
  }

  if (!/risk disclaimer|for reference only/.test(text)) {
    score -= 20
    reasons.push('Missing risk disclaimer')
  }

  score = Math.max(0, score)

  return {
    score,
    explanation: reasons.length === 0 ? 'Passed all quality checks' : reasons.join('; '),
  }
}
module.exports = evaluate

Use the evaluator in an evaluation task

After you save the evaluator, apply it to real data through an evaluation task. On the Evaluation Task tab of the Evaluation module, click Create Task. The flow has two steps.

Step 1: Data Configuration

Enter the task name and description, and select the data source. Supported data sources are traces, agent trajectories, logs, and datasets. If you select traces, also specify the evaluation granularity: single-turn conversation or multi-turn conversation. Choose whether to include the input and output of previous turns, and narrow the evaluation scope with the Agent drop-down list and a data filter expression.

Step 2: Select Evaluator

This is the critical step. The Preview Data area at the top of the page shows all available fields of one real sample. Click Switch Sample to load another sample and confirm the shape of your data. In the evaluator list on the left, switch to the Custom tab and select the Code evaluator you just created. The Field Mapping area on the right lists all variables of that evaluator. Select the corresponding data field for each variable.

The fields available for mapping depend on the data source. For a trace data source, the following fields are available for mapping.

After you map all required variables, the evaluator card shows the Ready tag.

Run a preview test and view the results

Run a preview test before the formal run. Click Run Test at the bottom of the page. A drawer opens on the right and shows the field values of the current sample. Click Run Test in the drawer to trial-run the selected evaluators in batch. After execution finishes, the Evaluation Results area shows the status, score, and evaluation rationale of each evaluator. This step exposes most problems before the formal run consumes quota, such as a wrong field mapping, code that throws an exception on the real data shape, or a score that does not match your intended convention.

After you confirm the results, click Create and Run to submit the task. The task details page has three tabs: Overview, Run Records, and Evaluation Results.

• Run Records — Shows the total count, completion rate, succeeded and failed counts, duration, and average score of each execution.

• Evaluation Results — Provides a detail view and a data item view. The detail view lists the following information record by record: the data item content, score, score range, normalized score, status, evaluator, evaluation duration, and generation time. You can filter the results by status, evaluator, score range, and time window.

The execution duration of a single Code evaluator record is usually a little over ten seconds. Records within one task use concurrent execution, so a task of about 100 records takes roughly one minute in total.

Authoring recommendations

Apply the following practices to your evaluation code:

• Normalize the output type before matching — The actual type of output is not always a string. trace.output in trace data can be a list or a dictionary. Normalize the type before you run a regular expression or a substring match. In Python, use json.dumps(output, ensure_ascii=False) if not isinstance(output, str) else output. In JavaScript, use typeof output === 'string' ? output : JSON.stringify(output).

• Turn exceptions into explicit low scores — Catch every operation that can raise an exception and turn it into an explicit low score with an explanation, instead of letting the function crash. JSON parsing failures, type conversion failures, and out-of-range indexes are normal in real data. Expressing them as a zero score plus a reason is far more valuable than one execution-failure record.

• Externalize thresholds as optional parameters — Judgment thresholds such as tolerance, the required tool list, and the minimum length become optional variables when you write them as function parameters with default values. You can then override them per task, and one piece of code can serve multiple evaluation tasks. There is no need to clone a new evaluator every time you tune a parameter.

What's next

Use the evaluator name to reference this evaluator in API calls.