Custom instruction practices

Write clear and accurate tool descriptions

• Goal: Help the intent model understand what the tool does.

• Note: Avoid vague language. Ensure the description matches the tool’s actual function.

Example:

• Good: "Query the weather for a specific date"

• Bad: "Check weather" (Too broad. It might be mistaken for other weather-related operations.)

Define parameters without ambiguity

• Goal: Clearly define the purpose and value range for each parameter.

• Recommendation: Use enumerations to limit parameter options whenever possible. Enumeration definition is not yet available, so define them in the description. For open-ended inputs, provide a detailed description.

Example — Define a tool to play music:

{
  "type": "object",
  "required": ["action", "media"],
  "properties": {
    "action": {
      "type": "string",
      "description": "Playback control action",
      "enum": ["play", "pause", "stop"]
    },
    "media": {
      "type": "string",
      "description": "Media file name or URL"
    }
  }
}

Be concise but comprehensive

• Goal: Keep tool definitions simple but complete.

• Note: Do not list too many examples. Include only necessary explanations.

Example — Tool definition for a query:

{
  "name": "query_city_news",
  "description": "Query news for a specific city",
  "parameters": {
    "type": "object",
    "required": ["city"],
    "properties": {
      "city": {
        "type": "string",
        "description": "The name of the city to query"
      },
      "date": {
        "type": "string",
        "description": "The date to query, in yyyy-mm-dd format. If empty, queries the current date."
      }
    }
  }
}

Merge similar parameters logically

When designing tool calling interfaces, merging similar parameters means combining multiple parameters that have similar functions or can be processed together into a single parameter. This simplifies the interface definition, reduces redundancy, and makes tool calls more intuitive. The following examples show how to merge parameters logically:

Human-readable time format

{
        "humanReadableDate": {
            "title": "Human-readable date",
            "type": "string",
            "x-type": "humanReadableTime",
            "description": "Can be an absolute, relative, or standard date. The system automatically parses it into yyyy-mm-dd format.",
            "examples": [
                "Tomorrow",
                "Next Tuesday",
                "Three days later"
            ]
        },
        "humanReadableTime": {
            "title": "Human-readable time",
            "type": "string",
            "x-type": "humanReadableTime",
            "description": "Can be an absolute or relative time described in natural language. The system automatically parses it into HH:MM:SS format.",
            "examples": [
                "In three hours",
                "5 PM today",
                "12:15"
            ]
        }
    }

If a parameter name starts with `date_`, the system interprets it as a humanReadableDate.

If a parameter name starts with `time_`, the system interprets it as a humanReadableTime.

parameters internal structure

• type: Must be "object".

• properties: Defines the specific key-value pairs. Each key represents a parameter.

• required: A string array that lists the names of all required parameters.

Meta-definition for the tool list

{
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string", "pattern": "^[a-zA-Z0-9_-]+$" },
          "description": { "type": "string", "minLength": 1 },
          "parameters": { "type": "object" }
        },
        "required": ["name", "description"],
        "additionalProperties": false
      }
}

Console configuration

To provide a model response while sending an instruction, select a reply mode on the Settings page:

• Reply based on execution status (default): After sending the instruction, the system waits for a success or failure message from the device. The model then generates a corresponding reply, such as "The air conditioner has been set to 26 degrees" or "The volume is already at maximum and cannot be increased further."

• Auto-reply: Automatically outputs a reply when the instruction is sent.

Device-side instruction acceptance

For example, if the unmute skill is configured to require a reply.

The result of sending the "unmute" voice command is:

{
    "payload": {
        "output": {
            "event": "RespondingContent",
            "text": "",
            "spoken": "",
            "finished": true,
            "finish_reason": "stop",
            "extra_info": {
                "tool_infos": [

                ],
                "agent_info": {
                    "intent_infos": [
                        {
                            "domain": "general_command",
                            "intent": "unmute"
                        }
                    ],
                    "round": 1
                },
                "commands": "[{\"intent_info\":{\"domain\":\"general_command\",\"intent\":\"unmute\"},\"command_request_id\":\"35b635f3-6511-450e-8fa1-6955d5279367\",\"name\":\"unmute\",\"params\":[]}]",
                "query": "unmute"
            },
            "dialog_id": "56fe9300-d80c-471a-80bd-20c0296acd93",
            "round_id": "70f636e1d0284bc8a8bac2767e61b4b5",
            "llm_request_id": "1d8f3469274e4d13bbc3b28460eabb76"
        }
    }
}

In the result, `payload.output.extra_info.commands` is the instruction execution result.

When you select "Reply based on execution status", the result includes a `command_request_id`. Use this ID as the key when replying with the result.

Device-side instruction reply

After you receive an instruction or an Agent result, execute it on the device. You obtain a success or failure result. Sync this result with the server-side by submitting it through the following interface:

`invoke_result.content` is the data part passed from the device to the Agent. We recommend using the following unified device-side return format:

{
  "type": "text",
  "text": "Tool result text"
}

{
  "type": "image",
  "data": "base64-encoded-data",
  "mimeType": "image/png"
  "annotations": {
    "audience": ["user"],
    "priority": 0.9
  }
}

{
  "type": "audio",
  "data": "base64-encoded-audio-data",
  "mimeType": "audio/wav"
}

[
    {
        "command_request_id": "xxxx",
        "invoke_result": {
            "content": {
                "type": "text",
                "text": "Successfully opened"
            },
            "structuredContent": {
                "success": true
            }
        }
    }
]

Request code example - Device-side callback for execution result (Java)

public static HashMap<String,Object> getAgentOrCommandResult() {

  HashMap<String,Object> commandResult = new HashMap<>();
  commandResult.put("command_request_id", "35b635f3-6511-450e-8fa1-6955d5279367");
  commandResult.put("invoke_result", new Object<>());
    // invoke_result.structuredContent.success = true/false execution result

  ArrayList<Object> commandResults = new ArrayList<>();
  commandResults.add(commandResult);

  HashMap<String,Object> passThroughParams = new HashMap<>();
  passThroughParams.put("command_results", commandResults);
  return passThroughParams;
}
 
// Initiate a request to open the agent
MultiModalRequestParam.UpdateParams updateParams = MultiModalRequestParam.UpdateParams.builder()
    .bizParams(MultiModalRequestParam.BizParams.builder()
        .passThroughParams(getAgentOrCommandResult())
        .build())
    .build();
conversation.requestToRespond("prompt", "", updateParams);