Server-side Go SDK

1. NewMultiModalDialog

Creates an interactive session and sets up callbacks.

// Creates a voice dialog session.
//
// This method initializes a new session and sets the necessary parameters
// to prepare for interaction with the model.
//
// Parameters:
//   - requestParams: The collection of request parameters.
//   - multimodalCallback: The callback object for handling messages from the server.
//   - appID: The application ID created in the console.
//   - workspaceID: The customer's Workspace ID (optional).
//   - url: The API URL (optional, defaults to wss://dashscope.aliyuncs.com/api-ws/v1/inference).
//   - apiKey: The unique key for application access.
//   - dialogID: The dialog ID. If provided, the conversation continues from the specified context (optional).
//   - model: The model to use (optional).
func NewMultiModalDialog(
    requestParams *RequestParameters,
    multimodalCallback MultiModalCallback,
    appID string,
    workspaceID string,
    url string,
    apiKey string,
    dialogID string,
    model string,
) (*MultiModalDialog, error)

2. Start

Starts the dialog service and triggers the OnStarted callback. Note that the OnStarted callback returns a dialogID.

// Initializes the WebSocket connection and sends a start request.
//
// Parameters:
//   - dialogID: Identifies a specific dialog session. Pass an empty string for a new dialog.
func (d *MultiModalDialog) Start(dialogID string) error

3. StartSpeech

Notifies the server to start audio upload. This method can only be called when the state is LISTENING.

// Starts uploading speech data.
func (d *MultiModalDialog) StartSpeech() error

// Sends speech data.
//
// Parameters:
//   - speechData: The binary speech data.
func (d *MultiModalDialog) SendAudioData(speechData []byte) error

// Stops uploading speech data.
func (d *MultiModalDialog) StopSpeech() error

// Requests the server to let the client start speaking (interrupt).
func (d *MultiModalDialog) Interrupt() error

// Notifies that local TTS playback has started.
func (d *MultiModalDialog) LocalRespondingStarted() error

// Notifies that local TTS playback has ended.
func (d *MultiModalDialog) LocalRespondingEnded() error

// Stops the dialog.
func (d *MultiModalDialog) Stop() error

// Gets the dialog state.
//
// Returns:
//   - DialogState: The current dialog state (Idle / Listening / Thinking / Responding).
func (d *MultiModalDialog) GetDialogState() DialogState

11. RequestToRespond

Requests the server to directly synthesize speech from text or sends a command to the server.

// Requests the server to synthesize speech from text or execute a command.
//
// Parameters:
//   - requestType: The request type ("prompt" or "transcript").
//   - text: The request text.
//   - parameters: Additional parameters, such as images (optional).
//   - interruptible: Specifies whether the response can be interrupted.
func (d *MultiModalDialog) RequestToRespond(
    requestType string,
    text string,
    parameters *RequestToRespondParameters,
    interruptible bool,
) error

Request parameters

// Configure upstream parameters.
upstream := &dashscope.Upstream{
    Type:        "AudioOnly",
    Mode:        "push2talk",
    AudioFormat: "pcm",
}

// Configure downstream parameters.
downstream := &dashscope.Downstream{
    Voice:            "longanhuan",
    SampleRate:       16000,
    IntermediateText: "transcript,dialog",
    AudioFormat:      "pcm",
}

// Configure client information.
clientInfo := &dashscope.ClientInfo{
    UserID: "user123",
    Device: &dashscope.Device{
        UUID: "1234567890",
    },
}

requestParams := &dashscope.RequestParameters{
    Upstream:   upstream,
    Downstream: downstream,
    ClientInfo: clientInfo,
}

type MultiModalCallback interface {
    // OnStarted is called when the dialog has started.
    //   - dialogID: The dialog ID returned by the callback.
    OnStarted(dialogID string)

    // OnStopped is called when the dialog has stopped.
    OnStopped()

    // OnStateChanged is called when the dialog state changes.
    //   - state: The new dialog state.
    //   - inputType: The input source that triggered the state change.
    OnStateChanged(state DialogState, inputType InputType)

    // OnSpeechAudioData is the callback for synthesized audio data.
    //   - data: The binary audio data.
    OnSpeechAudioData(data []byte)

    // OnError is called when an error occurs.
    //   - err: The error information.
    OnError(err error)

    // OnConnected is called after a successful connection to the server.
    OnConnected()

    // OnRespondingStarted is called when the model starts responding.
    OnRespondingStarted()

    // OnRespondingEnded is called when the model finishes responding.
    //   - payload: Additional information at the end of the response.
    OnRespondingEnded(payload map[string]interface{})

    // OnSpeechStarted is called when the start of speech input is detected.
    OnSpeechStarted()

    // OnSpeechEnded is called when the end of speech input is detected.
    OnSpeechEnded()

    // OnSpeechContent provides the recognized speech text.
    //   - payload: Data containing the recognized text.
    OnSpeechContent(payload map[string]interface{})

    // OnRespondingContent provides the model's response text.
    //   - payload: Data containing the response text.
    OnRespondingContent(payload map[string]interface{})

    // OnRequestAccepted is called when an interrupt request is accepted.
    OnRequestAccepted()

    // OnClose is called when the connection is closed.
    //   - closeStatusCode: The closing status code.
    //   - closeMsg: The closing message.
    OnClose(closeStatusCode int, closeMsg string)
}

Image question answering (VQA)

VQA enables multimodal interaction with both images and speech during a conversation.

The core process involves triggering VQA by speaking an intent, such as "take a look at this," or by directly inputting a text request. The service then returns a question-answering result based on the image content.

• Requesting with voice:

  1. Say something like, "Take a look at what's in front."

  2. The service returns a visual_qa intent through the OnRespondingContent callback.

  3. After receiving this intent, the client calls the RequestToRespond API and submits the image content to trigger a question-answering response.

• // Handle the visual_qa intent in the callback.
    func (c *MyCallback) OnRespondingContent(payload map[string]interface{}) {
        if output, ok := payload["output"].(map[string]interface{}); ok {
            if extraInfo, ok := output["extra_info"].(map[string]interface{}); ok {
                if commands, ok := extraInfo["commands"].(string); ok {
                    if strings.Contains(commands, "visual_qa") {
                        // Trigger VQA.
                        sendImageVQA(dialog)
                    }
                }
            }
        }
    }
  
    // Send a VQA request.
    func sendImageVQA(dialog *dashscope.MultiModalDialog) {
        image := map[string]interface{}{
            "type":  "base64",
            "value": imageBase64Data,
        }
        // Alternatively, use a URL.
        // image := map[string]interface{}{
        //     "type":  "url",
        //     "value": imageURL,
        // }
  
        imagesParams := &dashscope.RequestToRespondParameters{
            Images: []interface{}{image},
        }
  
        // Call VQA with voice; pass an empty string for the text parameter.
        dialog.RequestToRespond("prompt", "", imagesParams, false)
    }

• Requesting with text:

  1. The client directly calls the RequestToRespond API, submitting both the image content and the text request to trigger a question-answering response.

  image := map[string]interface{}{
        "type":  "base64",
        "value": imageBase64Data,
    }
    // Alternatively, use a URL.
    // image := map[string]interface{}{
    //     "type":  "url",
    //     "value": imageURL,
    // }
  
    imagesParams := &dashscope.RequestToRespondParameters{
        Images: []interface{}{image},
    }
  
    // Request an image response directly with text.
    dialog.RequestToRespond("prompt", "What is in this picture?", imagesParams, false)

Note: VQA supports sending image links or Base64-encoded data (for images smaller than 180 KB).

LiveAI video call over WebSocket

LiveAI (video call) is an official agent provided by the Model Studio multimodal interaction service. You can implement video calls by sending a sequence of images through the Go SDK. We recommend that your server and client (web or app) use RTC to transmit video and audio. Then, your server should send the captured video frames to the SDK at a rate of one frame every 500 ms while maintaining real-time audio input.

Note: For LiveAI, images must be Base64-encoded and smaller than 180 KB.

• LiveAI call sequence

• Key code example

// 1. Set the request mode to AudioAndVideo.
upstream := &dashscope.Upstream{
    Type:        "AudioAndVideo",
    Mode:        "duplex",
    AudioFormat: "pcm",
}

// 2. Send a request to connect to the video dialog agent.
func sendConnectVideoCommand(dialog *dashscope.MultiModalDialog) {
    videoConnectCommand := []interface{}{
        map[string]interface{}{
            "action": "connect",
            "type":   "voicechat_video_channel",
        },
    }

    params := &dashscope.RequestToRespondParameters{
        BizParams: &dashscope.BizParams{
            Videos: videoConnectCommand,
        },
    }

    dialog.RequestToRespond("prompt", "", params, false)
}

// 3. Send a video frame (under 180 KB) every 500 ms.
func sendVideoFrameLoop(dialog *dashscope.MultiModalDialog, stopChan chan bool) {
    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()

    for {
        select {
        case <-stopChan:
            return
        case <-ticker.C:
            image := map[string]interface{}{
                "type":  "base64",
                "value": getVideoFrameBase64(),
            }
            params := &dashscope.RequestToRespondParameters{
                Images: []interface{}{image},
            }
            dialog.RequestToRespond("prompt", "", params, false)
        }
    }
}

Text-to-speech (TTS)

The SDK lets you request the server to synthesize audio directly from text.

You must send a RequestToRespond request when the client is in the Listening state.

If the current state is not Listening, you must first call the Interrupt API to interrupt the current playback.

dialog.RequestToRespond("transcript", "The weather is nice today.", nil, false)

Requesting results with text

The SDK supports requesting LLM results and synthesized speech data directly using text.

You must send a RequestToRespond request when the client is in the Listening state.

dialog.RequestToRespond("prompt", "The weather is nice today.", nil, false)

1. Resource management

Call the Cleanup() method to release all resources when the session is complete:

defer dialog.Cleanup()

2. Heartbeat management

Send heartbeats periodically to keep the connection alive:

ticker := time.NewTicker(45 * time.Second)
go func() {
    for range ticker.C {
        dialog.SendHeartBeat()
    }
}()

3. State monitoring

Monitor dialog state changes by implementing the MultiModalCallback interface:

func (c *MyCallback) OnStateChanged(state dashscope.DialogState, inputType dashscope.InputType) {
    switch state {
    case dashscope.DialogStateListening:
        // The system is listening; you can start sending audio.
    case dashscope.DialogStateThinking:
        // The AI is thinking.
    case dashscope.DialogStateResponding:
        // The AI is responding; prepare to receive audio data.
    }
}