iOS Lite SDK for Mobile

The MultiModalDialog SDK is a software development kit (SDK) from Alibaba Cloud’s Tongyi team. It enables end-to-end, real-time multimodal interaction for audio and video. By integrating this SDK with Qwen large language models and various backend agents, you can support voice conversations, weather queries, music playback, news updates, and other capabilities. It also supports large model–powered dialogue using video and images.

Prerequisites

Enable an Alibaba Cloud Model Studio real-time multimodal interaction application to get your workspace ID, app ID, and API key.

Interaction Data Link

The multimodal dialog iOS Lite SDK uses WebSocket connections exclusively to interact with the server. It supports AudioOnly mode for voice conversations.

• Audio interaction: Uses WebSocket connections. These connections establish quickly and require low performance overhead.

  • Audio format for WebSocket links:

    • Uplink: Supports PCM and Opus formats for speech recognition.

    • Downlink: Supports PCM and MP3 audio streams.

Interaction Modes

The SDK supports three interaction modes: Push2Talk, Tap2Talk, and Duplex (full-duplex).

• Push2Talk: Press and hold to speak. Release to stop. Alternatively, tap to start and tap again to stop.

• Tap2Talk: Tap to start speaking. The system automatically detects when you finish speaking.

• Duplex: Full-duplex interaction. You can speak at any time after connecting. Voice interruption is supported.

  Note: Full-duplex interaction requires echo cancellation (AEC) on the client side. The Lite SDK does not include an AEC module. You can integrate your own AEC module or use the full-featured Alibaba Cloud Model Studio iOS SDK for Mobile.

Environment and Dependencies

• Import the SDK dashscope-lite-ios-demo-1.0.1.zip

  • DashscopeLite.framework: The Lite SDK for multimodal dialog.

  • Other dependencies:

    • 'SocketRocket', '~> 0.7.1'

• Connect to the Alibaba Cloud Model Studio environment

  • The multimodal dialog service runs on Alibaba Cloud Model Studio. Authentication uses your Alibaba Cloud Model Studio API key. Create an API key in the Alibaba Cloud Model Studio console. For security, you can also generate short-term tokens on your server and send them to the client. See short-term token.

Interface Reference

MultimodalDialog

Service entry class

1 MultiModalDialog

Initialize the service object. Set dialog parameters and callbacks.

/// Initialize the multimodal dialog manager.
/// - Parameters:
///   - url: Service URL
///   - workSpaceId: Workspace ID. Get it from the Alibaba Cloud Model Studio console.
///   - appId: App ID. Configure it in the multimodal dialog console.
///   - mode: Dialog mode (tap2talk, push2talk, or duplex)
public init(url: String?, workSpaceId: String, appId:String ,mode: DialogMode)

2 start

Start the dialog service. Returns the onConversationStarted callback.

/// Start a dialog request and connect to the dialog service.
/// - Parameters:
///   - apiKey: API key
///   - params: Multimodal request parameters
///   - completion: Completion callback. Returns whether the connection succeeded and any error message.
public func start(apiKey: String, params: MultiModalRequestParam, completion: @escaping (Bool, TYError?) -> Void ) 

3 startSpeech

Notify the server to start uploading audio. Call this only when the state is Listening. Use it only in Push2Talk or Tap2Talk mode.

/// Start speaking (press-and-hold mode).
public func startSpeech()

4 sendAudioData

Send audio data to the server.

/// Send audio data.
/// - Parameters:
///   - data: Pointer to audio data
///   - length: Data length
/// - Returns: Status code
public func sendAudioData(data: UnsafeMutablePointer<UInt8>, length: Int32) -> Int32? 

5 stopSpeech

Notify the server to stop uploading audio. Call this only in Push2Talk mode.

/// Stop speaking (press-and-hold mode).
public func stopSpeech() 

6 interrupt

Notify the server that the client wants to interrupt the current interaction and start speaking. Returns RequestAccepted.

// Interrupt the current dialog.
public func interrupt()

7 sendLocalRespondingStarted

Notify the server that the client has started playing TTS audio.

/// Send a signal that response playback has started.
public func sendLocalRespondingStarted()

8 sendLocalRespondingEnded

Notify the server that the client has stopped playing TTS audio.

/// Send a signal that response playback has ended.
public func sendLocalRespondingEnded()

9 stop

End the current round of dialog. Disconnect from the server.

/// Disconnect.
public func stop()

10 requestToRespond

Initiate text-to-speech (TTS) synthesis from the client side. Or send other requests—such as image requests—to the server.

/// Request a response.
/// - Parameters:
///   - type: Request type
///   - text: Request text
///   - params: Additional parameters
public func requestToRespond(type:String, text:String, params:[String : Any])

11 updateInfo

Update parameter information.

/// Update information.
/// - Parameter params: Parameters to update
public func updateInfo(params:[String : Any])

Callback

Callbacks for your app layer.

/// Callback when the server connection succeeds.
public var onConnected: (() -> Void)?

/// Callback when the dialog protocol completes and you can start speaking.
public var onConversationStarted: (() -> Void)?

/// Callback for audio events: SpeechStarted, SpeechEnded, RespondingStarted, RespondingEnded.
public var onConversationEvent: ((DialogEvent) -> Void)?

/// Callback when the dialog state changes: Idle, Listening, Thinking, or Responding.
public var onConversationStatechanged: ((DialogState) -> Void)?

/// Callback when speech recognition text is received.
public var onSpeechContent: (([String: Any]?) -> Void)?

/// Callback when the LLM response text is received.
public var onRespondingContent: (([String: Any]?) -> Void)?

/// Callback for errors.
public var onErrorReceived: ((TYError) -> Void)?

/// Callback for synthesized audio data.
public var onSynthesizedData:  ((UnsafeMutablePointer<UInt8>, Int32) -> Void)?

/// Callback when the WebSocket connection closes.
public var onWebsocketClosed:((_ code: Int,_ reason: String?) -> Void)?

MultiModalRequestParam

Request parameter class

All request parameters support builder-style setup. See the table below for parameter values and descriptions. These are parameters your client must configure or may optionally configure.

Parameters for the Start Connection Request

Parameters for requestToRespond

Parameters for updateInfo

Dialog States (DialogState)

The voice chat service has three states: LISTENING, THINKING, and RESPONDING.

IDLE("Idle"), System has not connected yet.
LISTENING("Listening"), Bot is listening for user input. You can send audio.
THINKING("Thinking"), Bot is thinking.
RESPONDING("Responding"), Bot is generating or playing back speech.

LLM Output from Dialog

Dialog results are delivered through the onMessageReceived callback. The format is shown below.

Call Sequence Diagram

Usage Examples

Initialize Dialog Parameters

Use builder methods in the MultiModalRequestParam class to build parameters.

var params = MultiModalRequestParam{ multiBuilder in
            multiBuilder.upStream = MultiModalRequestParam.UpStream(builder: { upstreamBuilder in
                upstreamBuilder.mode = DialogMode.duplex.rawValue
                upstreamBuilder.type = "AudioOnly"
            })
            multiBuilder.clientInfo = MultiModalRequestParam.ClientInfo(builder: {
                clientInfoBuilder in
                clientInfoBuilder.userId = "test-ios-user"
                clientInfoBuilder.device = MultiModalRequestParam.ClientInfo.Device(uuid: "12345")
            })
            multiBuilder.downStream = MultiModalRequestParam.DownStream(builder: {
                downStreamBuilder in
                downStreamBuilder.sampleRate = 48000
            })
        }

Create a MultiModalDialog Object

self.conversation = MultiModalDialog(url: self.HOST,  workSpaceId:self.WORKSPACE_ID ,
     appId: self.APP_ID, mode: DialogMode.duplex)

Complete Usage Example

See the ChatViewController calls in the Demo project downloaded in the Environment and Dependencies section.

More SDK Interface Details

// Create image request parameters.
private func createImageParams() -> [String: Any]{
    var imageObject:[String: Any] = [
        "type":"url",
        "value":self.image_url ?? ""
    ]
    // Or upload a Base64-encoded image. Only images under 180 KB are supported.
    var imageObjectBase64:[String: Any] = [
                "type":"base64",
                "value":"imagebase64"
    ]
    var images = [imageObject]
    
    var updateParam = MultiModalRequestParam{ multiBuilder in
        multiBuilder.images = images
    }
    return updateParam.parameters
}
// Send the request.
self.conversation?.requestToRespond(type: "prompt", text: "", params: self.createImageParams())

// 1. Set interaction type to AudioAndVideo.
multiBuilder.upStream = MultiModalRequestParam.UpStream(builder: { upstreamBuilder in
                upstreamBuilder.mode = mode.rawValue
                upstreamBuilder.type = "AudioAndVideo"
            })
// 2. Initialize the MultiModalDialog instance. Choose RTC chain mode: ChainMode.RTC.
self.conversation = MultiModalDialog(url: self.HOST, chainMode: self.chain,  workSpaceId:self.WORKSPACE_ID ,
                                             appId: self.APP_ID, mode: mode)

// 3. After connecting, send a voicechat_video_channel request.
private func createVideoChatParams() -> [String: Any]{
        var video:[String: Any] = [
            "action":"connect",
            "type" : "voicechat_video_channel"
        ]
        var videos = [video]
        
        var updateParam = MultiModalRequestParam{ multiBuilder in
            multiBuilder.bizParams = MultiModalRequestParam.BizParams(builder: {
                bizBuilder in
                bizBuilder.videos = videos
            })
        }
        return updateParam.parameters
}

// 4. Submit one video frame every 500 ms. Make sure each image is under 180 KB.
/**
 * The demo does not call this. It shows how to use LiveAI over WebSocket.
 * Start streaming video frames. Send one frame every 500 ms.
 */
// Start sending images in the background.
func startPeriodicExecution() {
    timer = DispatchSource.makeTimerSource(queue: queue)
    timer?.schedule(deadline: .now(), repeating: .milliseconds(500))
    timer?.setEventHandler { [weak self] in
        self?.sendLocalImageBase64()
    }
    timer?.resume()
}

// Update image Base64.
private func sendLocalImageBase64() {
    print("send_local_image_base64() called : \(Thread.current)")
    var imageObjectBase64:[String: Any] = [
                "type":"base64",
                "value":getImageAsBase64FromAssets(imageName:"bridge") // Encode image to Base64.
    ]
    var images = [imageObjectBase64]
    
    var updateParam = MultiModalRequestParam{ multiBuilder in
        multiBuilder.images = images
    }
    
    self.conversation?.updateInfo(params: updateParam.parameters)
    
}
// See the full example in the demo code.

self.conversation?.requestToRespond(type: "prompt", text: "Happiness is a skill—the inner peace you gain after letting go of unnecessary external desires.", params:[:])

// Pass biz_params.user_prompt_params in the request parameter builder.
var promptParam = ["user_name" : "Da Mi"]
var params = MultiModalRequestParam{ multiBuilder in
            multiBuilder.bizParams = MultiModalRequestParam.BizParams(builder: {
                bizBuilder in
                bizBuilder.userPromptParams = promptParam
            })
}