Harmony API details

Retrieves the user ID of the current call.

readonly userId: string | undefined

Type: string | undefined

Note: Read-only property. Retrieves the user ID of the current call.

Indicates whether a call is in progress.

readonly isOnCall: boolean

Type: boolean

Note: Read-only property. The value is true from when the call is connected until it is hung up or an error occurs. Otherwise, the value is false.

Retrieves information about the current agent.

readonly agentInfo: ARTCAICallAgentInfo | undefined

Type: ARTCAICallAgentInfo | undefined

Note: Read-only property. Retrieves information about the agent in the current call.

Retrieves the state of the current agent.

readonly agentState: ARTCAICallAgentState

Type: ARTCAICallAgentState

Note: Read-only property. Retrieves the state of the current agent.

Audio configuration.

audioConfig: ARTCAICallAudioConfig

Type: ARTCAICallAudioConfig

Note: Audio configuration. This must be set before the call starts to take effect.

Video configuration.

videoConfig: ARTCAICallVideoConfig

Type: ARTCAICallVideoConfig

Note: Video configuration. For a VisionAgent or VideoAgent, this must be set before the call starts to take effect.

Sets and retrieves callback events.

listener: ARTCAICallEngineListener | undefined

Type: ARTCAICallEngineListener | undefined

Note: Used to set and retrieve the callback event listener.

Starts a call.

call(userId: string, token: string, agentInfo: ARTCAICallAgentInfo): boolean

Parameters:

Return value: boolean

Note: Starts an AI call.

Starts a call using the call configuration.

callWithConfig(config: ARTCAICallConfig): boolean

Parameters:

Return value: boolean

Note: Starts an AI call using the complete call configuration.

Hangs up and destroys the current call engine.

handup(destroy: boolean): void

Parameters:

Return value: void

Note: Hangs up and destroys the current call engine. A message is first sent to notify the agent to stop the call. The system then waits for 100 ms before ending the call. If destroy is set to true, internal resources are also released.

Sets the rendering view and configuration for the agent.

setAgentViewConfig(viewConfig: ARTCAICallViewConfig | undefined): void

Parameters:

Return value: void

Note: Set this parameter when the agent requires image rendering. This is currently valid only for AvatarAgent and VideoAgent.

Sets the rendering view and configuration for the local camera.

setLocalViewConfig(viewConfig: ARTCAICallViewConfig | undefined): void

Parameters:

Return value: void

Note: The rendering view and configuration for the local camera. This is currently valid only for VisionAgent and VideoAgent.

Interrupts the agent's speech.

interruptSpeaking(): boolean

Return value: boolean

Note: This must be called after the call is connected. The return value indicates whether the operation was successful.

Sends a text message to the agent.

sendTextToAgent(req: ARTCAICallSendTextToAgentRequest): boolean

Parameters:

Return value: boolean

Note: This must be called after the call is connected. The return value indicates whether the operation was successful.

Sends a custom message to the server.

sendCustomMessageToServer(msg: string): boolean

Parameters:

Return value: boolean

Note: This must be called after a connection is established and returns a value that indicates whether the call was successful.

Updates the system prompt for the LLM.

updateLlmSystemPrompt(prompt: string): boolean

Parameters:

Return value: boolean

Note: This must be called after the call is connected. For example: "You are a friendly and helpful assistant focused on providing users with accurate information and advice." Note: This is not supported if the LLM node is a Model Studio workflow. A successful return value does not indicate that the update has taken effect.

Updates the parameters for the Model Studio application center.

updateBailianAppParams(params: Record<string, RecordValueType>): boolean

Parameters:

Return value: boolean

Note: Updates the parameters for the Model Studio application center. The return value indicates whether the message was sent successfully, not whether the update has taken effect.

Updates the VCR configuration.

updateVcrConfig(vcrConfig: ARTCAICallAgentVcrConfig): boolean

Parameters:

Return value: boolean

Note: Updates the VCR configuration. The return value indicates whether the message was sent successfully, not whether the update has taken effect.

Updates the threshold for voice activity detection.

updateAsrMaxSilence(duration: number): boolean

Parameters:

Return value: boolean

Note: A period of silence longer than this threshold is considered a sentence break. The parameter range is 200 ms to 1200 ms. The return value indicates whether the message was sent successfully, not whether the update has taken effect.

Updates the TTS speech rate.

updateTtsSpeechRate(rate: number): boolean

Parameters:

Return value: boolean

Note: All TTS types are supported. The value ranges from 0.5 to 2.0. The return value indicates whether the message was sent successfully, not whether the update has taken effect.

Queries the current state of the agent.

queryCurrentAgentState(): boolean

Return value: boolean

Note: After the query is sent successfully, the onAgentStateChanged event is triggered. The return value indicates whether the query was sent successfully.

For a Vision agent, starts custom frame capture.

startVisionCustomCapture(req: ARTCAICallVisionCustomCaptureRequest): boolean

Parameters:

Return value: boolean

Note: After this is started, voice communication with the agent is disabled. This must be called after the call is connected. The return value indicates whether the operation was successful.

For a Vision agent, stops custom frame capture.

stopVisionCustomCapture(): boolean

Return value: boolean

Note: This must be called after the call is connected. The return value indicates whether the operation was successful.

Enables or disables smart interruption.

enableVoiceInterrupt(enable: boolean): boolean

Parameters:

Return value: boolean

Note: This must be called after the call is connected. The return value indicates whether the operation was successful.

Switches the voice.

switchVoiceId(voiceId: string): boolean

Parameters:

Return value: boolean

Note: Switches the agent's voice. The return value indicates whether the operation was successful.

Enables or disables the speaker.

enableSpeaker(enable: boolean): boolean

Parameters:

Return value: boolean

Note: Enables or disables the speaker. The return value indicates whether the operation was successful.

Stops or resumes playback of the agent's audio stream.

muteAgentAudioPlaying(mute: boolean): boolean

Parameters:

Return value: boolean

Note: Stops or resumes playback of the agent's audio stream. The return value indicates whether the operation was successful.

Mutes or unmutes the microphone.

muteMicrophone(mute: boolean): boolean

Parameters:

Return value: boolean

Note: Mutes or unmutes the microphone. The return value indicates whether the operation was successful.

Disables or enables the camera.

muteLocalCamera(mute: boolean): boolean

Parameters:

Return value: boolean

Note: This is valid for VisionAgent and VideoAgent. Disables or enables the camera. The return value indicates whether the operation was successful.

Switches between the front and rear cameras.

switchCamera(): boolean

Return value: boolean

Note: This is valid for VisionAgent and VideoAgent. Switches between the front and rear cameras. The return value indicates whether the operation was successful.

Enables or disables push-to-talk mode.

enablePushToTalk(enable: boolean): boolean

Parameters:

Return value: boolean

Note: In push-to-talk mode, the agent reports the result only after finishPushToTalk is called. The return value indicates whether the operation was successful.

Starts speaking.

startPushToTalk(): boolean

Return value: boolean

Note: Starts speaking. The return value indicates whether the operation was successful.

Finishes speaking.

finishPushToTalk(): boolean

Return value: boolean

Note: Finishes speaking. The return value indicates whether the operation was successful.

Cancels the current speech.

cancelPushToTalk(): boolean

Return value: boolean

Note: Cancels the current speech. The return value indicates whether the operation was successful.

Specifies whether to use voiceprint-based denoising for the current speech segment.

useVoiceprint(isUse: boolean): boolean

Parameters:

Return value: boolean

Note: This feature is in invitational preview. To use it, contact the relevant personnel. The return value indicates whether the operation was successful.

Purges the current voiceprint data.

clearVoiceprint(): boolean

Return value: boolean

Note: This feature is in invitational preview. To use it, contact the relevant personnel. The return value indicates whether the operation was successful.

Creates an RTC engine.

createRTCEngine(): void

Return value: void

Note: Call this API operation to create an RTC engine in advance if needed. Otherwise, the engine is created after the call API operation is invoked.

Retrieves the RTC engine.

getRTCInstance(): object | undefined

Return value: object | undefined

Note: Retrieves the RTC engine instance.

Releases resources.

destroy(): void

Return value: void

Note: Releases resources.

Parses shared agent information.

parseShareAgentCall(shareInfo: string): Promise<ARTCAICallAgentShareConfig | undefined>

Parameters:

Return value: Promise<ARTCAICallAgentShareConfig | undefined>

Note: Parses the shared agent information. The shared agent configuration is returned.

Starts a call with a shared agent.

generateShareAgentCall(shareConfig: ARTCAICallAgentShareConfig, userId: string): Promise<ARTCAICallAgentShareGenerateResult | undefined>

Parameters:

Return value: Promise<ARTCAICallAgentShareGenerateResult | undefined>

Note: Starts a call with a shared agent.

An error occurred.

onErrorOccurs?(code: ARTCAICallErrorCode): void

Parameters:

Return value: void

Note: Triggered when an error occurs.

The connection status changes during a call.

onConnectionStatusChange?(status: ARTCAICallConnectionStatus, reason: number): void

Parameters:

Return value: void

Note: Triggered when the connection status changes.

The call starts (joins the session).

onCallBegin?(): void

Return value: void

Note: Triggered when the call starts.

The call ends (leaves the session).

onCallEnd?(): void

Return value: void

Note: Triggered when the call ends.

The call agent has started.

onAgentStarted?(): void

Return value: void

Note: Triggered when the agent starts.

The RTC engine is created. Call getRTCInstance in this callback to get the RTC engine instance.

onRTCEngineCreated?(): void

Return value: void

Note: Triggered when the RTC engine is created.

Callback for when the agent's message channel is available. Messages can be sent to the agent only after this callback is triggered.

onAgentDataChannelAvailable?(): void

Return value: void

Note: Triggered when the agent's tunnel is available. You must wait for this callback to be triggered before sending messages to the agent.

Indicates whether the agent's video stream is available (stream ingest).

onAgentVideoAvailable?(available: boolean): void

Parameters:

Return value: void

Note: Triggered when the availability of the agent's video stream changes.

Indicates whether the agent's audio stream is available (stream ingest).

onAgentAudioAvailable?(available: boolean): void

Parameters:

Return value: void

Note: Triggered when the availability of the agent's audio stream changes.

The first frame of the agent's digital human is rendered.

onAgentAvatarFirstFrameDrawn?(): void

Return value: void

Note: Triggered after the first frame of the agent's digital human is rendered.

The agent's state changes.

onAgentStateChanged?(state: ARTCAICallAgentState): void

Parameters:

Return value: void

Note: Triggered when the agent's state changes.

The network status changes.

onNetworkStatusChanged?(uid: string, quality: ARTCAICallNetworkQuality): void

Parameters:

Return value: void

Note: Triggered when the network status changes.

Notification of a volume change.

onVoiceVolumeChanged?(uid: string, volume: number): void

Parameters:

Return value: void

Note: Triggered when the volume changes.

Notification of the agent's recognition result for a user's question.

onUserSubtitleNotify?(data: ARTCAICallUserSubtitleResult): void

Parameters:

Return value: void

Note: Triggered when the agent recognizes a user's question.

Notification of the agent's answer.

onVoiceAgentSubtitleNotify?(data: ARTCAICallAgentSubtitleResult): void

Parameters:

Return value: void

Note: Triggered when the agent's answer is reported.

Notification of the agent's emotion.

onAgentEmotionNotify?(emotion: string, userAsrSentenceId: number): void

Parameters:

Return value: void

Note: Triggered when the agent's emotion is reported.

The voice of the current call has changed.

onVoiceIdChanged?(voiceId: string): void

Parameters:

Return value: void

Note: Triggered when the call's voice changes.

Indicates whether voice interruption is enabled for the current call.

onVoiceInterrupted?(enable: boolean): void

Parameters:

Return value: void

Note: Triggered when the voice interruption status changes.

The agent's current speech is interrupted.

onSpeakingInterrupted?(reason: ARTCAICallSpeakingInterruptedReason): void

Parameters:

Return value: void

Note: Triggered when the agent's speech is interrupted.

Indicates whether push-to-talk mode is enabled for the current call.

onPushToTalk?(enable: boolean): void

Parameters:

Return value: void

Note: Triggered when the push-to-talk mode status changes.

Indicates whether voiceprint-based denoising is enabled for the current call.

onVoiceprint?(enable: boolean): void

Parameters:

Return value: void

Note: Triggered when the status of the voiceprint-based denoising feature changes (invitational preview).

The voiceprint data for the current call is purged.

onVoiceprintCleared?(): void

Return value: void

Note: Triggered when the voiceprint data is purged (invitational preview).

The current agent is about to leave (end the current call).

onAgentWillLeave?(reason: number, message: string): void

Parameters:

Return value: void

Note: Triggered when the agent is about to leave.

A custom message is received from the current agent.

onReceivedAgentCustomMessage?(data: Record<string, RecordValueType> | undefined): void

Parameters:

Return value: void

Note: Triggered when a custom message is received from the agent.

A VCR result is received from the current agent.

onReceivedAgentVcrResult?(result: ARTCAICallAgentVcrResult): void

Parameters:

Return value: void

Note: Triggered when a VCR result is received from the agent.

A human is about to take over from the current agent.

onHumanTakeoverWillStart?(takeoverUid: string, takeoverMode: number): void

Parameters:

Return value: void

Note: Triggered when a human is about to take over from the agent.

The human takeover is connected.

onHumanTakeoverConnected?(takeoverUid: string): void

Parameters:

Return value: void

Note: Triggered when the human takeover is successfully connected.

Audio loopback delay.

onAudioDelayInfo?(sentenceId: number, delayMs: number): void

Parameters:

Return value: void

Note: Triggered when there is information about audio loopback delay.

Indicates whether custom frame capture mode is enabled for the current Vision call.

onVisionCustomCapture?(enable: boolean): void

Parameters:

Return value: void

Note: Triggered when the status of the custom frame capture mode for a Vision call changes.

The LLM has finished answering in the current call.

onLLMReplyCompleted?(text: string, userAsrSentenceId: number): void

Parameters:

Return value: void

Note: Triggered when the LLM finishes answering.

If music accompaniment is played through the RTC instance during the current call, this playback callback is triggered.

onAudioAccompanyStateChanged?(state: ARTCAICallAudioAccompanyState, errorCode: ARTCAICallAudioAccompanyErrorCode): void

Parameters:

Return value: void

Note: Triggered when the audio accompaniment state changes.

Creates a default AICallEngine.

public static createEngine(context : common.Context): ARTCAICallEngineInterface

Parameters:

Return value: ARTCAICallEngineInterface

Note: Creates and returns an AI call engine instance.