AI real-time chat agent

Introduction

You can use the real-time messaging (RTM) feature of DingRTC to receive the service status, transcription messages, and control commands from the AI real-time chat agent. The RTM feature operates within meetings, so you must join a meeting before using this feature.

Concepts

Session: A real-time messaging session that is uniquely identified by a sessionId. A session supports join, leave, and close operations. You must join the session before you can send or receive real-time messages for an agent.

Agent ID: The user ID of the agent.

Transcription message: The transcribed text from the conversation with the agent.

Natural conversation/Walkie-talkie mode: The conversation mode of the agent. This mode is created using an OpenAPI.

Integration method

1. Before you join a meeting, integrate the basic RTM feature. For more information, see Windows integration method.

2. Obtain the RtmAgentClient from the RtmClient:

   ding::rtc::RtmClient *rtmclient = rtcengine->GetRtmClient();
   ding::rtc::RtmAgentClient *agentclient = rtmclient->GetAgentClient();

3. Set a callback listener for the RtmAgentClient instance:

   class MyRtmAgentEventListener : public ding::rtc::RtmAgentEventListener
   {
   public:
   void OnAgentUserReady(const ding::rtc::String &sessionId, const ding::rtc::String &agentId) override {
   }
   
   void OnAgentStateChanged(const ding::rtc::String &sessionId, const ding::rtc::String &agentId, AgentState state) override {
   }
   
   void OnTranscriptionMessage(const ding::rtc::String &sessionId, const ding::rtc::String &agentId, const TranscriptionData &data) override {
   }
   };

4. After you join the meeting, call the StartAgent OpenAPI operation to make the agent join the meeting and the session.

5. Other members in the meeting receive an OnSessionCreate event notification. After receiving the notification, they can join the same session:

   int rc = rtmclient->JoinSession(sessId); // Call joinSession to join an existing session.

6. You receive an OnAgentUserReady event notification when an agent is present in the session you join, or when an agent joins a session you are already in. The application layer must maintain the list of agents for the session:

   void OnAgentUserReady(const ding::rtc::String &sessionId, const ding::rtc::String &agentId) override {
       // Save the agentId at the application layer to call agent-related methods.
   }

7. Use the onAgentStateChanged event to monitor the current status of the agent:

   void OnAgentStateChanged(const ding::rtc::String &sessionId, const ding::rtc::String &agentId, AgentState state) override {
       // Use state to get the current status of the agent: Listening, Thinking, or Responding.
   }

8. You can use the SDK to send specific signals to control the agent's behavior. For example, to make the agent send a greeting:

   int rc = agentclient->SendGreetingReady(sessId, agentId);

   To manually interrupt the agent:

   int rc = agentclient->SendInterrupt(sessId, agentId);

   In walkie-talkie mode, you can send control signals. Note that you must send ding::rtc::AgentPushToTalkCmd::Start before you send ding::rtc::AgentPushToTalkCmd::Send:

   // Make the agent start accepting user speech input.
   int rc = agentclient->SendPushToTalk(sessId, agentId, ding::rtc::AgentPushToTalkCmd::Start);
   ...
   // Make the agent start processing the speech data that the user has already input.
   int rc = agentclient->SendPushToTalk(sessId, agentId, ding::rtc::AgentPushToTalkCmd::Send);
   ...
   // Cancel the user's current speech input.
   int rc = agentclient->SendPushToTalk(sessId, agentId, ding::rtc::AgentPushToTalkCmd::Cancel);

9. If you no longer need the RTM feature for the agent, leave the session:

   int rc = rtmclient->LeaveSession(sessId);

Notes

1. The server-side StartAgent OpenAPI operation controls the agent's creation and its entry into the RTM session.

2. Do not close an RTM session that contains an agent. The session is managed by the server-side agent service using OpenAPI.

3. If a remote user leaves the channel without calling LeaveSession, you might not receive the OnSessionRemoteUserLeave event notification for that user. When you receive the OnRemoteUserOffLineNotify event notification for the user, the application layer must remove that user from the member list of all sessions.

4. Set the callback listener for the RtmAgentClient instance before you join the meeting to avoid missing callback messages.