This document describes the design and usage of the IoT version of the DingRTC SDK.
1. System framework of the DingRTC IoT SDK
The grayed-out sections in the diagram represent video features, which are not supported in the current version.
2. Package size
For example, the package size on a 64-bit Linux system is less than 1 MB.
3. Dependency libraries
These libraries require the operating system to provide the following:
libwebsockets
openssl
libcurl
Note: Porting is required if mbed TLS is used instead of OpenSSL in an embedded system.
Note: The interfaces provided by libwebsockets vary significantly across different systems, which requires porting.
Note: The dependency on libcurl is not strict. You can use other methods for HTTPS access instead.
Note: If you have sufficient network bandwidth, you can use G.711 as the audio compression format instead of Opus. This removes the Opus dependency and reduces the package size.
Note: If your system provides the json-c library, you can reuse it to save 70 KB of file size.
Note: Supporting the data channel increases the package size.
Note: The build toolchain must support basic C++11 features, such as std::thread, std::mutex, lambda expressions, std::string, std::list, std::map, std::queue, and std::vector.
4. Integration method
The IoT SDK uses a microkernel and pipeline design. The Real-Time Communication (RTC) stream ingest and pulling module is an optional component in the pipeline that is required only for audio and video calls.
For real-time audio communication, integration is a two-step process. First, build the pipeline. Then, start RTC stream ingest and pulling.
The application implements the microphone and speaker. The following diagram shows an example of microphone and speaker modules that are encapsulated using SDL2.
The diagram shows that RtcSender and RtcReceiver are the stream ingest and pulling modules embedded in the pipeline. These two modules are required only for RTC communication and can be removed from the pipeline for non-RTC scenarios.
Application development steps:
1. In the business layer, create a microphone module and a speaker module. The following code provides an example:
class SdlSpeaker : public NetBit::ExternalSpeaker
{
public:
SdlSpeaker(NetBit::Callback *cb);
virtual ~SdlSpeaker();
// Use SetParameters to change the working parameters of the speaker module.
// For supported parameters, see the corresponding implementation function.
void SetParameters(const char **keys,
const void **vals, int count) override;
// For RTC applications, the difference from a normal player is that
// audio playback relies on a callback from the sound card to pull audio data from upstream.
// direct source provides the data source.
// Typically, the direct source is the neteq module.
void SetDirectSource(AVFrame *(*getter)(void *userdata), void *userdata) override {
get_one_frame_ = getter;
userdata_ = userdata;
}
// protected:
int32_t PreLoop() override;
void PostLoop() override;
protected:
// Sound card playback callback function
static void cb_fill_audio(void *usrdata, uint8_t *stream, int32_t len);
private:
AVFrame *current_;
int32_t read_pos_in_bytes_;
// Called by cb_fill_audio() to read audio data
int32_t fetch_bytes(void *dst, int bytes);
int32_t freq_;
int32_t channels_;
// In player scenarios, this indicates whether buffering is in progress.
// This is not meaningful in RTC scenarios.
bool buffering_;
// if get_one_frame_ is set, then use this function
// as source, not input_frame_queue_
AVFrame * (*get_one_frame_)(void *userdata);
void *userdata_;
};#define PLAYBACK_BUFFERING_MS 40
void initonce_sdl()
{
SDL_Init(SDL_INIT_EVERYTHING);
}
SdlSpeaker::SdlSpeaker(NetBit::Callback *cb) : NetBit::ExternalSpeaker(cb)
{
current_ = NULL;
read_pos_in_bytes_ = 0;
freq_ = 16000;
channels_ = 1;
buffering_ = true;
get_one_frame_ = NULL;
userdata_ = NULL;
}
SdlSpeaker::~SdlSpeaker()
{
}
void SdlSpeaker::SetParameters(const char **keys,
const void **vals, int count)
{
// TODO: lock protection
for (int i = 0; i < count; i++)
{
if (strcmp(keys[i], "sampleRate") == 0) {
freq_ = (uint64_t) vals[i];
}
else if (strcmp(keys[i], "channels") == 0) {
channels_ = (uint64_t) vals[i];
}
}
}
int32_t SdlSpeaker::fetch_bytes(void *dst, int bytes)
{
int32_t read_bytes = 0;
if (get_one_frame_ != NULL) {
if (current_ == NULL) {
current_ = get_one_frame_(userdata_);
read_pos_in_bytes_ = 0;
}
if (current_ == NULL) {
return 0;
}
int32_t frame_bytes = current_->content_->samples_per_channel * current_->content_->channels * sizeof(short);
int32_t remain = frame_bytes - read_pos_in_bytes_;
read_bytes = (remain < bytes) ? remain : bytes;
memcpy(dst,
(uint8_t *) current_->content_->buffers[0] + read_pos_in_bytes_,
read_bytes);
read_pos_in_bytes_ += read_bytes;
if(read_pos_in_bytes_ == frame_bytes) {
delete current_;
current_ = NULL;
}
return read_bytes;
}
// buffering management
// quit buffering state?
if (buffering_) {
int latency = GetInputQueueSize() * 20; // FIXME! not always 20 ms!
if (latency > PLAYBACK_BUFFERING_MS) {
buffering_ = false;
printf("leaving buffering\n");
}
}
if (!buffering_) {
if(current_ == NULL) {
current_ = Dequeue();
read_pos_in_bytes_ = 0;
}
if(current_ != NULL) {
int32_t frame_bytes = current_->content_->samples_per_channel * current_->content_->channels * sizeof(short);
int32_t remain = frame_bytes - read_pos_in_bytes_;
read_bytes = (remain < bytes) ? remain : bytes;
memcpy(dst,
(uint8_t *) current_->content_->buffers[0] + read_pos_in_bytes_,
read_bytes);
read_pos_in_bytes_ += read_bytes;
if(read_pos_in_bytes_ == frame_bytes) {
delete current_;
current_ = NULL;
}
}
else {
// underrun, buffering
buffering_ = true;
printf("entering buffeing\n");
}
}
return read_bytes;
}
void SdlSpeaker::cb_fill_audio(void *udata, uint8_t *stream, int32_t len)
{
SdlSpeaker *pThis = (SdlSpeaker *) udata;
uint8_t *dst = stream;
do {
int32_t read = pThis->fetch_bytes(dst, len);
if(read == 0)
break;
dst += read;
len -= read;
} while(1);
// mute unfilled part
SDL_memset(dst, 0, len);
}
int32_t SdlSpeaker::PreLoop()
{
MYASSERT(current_ == NULL);
MYASSERT(read_pos_in_bytes_ == 0);
/* initonce_sdl is moved to the global scope */
// Initialize SDL
// if (SDL_Init(SDL_INIT_EVERYTHING) < 0) {
// printf("Error: SDL failed to init!\n");
// return -1;
// }
//SDL_AudioSpec
SDL_AudioSpec wanted_spec;
SDL_memset(&wanted_spec, 0, sizeof(wanted_spec));
wanted_spec.freq = freq_;
wanted_spec.format = AUDIO_S16;
wanted_spec.channels = channels_;
wanted_spec.samples = freq_ * 20 / 1000; //1024; // audio buffer size in samples (must be power of 2)
wanted_spec.callback = cb_fill_audio;
wanted_spec.userdata = this;
if (SDL_OpenAudio(&wanted_spec, NULL) < 0) {
printf("can't open audio.\n");
return -1;
}
if(wanted_spec.format != AUDIO_S16) {
printf("not supported format %d\n", wanted_spec.format);
return -1;
}
freq_ = wanted_spec.freq;
channels_ = wanted_spec.channels;
// Play
SDL_PauseAudio(0);
return 0;
}
void SdlSpeaker::PostLoop()
{
SDL_CloseAudio();
if(current_ != NULL)
{
delete current_;
current_ = NULL;
}
read_pos_in_bytes_ = 0;
}
2. Create the pipeline.
The following code is an excerpt from the demo:
class Rtc_Pipeline : public Pipeline
{
public:
Rtc_Pipeline(ding::rtc::RtcEngine *engine);
virtual ~Rtc_Pipeline();
#if defined(PLATFORM_MAC)
SdlRecorder *external_mic_ = NULL;
#endif
NetBit::ExternalSpeaker *extSpeaker_ = NULL;
NetBit::MediaModuleEx *neteq_speaker_ = NULL;
NetBit::MediaModuleEx *opus_encoder_ = NULL;
NetBit::MediaModuleEx *sender_ = NULL;
NetBit::MediaModuleEx *receiver_ = NULL;
};Rtc_Pipeline::Rtc_Pipeline(ding::rtc::RtcEngine *engine)
: Pipeline(engine)
{
/* Create all modules and configure parameters
*/
{
external_mic_ = new SdlRecorder(&_callback);
external_mic_->SetModuleID("mic");
const char *keys[] = {"sampleRate", "channels", "frameDuration"};
const void *vals[] = {(void *)(uint64_t)16000, (void *)(uint64_t)1, (void *)(uint64_t)20};
external_mic_->SetParameters(keys, vals, sizeof(keys) / sizeof(keys[0]));
}
{
extSpeaker_ = new SdlSpeaker(&_callback);
extSpeaker_->SetModuleID("speaker");
}
{
neteq_speaker_ = ding::rtc::RtcEngine::CreateModule("NeteqPlayer", "neteqspeaker", &_callback);
const char *keys[] = {"speaker-device", "sampleRate", "channels"};
// use 48000, even if the actual samplerate is 16000
const void *vals[] = {extSpeaker_, (void *)(uint64_t)48000, (void *)(uint64_t)1};
neteq_speaker_->SetParameters(keys, vals, sizeof(keys) / sizeof(keys[0]));
}
{
opus_encoder_ = ding::rtc::RtcEngine::CreateModule("OpusEncoder", "opusenc", &_callback);
}
{
sender_ = engine_->GetRtcSender(true, &_callback);
}
{
receiver_ = engine_->GetRtcReceiver(true, &_callback);
}
/* Build the pipeline
*/
ding::rtc::RtcEngine::Connect(external_mic_, opus_encoder_);
ding::rtc::RtcEngine::Connect(opus_encoder_, sender_);
ding::rtc::RtcEngine::Connect(receiver_, neteq_speaker_);
/* Start all modules
*/
external_mic_->Start();
opus_encoder_->Start();
sender_->Start();
receiver_->Start();
neteq_speaker_->Start();
extSpeaker_->Start();
}3. Establish stream ingest and pulling between the client and the server.
In the pipeline, the RtcSender module ingests the audio stream, and the RtcReceiver module pulls the stream. When you call the JoinChannel interface to join a channel, the upstream and downstream audio links are automatically established between the RtcSender and RtcReceiver modules and the server.
After this step is complete, the entire pipeline is connected. This enables full-duplex, real-time voice communication with other users in the channel.
5. API reference
Unlike other DingRTC native SDKs, the IoT SDK provides pipeline interfaces in addition to RTC interfaces.
RTC interfaces: Establish RTC stream ingest and pulling links.
JoinChannel: Joins a channel. By default, this call automatically establishes stream ingest and pulling links.
LeaveChannel: Leaves the channel and disconnects the RTC stream ingest and pulling links without affecting the pipeline.
PublishLocalTrack: Establishes a media link with the server for stream ingest. By default, this is executed automatically after you join a channel, so your application does not typically need to call this interface.
SubscribeRemoteTrack: Establishes a media link with the server for stream pulling. By default, this is executed automatically after you join a channel, so your application does not typically need to call this interface.
Pipeline interfaces: Build the audio pipeline. Your application can extend and customize the pipeline.
CreateModule: Creates a module. Do not use this interface to create RTC stream ingest and pulling modules or custom application modules.
DeleteModule: Deletes a module. The engine maintains the RTC stream ingest and pulling modules, which are not created using this interface. Use this interface to delete other modules. Your application is responsible for deleting its custom modules.
Connect: Connects two modules.
Disconnect: Disconnects two modules.
Error detection: RTC stream ingest and pulling are susceptible to interference, which can cause connection issues with the server. Your application must listen for specific callbacks and notifications to handle these issues. If an error occurs, you can try to recover the connection by calling LeaveChannel and then JoinChannel. If the connection repeatedly fails, you must implement a recovery policy, such as sending a notification for manual inspection.
OnJoinChannelResult
OnPublishResult
OnSubscribeResult
OnOccurError