Audio capture and playback

Basics

Common audio format terms:

• Sample rate: The number of audio samples captured per second. For example, 8000 Hz (8 kHz) and 16000 Hz (16 kHz) represent 8,000 and 16,000 samples per second, respectively.

• Sample bit depth: The amount of information stored in each sample. For example, a 16-bit sample bit depth means each sample is stored using 16 bits (2 bytes).

• Channel: The number of audio streams. Common formats are mono and stereo.

Use the following formula to calculate the file size from its duration:

• File size (MiB) = (sample rate (Hz) × sample bit depth × channel count × duration (s)) / (8 × 1024 × 1024)

  • Example: (16000 Hz × 16 bit × 1 channel × 60 s) / (8 × 1024 × 1024) ≈ 1.83 MiB

Checking audio formats on Linux

1. Run the following command:

file input.wav

2. Expected result:

   • The output for a mono, uncompressed WAV file with a 16000 Hz sample rate and 16-bit sample bit depth appears as follows:

     $file test-16000.wav
     test-16000.wav: RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 16000 Hz

Checking audio formats on Windows

1. On Windows, select the target audio file, right-click, and then choose Properties to view more details.

2. Expected result:

   • For a mono, uncompressed WAV file with a 16000 Hz sample rate and 16-bit sample bit depth, the file properties display the following: Duration: 00:05, Audio channels: Mono, Sample rate: 16 kHz, and Bits per sample: 16.

Converting audio formats on Linux

Use the following common FFmpeg commands. For more information, see Download FFmpeg.

# Check audio format details such as sample rate, channels, and encoding.
ffmpeg -i input.mp3
# Convert a WAV file to an 8 kHz, 16-bit, mono WAV file.
ffmpeg -i input.wav  -ar 8000 -ac 1 -acodec pcm_s16le -f s16le output.wav
# Convert a WAV file to a 16 kHz, 16-bit, mono WAV file.
ffmpeg -i input.wav  -ar 16000 -ac 1 -acodec pcm_s16le -f s16le output.wav
# Convert a PCM file to a 16 kHz, 16-bit, mono WAV file.
ffmpeg -i input.pcm -f s16le -ar 16000 -ac 1 -acodec pcm_s16le  output.wav
# Convert a WAV file to a 16 kHz, 16-bit, mono PCM file.
ffmpeg -y -i input.wav -acodec pcm_s16le -f s16le -ac 1 -ar 16000 output.pcm
# Convert an MP3 file to a 16 kHz, 16-bit, mono WAV file.
ffmpeg -y -i input.mp3 -acodec pcm_s16le -f s16le -ac 1 -ar 16000 output.wav
# Convert a 44.1 kHz, 16-bit WAV file to a 16 kHz, 16-bit, mono WAV file.
ffmpeg -y -f s16le -ar 44100 -ac 1 -i input.wav -acodec pcm_s16le -f s16le -ac 1 -ar 16000 output.wav
# Convert an 8 kHz A-law file to an 8 kHz, 16-bit, mono WAV file.
ffmpeg -f alaw -ar 8000 -i input.wav -ar 8000 -ac 1 -acodec pcm_s16le -f s16le output.wav
# Convert an 8 kHz mu-law file to an 8 kHz, 16-bit, mono WAV file.
ffmpeg -f mulaw -ar 8000 -i input.wav -ar 8000 -ac 1 -acodec pcm_s16le -f s16le output.wav
# Convert an AMR file to a 16 kHz, 16-bit, mono WAV file.
ffmpeg -i input.amr -ar 16000 -ac 1 -acodec pcm_s16le -f s16le output.wav

Converting audio formats on Windows

On Windows, you can use audio conversion tools such as Adobe Audition, Cool Edit Pro, or other online and offline utilities to convert audio formats.

The following example shows how to export 16 kHz audio data using Adobe Audition. Open the Batch Process panel, and click the folder icon to add the audio file to convert, such as test.wav. Click the Export Settings... button at the bottom. In the Export Settings dialog box, set the export location, set Format to Wave PCM (*.wav, *.bwf, *.rf64, *.amb), and set New Sample Type to 16000 Hz Mono, 16 bit. Click OK, and then click Run to start the batch export.

Opus is a widely used open-source, compressed audio format. It is often integrated as a default audio codec, especially on devices that use a Real-Time Operating System (RTOS).

Because RTOS solutions typically provide core Opus encoding and decoding capabilities but may not support the Ogg container, the multimodal interaction development kit supports raw OPUS audio data for both input and output. This audio format is named "raw-opus".

1. Upstream sample rate: upstream.sample_rate: 48000
2. Upstream format: upstream.audio_format: raw-opus
3. Downstream sample rate: downstream.sample_rate
4. Downstream format: downstream.audio_format: raw-opus
5. Downstream Opus frame size: downstream.frame_size: 20

No recognition results with my own audio

• Check the audio file format.

  • Check if your test audio format meets the speech recognition input requirements. For more information, see the Audio formats section.

  • Convert the test audio to a mono, uncompressed WAV file with a 16 kHz sample rate and 16-bit sample bit depth.

No final results in Tap2Talk or Duplex mode

• Tap2Talk/Duplex modes use cloud VAD (Voice Activity Detection) to detect the end of speech. If you are using an audio file for input, you must append at least 800 to 1000 ms of silence to the end of the file. Otherwise, the service cannot determine when the speech ends.

Unexpected speech synthesis playback speed

• Check your playback parameter settings.

  • Check your playback parameters, especially the sample rate, sample bit depth, and channel count. Incorrect settings can cause audio distortion, particularly when playing PCM data.

  • For example, a typical Android client configuration uses AudioTrack for playback, as shown below.

AudioFormat format = new AudioFormat.Builder()
        .setSampleRate(16000)
        .setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
        .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
        .build();