A typical voiceprint retrieval solution based on AnalyticDB for MySQL-AnalyticDB(AnalyticDB)-阿里云帮助中心

This topic describes a voiceprint recognition solution based on AnalyticDB for MySQL and presents a case study on detecting sensitive content from ride-hailing drivers. The solution supports scenarios such as ride-hailing, multi-person meetings, offline sales, AI voice recorders, and voice assistants. It enables speaker identification and content quality inspection for building efficient voiceprint retrieval systems.

Background

Voice is a key biometric identifier for identity authentication, security control, and interactive applications. Voiceprint recognition technology extracts voice features and converts them into structured vectors to efficiently verify and retrieve speakers.

AnalyticDB for MySQL provides an end-to-end voiceprint recognition solution using its native vector storage and retrieval capabilities. It supports three core features: voiceprint comparison, voiceprint retrieval, and voiceprint clustering. The solution can be extended to include speaker diarization, speech-to-text (ASR), and content quality inspection, helping you quickly build a high-precision voiceprint retrieval system.

Limitations

The voiceprint retrieval feature is currently available by invitation only. To activate this feature, submit a ticket.

Features

Voiceprint comparison

Using a built-in voiceprint model, this feature extracts voiceprint features from raw audio and converts them into structured vectors. It then calculates the similarity score between two audio vectors to verify if they are from the same speaker in a 1:1 comparison.

Voiceprint retrieval

This feature uses voiceprint feature vectors and an efficient indexing mechanism to quickly retrieve a target speaker from a pre-built voiceprint library. It supports 1:N voiceprint recognition scenarios, making it ideal for efficient identity matching in large-scale voiceprint libraries.

Voiceprint clustering

This feature uses unsupervised learning techniques to analyze unlabeled audio data and automatically classify it based on speaker identity. It groups and manages unlabeled audio from multi-person scenarios.

Usage

API

Audio embedding - `/audio/embedding`

Method: POST
Function: Generates an audio embedding vector for an audio file.

Parameters:

Parameter	Type	Required	Description
input_audio	string	Yes	The URL of the input audio file.
oss_ak	string	No	The AccessKey ID for OSS.
oss_sk	string	No	The AccessKey secret for OSS.
oss_token	string	No	The STS token for OSS.
start_time	float	No	The start timestamp.
end_time	float	No	The end timestamp.

Example request:

curl -X POST "http://addr:8100/audio/embedding" \
  -H "Authorization: Bearer {api-key}" \
  -H "Content-Type: application/json" \
  -d '{
    "input_audio": "https://dashscope.oss-cn-beijing-internal.aliyuncs.com/samples/audio/paraformer/hello_world_female2.wav",
    "start_time": 1220,
    "end_time": 3200
  }'

Example response:

{
  "code": 200,
  "message": "success",
  "result": [1.861976, 0.151182, -0.888397, ...]
}

Voiceprint enrollment - `/voice/enroll`

Method: POST
Function: Enrolls a new voiceprint sample.

Parameters:

Parameter	Type	Required	Description
audio_url	string	Yes	The URL of the input audio file.
name	string	Yes	The name of the voiceprint.

Example request:

curl -X POST "http://addr:8100/voice/enroll" \
  -H "Authorization: Bearer {api-key}" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://dashscope.oss-cn-beijing-internal.aliyuncs.com/samples/audio/paraformer/hello_world_female2.wav",
    "name": "test1"
  }'

Example response:

{"code": 200, "message": "Voiceprint enrollment successful", "result": true}

Voiceprint query - `/voice/query`

Method: GET or POST
Function: Queries enrolled voiceprint records.

Parameters:

Parameter	Type	Required	Description
name	string	No	The name of the voiceprint.
id	integer	No	The ID of the voiceprint.

Example request:

curl "http://addr:8100/voice/query" \
  -H "Authorization: Bearer {api-key}"

Example response:

{
  "code": 200,
  "message": "Found 1 voiceprint records",
  "result": [{"id": 1968033551534260224, "name": "test1", "location": null}]
}

Voiceprint deletion - `/voice/delete`

Method: DELETE or POST
Function: Deletes a specified voiceprint record.

Parameters:

Parameter	Type	Required	Description
name	string	Conditional	The name of the voiceprint. You must specify either `name` or `id`.
id	integer	Conditional	The ID of the voiceprint. You must specify either `name` or `id`.

Example request:

curl -X POST "http://addr:8100/voice/delete" \
  -H "Authorization: Bearer {api-key}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "test1"
  }'

Example response:

{"code": 200, "message": "Voiceprint deletion successful", "result": true}

Voiceprint search - `/voice/search`

Method: POST
Function: Searches for the best matching voiceprints based on an input audio file.

Parameters:

Parameter	Type	Required	Description
audio_url	string	Yes	The URL of the query audio file.
names	array	No	A list of voiceprint names to search against.
top_k	integer	No	The number of most similar results to return. The default value is 1.

Example request:

curl -X POST "http://addr:8100/voice/search" \
  -H "Authorization: Bearer {api-key}" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://dashscope.oss-cn-beijing-internal.aliyuncs.com/samples/audio/paraformer/hello_world_female2.wav",
    "top_k": 1
  }'

Example response:

{
  "code": 200,
  "message": "Found 1 matching voiceprints",
  "result": [{"name": "test1", "similarity": 0.99999994}]
}

ASR (speech-to-text) - `/bailian/funasr/asr`

Method: POST
Function: Converts speech to text using the FunASR model from Alibaba Cloud Model Studio (Bailian).

Parameters:

Parameter	Type	Required	Description
source_url	string	Yes	The path of the audio file in OSS.
model_name	string	No	The model name. The default value is `fun-asr`. For available models, see the Alibaba Cloud Model Studio (Bailian) FunASR model list.
diarization	bool	No	Specifies whether to enable speaker diarization. The default value is `true`.
speaker_count	int	No	The number of speakers. If you do not specify this parameter, the system automatically detects the number of speakers.
output_type	string	No	The output format. Set the value to `url` to return a temporary link to the result file, or set it to `json` to return structured data. The default value is `url`.
lang	string	No	The language setting. For supported languages, see the `language_hints` parameter in the Alibaba Cloud Model Studio (Bailian) FunASR request parameters.
diarization_mode	string	No	The aggregation dimension for speaker diarization results. Supported values: `word`, `sentence`, and `speaker`.

Example request:

curl -X POST "http://addr:8100/bailian/funasr/asr" \
  -H "Authorization: Bearer {api-key}" \
  -H "Content-Type: application/json" \
  -d '{
    "source_url": "https://dashscope.oss-cn-beijing-internal.aliyuncs.com/samples/audio/paraformer/hello_world_female2.wav",
    "output_type": "json",
    "diarization_mode": "sentence",
    "model_name": "fun-asr-mtl",
    "lang": "zh"
  }'

Example response:

{
  "code": 200,
  "message": "success",
  "result": {
    "file_url": "https://dashscope.oss-cn-beijing-internal.aliyuncs.com/samples/audio/paraformer/hello_world_female2.wav",
    "properties": {
      "audio_format": "pcm_s16le",
      "channels": [0],
      "original_sampling_rate": 16000,
      "original_duration_in_milliseconds": 3834
    },
    "transcripts": [
      {
        "channel_id": 0,
        "speakers": [
          {
            "speaker_id": 0,
            "sentences": [
              {
                "sentence_id": 1,
                "start_time": 100,
                "end_time": 3300,
                "text": "hello world，这里是阿里巴巴语音实验室。"
              }
            ]
          }
        ]
      }
    ]
  }
}

SQL

AnalyticDB for MySQL supports voiceprint retrieval operations directly in SQL using AI functions.

ai_audio_embed

Generates a voiceprint embedding vector for an audio file. For more information, see ai_audio_embed.

ai_audio_embed(text)
ai_audio_embed(model_name, text)
ai_audio_embed(model_name, text, options)

ai_audio_transcribe

Transcribes audio to text. For more information, see ai_audio_transcribe.

ai_audio_transcribe(url)
ai_audio_transcribe(model_name, url)
ai_audio_transcribe(model_name, url, options)

Examples

Create a voiceprint library table

CREATE DATABASE ai;

CREATE TABLE IF NOT EXISTS
  ai.voiceTest (
    id bigint NOT NULL AUTO_INCREMENT,
    name varchar NOT NULL,
    voiceprint_feature ARRAY<float>(512) ENCODE = 'no' COMPRESSION = 'no',
    ANN INDEX idx_voiceprint_feature (voiceprint_feature),
    PRIMARY KEY (id)
  ) INDEX_ALL = 'Y' STORAGE_POLICY = 'HOT' ENGINE = 'XUANWU'
    TABLE_PROPERTIES = '{"format":"columnstore"}'
    DISTRIBUTE BY HASH (id);

Enroll a voiceprint

INSERT INTO ai.voiceTest (name, voiceprint_feature)
SELECT '{name}', ai_audio_embed('{audio_file}');

Query a voiceprint

SELECT id, name FROM ai.voiceTest WHERE name = '{name}';

Delete a voiceprint

DELETE FROM ai.voiceTest WHERE name = '{name}';

Perform voiceprint retrieval

SELECT name, similarity
FROM (
    SELECT
        name,
        cosine_similarity(
            voiceprint_feature,
            ai_audio_embed('audio_embedding', '{audio_file}',
              '{''start_time'':1000, ''end_time'':3000}')
        ) AS similarity
    FROM ai.voiceTest
) t
ORDER BY similarity DESC
LIMIT 3;

Use case: Driver monitoring and content detection

Background

A ride-hailing company needed to analyze in-vehicle audio to isolate the driver's voice from conversations and check it for policy violations.

Using the AnalyticDB for MySQL voiceprint recognition solution, the company built an end-to-end pipeline that includes speaker diarization, noise reduction, speech-to-text (ASR), automatic construction of a voiceprint library, voiceprint retrieval, and content quality inspection of the transcribed text.

Workflow

Noise reduction: Pre-process the raw audio to reduce background noise and enhance human speech.
Speaker diarization: Use speaker diarization to separate and tag each speaker's voice in a multi-person conversation.
Voice segmentation: Split the audio into per-speaker segments based on the diarization results for individual analysis.
Voiceprint recognition and speech-to-text (ASR): Extract spoken content from each audio segment using voiceprint recognition and speech-to-text (ASR).
Voiceprint retrieval: Match the current audio segment to the driver's identity by querying the historical voiceprint library.
Content quality inspection: Analyze the speech-to-text transcript associated with the driver's identity using a large language model (LLM) to detect policy violations.

Background

Limitations

Features

Voiceprint comparison

Voiceprint retrieval

Voiceprint clustering

Usage

API

Audio embedding - /audio/embedding

Voiceprint enrollment - /voice/enroll

Voiceprint query - /voice/query

Voiceprint deletion - /voice/delete

Voiceprint search - /voice/search

ASR (speech-to-text) - /bailian/funasr/asr

SQL

ai_audio_embed

ai_audio_transcribe

Examples

Use case: Driver monitoring and content detection

Background

Workflow

Audio embedding - `/audio/embedding`

Voiceprint enrollment - `/voice/enroll`

Voiceprint query - `/voice/query`

Voiceprint deletion - `/voice/delete`

Voiceprint search - `/voice/search`

ASR (speech-to-text) - `/bailian/funasr/asr`