API reference

Updated at:

Sound event detection identifies sounds such as background music, crying, laughter, and explosions in audio and returns their start and end times.

Sound event types

The following sound event types are supported:

  • Music: background music.

  • Singing: singing.

  • Laughter: laughter.

  • Knock: knocking on a door or table.

  • Keyboard: keyboard sounds.

  • Cry: crying.

  • Explosion: explosions.

  • Water: water sounds.

  • SirenAlarm: sirens.

Limits

  • The input must be an uncompressed PCM file or a PCM-encoded WAV file with a 16-bit audio bit depth and one channel.

  • The audio sample rate must be 16,000 Hz.

  • Audio must be sent as a real-time stream.

Billing and concurrency limits

Sound event detection is available in the Free Edition and Commercial Edition. For service editions and billable items, see Billable items. For billing methods and upgrade instructions, see Billing methods. For concurrency limits, see Concurrency and QPS.

Endpoints

Access method

Description

Endpoint

Public network

Available to all servers over the public network.

wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1

Internal network from Shanghai ECS

For ECS instances in the China (Shanghai) region through a VPC. The classic network does not support access to the speech service.

ws://nls-gateway-cn-shanghai-internal.aliyuncs.com:80/ws/v1

The Java SDK's NlsClient(String token) constructor uses the Shanghai public endpoint by default. To use the internal network, explicitly pass the internal endpoint when creating NlsClient. Internal network access does not incur public data transfer fees for the ECS instance.

Interaction flow

image

1. Authentication

The client uses an NLS token to authenticate the WebSocket connection. For token acquisition, see Obtain an access token. To obtain a project Appkey, see Manage projects. The Appkey and token must belong to the same account.

2. Start detection

In the Java SDK, pass the namespace to the CommonRequest constructor, set the Appkey with setAppKey(), and set audio parameters with addCustomedParam(). The parameters are described below:

Parameter

Type

Description

namespace

String

The namespace. Set to AudioEventDetection when creating CommonRequest.

format

String

The audio format. Valid values: pcm and wav. Default: pcm.

sample_rate

Integer

The audio sample rate in Hz. Default: 16000.

For a complete example of client initialization, listener implementation, and audio transmission, see Java SDK. The following snippet assumes that NlsClient client and CommonRequestListener listener have been created and that appkey contains the project Appkey:

CommonRequest request = new CommonRequest(client, listener, "AudioEventDetection");
request.setAppKey(appkey);
request.addCustomedParam("format", "pcm");
request.addCustomedParam("sample_rate", 16000);
request.start();

Call start() to start the task. When the service returns TaskStarted, the SDK invokes onStarted. Task failures are reported through onFailed.

3. Receive detection results

Send audio data and receive sound event detection results through CommonRequestListener.onEvent. Each TaskResult message can contain a list of sound events. Example response:

{
    "header":{
        "namespace":"AudioEventDetection",
        "name":"TaskResult",
        "status":20000000,
        "message_id":"1727504541a143e2bf8cfc4037d4****",
        "task_id":"f926a956efaf4d30a823abafe97d****",
        "status_text":"Gateway:SUCCESS:Success."
    },
    "payload":{
        "number":1,
        "result":[
            {
                "confidence":0.81,
                "end_time":32000,
                "event_type":"Music",
                "start_time":30000
            }
        ]
    }
}

header object

Parameter

Type

Description

namespace

String

The namespace of the message.

name

String

The message name. TaskResult indicates a sound event detection result.

status

Integer

The service status code. 20000000 indicates success. For error handling, see Service status codes.

status_text

String

The status message.

task_id

String

The globally unique task ID. Use it for troubleshooting.

message_id

String

The ID of the current message.

payload object

Parameter

Type

Description

number

Integer

The number of sound events in the current result.

result

Array

The list of sound events. The fields of each element are described below.

Elements of the result array

Parameter

Type

Description

start_time

Integer

The start time of the sound event relative to the beginning of the input audio, in milliseconds.

end_time

Integer

The end time of the sound event relative to the beginning of the input audio, in milliseconds.

event_type

String

The sound event type. See the list of sound event types above.

confidence

Float

The confidence score of the current result. Valid values: [0.0, 1.0]. A higher value indicates greater confidence.

4. End detection

After all audio data has been sent, call stop() to send StopTask. When processing is complete, the service returns TaskCompleted and the SDK invokes onStopped, indicating that the detection task has ended.

Service status codes

The header.status field indicates the service status. Handle errors according to the status code and status_text. For more information about common errors, see Error codes.

General error codes

Error code

Cause

Solution

40000001

Authentication failed.

Check whether the token is correct and has expired.

40000002

Invalid message.

Check whether the message meets the API requirements.

40000004

Idle timeout.

Check whether no data has been sent to the service for 10 consecutive seconds.

40000005

Too many requests.

Check whether the concurrency or requests-per-second limit has been exceeded. If the concurrency limit has been exceeded, upgrade to the Commercial Edition or increase its concurrency capacity as needed.

40000000

Default client error.

Troubleshoot according to the status message.

40000010

The free trial has expired.

To continue using the service, activate the Commercial Edition. For instructions, see Billing methods.

41010120

Client timeout.

The client has sent no data for 10 seconds or longer. Check whether audio transmission has been interrupted.

41160001

Invalid sample rate.

Check the sample rate against the input requirements. It must be 16,000 Hz.

41160002

Invalid audio format.

Check the audio format parameter.

41160003

Audio decoding failed.

Check whether the audio format is valid.

Gateway error codes

Error code

Cause

Solution

40010001

Unsupported API.

Check whether the namespace is correct and the endpoint provides the service.

40010002

Unsupported directive.

Check whether the directive name meets the API requirements.

40010003

Invalid directive.

Check the directive format and parameters.

40010004

The client disconnected prematurely.

Check whether the connection was closed before the task completed.

40010005

Invalid task state.

Check whether the directive can be processed in the current task state.

Configuration error codes

Error code

Cause

Solution

40020105

The application does not exist.

Check whether the project Appkey exists.

40020106

The Appkey and token do not match.

Check whether the Appkey is correct and belongs to the same account as the token.

40020503

RAM user authentication failed.

Use the Alibaba Cloud account to grant the RAM user permissions to access the POP API.