Real-time speech recognition SDK for WeChat mini programs-Intelligent Speech Interaction(ISI)-阿里云帮助中心

The Intelligent Speech Interaction SDK for WeChat mini programs recognizes recorded speech in real time and returns intermediate results, sentence results, and task completion messages through callbacks. This topic describes SDK integration, a code example, and the API.

Prerequisites

Activate Intelligent Speech Interaction, create a project in the console, and obtain its Appkey.
Install WeChat DevTools, configure a mini program project, obtain recording permission, and configure the privacy notice required by WeChat.
Provide an HTTPS endpoint on an application server to issue NLS tokens. In the mini program backend, add its HTTPS domain to the allowed request domains and wss://nls-gateway-cn-shanghai.aliyuncs.com to the allowed socket domains.

For the service protocol, audio requirements, and recognition parameters, see Real-time speech recognition API reference.

Usage notes

The audio format and sample rate must match the recognition parameters. The example uses mono PCM audio at 16 kHz. The WeChat recorder supports up to 10 minutes per recording.

Important

Handle the failed and closed callbacks and set application-side timeouts. On failure, stop recording and clean up the connection and page state. shutdown() only disconnects the connection. To complete recognition normally, supply all audio and then call close().

Download and install

Download the SDK from the GitHub repository, or download alibabacloud-nls-wx-sdk-master.zip. Copy the utils directory into the mini program project and import SpeechTranscription with require("../../utils/st"). Adjust the relative path for the page location.

Obtain a token

Obtain NLS tokens on an application server and provide them to the mini program through an authenticated HTTPS endpoint. For instructions, see Obtain a token.

Keep the AccessKey ID and AccessKey secret on the server. Do not store them in mini program code or local storage. The server can cache tokens until shortly before they expire to avoid requesting a new token for every call.

Code example

The example starts recognition before recording and sending audio frames. When recording stops manually or automatically, it waits for the last frame and the recording-stop event before calling close(). The timeouts in the example are application settings and can be adjusted as needed.

Replace APPKEY and TOKEN_URL and add the authentication required by the application server to the getToken() request. TOKEN_URL is an application endpoint, not an NLS OpenAPI endpoint. The example expects token (a string) and expireTime (a Unix timestamp in seconds). It requests a valid token from the application server before each recognition session.

pages/st/st.js

const SpeechTranscription = require("../../utils/st");
const APPKEY = "YOUR_APPKEY";
const TOKEN_URL = "https://your-server.example.com/nls/token";
const GATEWAY = "wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1";

function withTimeout(promise, milliseconds) {
  let timer;
  return Promise.race([
    promise,
    new Promise((_, reject) => {
      timer = setTimeout(() => reject(new Error("Operation timed out")), milliseconds);
    })
  ]).finally(() => clearTimeout(timer));
}

function getToken() {
  return new Promise((resolve, reject) => {
    wx.request({
      url: TOKEN_URL,
      method: "GET",
      timeout: 10000,
      // Add the authentication required by your application server.
      success(res) {
        const value = res.data;
        if (res.statusCode === 200 && value && typeof value.token === "string"
            && value.token && Number(value.expireTime) > Date.now() / 1000 + 60) {
          resolve(value.token);
        } else {
          reject(new Error("No valid token returned"));
        }
      },
      fail: reject
    });
  });
}

Page({
  data: { busy: false, recording: false, result: "", error: "" },

  onLoad() {
    this.recorder = wx.getRecorderManager();
    const listen = (name, handler) => {
      this[name] = (...args) => { if (!this.unloaded) handler(...args); };
      this.recorder[name](this[name]);
    };
    listen("onStart", () => {
      const session = this.session;
      if (!session) { this.recorder.stop(); return; }
      clearTimeout(session.startTimer);
      this.setData({ recording: true });
    });
    listen("onFrameRecorded", (frame) => {
      const session = this.session;
      if (!session || !session.st || session.closing) return;
      try {
        if (frame.frameBuffer.byteLength && !session.st.sendAudio(frame.frameBuffer)) {
          throw new Error("Audio connection unavailable");
        }
        if (frame.isLastFrame) session.lastFrame = true;
        this.finishRecording(session);
      } catch (error) { this.abort(error.message); }
    });
    listen("onStop", () => {
      this.recorderBusy = false;
      const session = this.session;
      if (!session) return;
      clearTimeout(session.startTimer);
      session.stopped = true;
      this.setData({ recording: false });
      this.waitForLastFrame(session);
      this.finishRecording(session);
    });
    listen("onError", () => {
      this.recorderBusy = false;
      this.abort("Recording failed. Check microphone permission.");
    });
  },

  async onStStart() {
    if (this.session || this.unloaded) return;
    if (this.recorderBusy) {
      this.setData({ error: "Recorder has not stopped. Reopen this page before retrying." });
      return;
    }
    const session = { sentences: {} };
    this.session = session;
    this.setData({ busy: true, recording: false, result: "", error: "" });
    try {
      const token = await getToken();
      if (this.session !== session) return;
      const st = new SpeechTranscription({ url: GATEWAY, appkey: APPKEY, token });
      session.st = st;
      const showResult = (message) => {
        if (this.session !== session) return;
        const payload = JSON.parse(message).payload;
        session.sentences[payload.index] = payload.result || "";
        const result = Object.keys(session.sentences)
          .sort((a, b) => Number(a) - Number(b))
          .map(index => session.sentences[index]).join("\n");
        this.setData({ result });
      };
      st.on("changed", showResult);
      st.on("end", showResult);
      st.on("completed", (message) => {
        if (this.session !== session) return;
        session.completed = true;
        console.log("task_id:", JSON.parse(message).header.task_id);
      });
      st.on("failed", (message) => {
        if (this.session !== session) return;
        const header = JSON.parse(message).header;
        console.error("status:", header.status, "task_id:", header.task_id);
        this.abort("Recognition failed. Check the status code.");
      });
      st.on("closed", () => {
        if (this.session === session && !session.completed) {
          this.abort("Connection closed before recognition completed.");
        }
      });
      await withTimeout(st.start(st.defaultStartParams()), 10000);
      if (this.session !== session) return;
      this.recorderBusy = true;
      session.startTimer = setTimeout(() => {
        if (this.session === session) this.abort("Timed out while starting recording.");
      }, 10000);
      try {
        this.recorder.start({
          duration: 600000,
          numberOfChannels: 1,
          sampleRate: 16000,
          format: "PCM",
          frameSize: 4
        });
      } catch (error) { this.recorderBusy = false; throw error; }
    } catch (error) {
      if (this.session === session) this.abort(error.message || "Unable to start recognition.");
    }
  },

  onStStop() {
    const session = this.session;
    if (!session || !this.data.recording || session.stopRequested) return;
    session.stopRequested = true;
    this.setData({ recording: false });
    this.waitForLastFrame(session);
    this.recorder.stop();
  },

  waitForLastFrame(session) {
    if (session.timer) return;
    session.timer = setTimeout(() => {
      if (this.session === session) this.abort("Timed out while stopping recording.");
    }, 5000);
  },

  async finishRecording(session) {
    if (!session.stopped || !session.lastFrame || session.closing) return;
    session.closing = true;
    clearTimeout(session.timer);
    try {
      await withTimeout(session.st.close(), 10000);
      if (this.session === session) {
        this.session = null;
        this.setData({ busy: false, recording: false });
      }
    } catch (error) {
      if (this.session === session) this.abort(error.message || "Unable to finish recognition.");
    }
  },

  abort(message) {
    const session = this.session;
    this.session = null;
    if (session) {
      clearTimeout(session.timer);
      clearTimeout(session.startTimer);
      if (session.st) session.st.shutdown();
    }
    if (this.recorderBusy) this.recorder.stop();
    if (!this.unloaded) this.setData({ busy: false, recording: false, error: message || "" });
  },

  onHide() { this.abort(); },
  onUnload() {
    this.unloaded = true;
    this.abort();
    for (const name of ["onStart", "onFrameRecorded", "onStop", "onError"]) {
      const off = this.recorder["off" + name.slice(2)];
      if (typeof off === "function") off.call(this.recorder, this[name]);
    }
  }
});

pages/st/st.wxml

<view>
  <button bindtap="onStStart" disabled="{{busy}}">Start recognition</button>
  <button bindtap="onStStop" disabled="{{!recording}}">Stop recognition</button>
  <text>{{result}}</text>
  <text>{{error}}</text>
</view>

result updates recognition text by sentence index without repeatedly appending intermediate results. error displays error messages. Service error codes and task IDs are logged, but tokens are not.

API reference

SpeechTranscription

Create a real-time speech recognition object with new SpeechTranscription(config). The config argument is a connection configuration object.

config fields

Parameter	Type	Description
`url`	String	The WebSocket endpoint. The example uses `wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1`.
`token`	String	A valid NLS token provided by the application server.
`appkey`	String	The Appkey of the Intelligent Speech Interaction project.

defaultStartParams()

Takes no arguments and returns the default recognition parameters: PCM at 16 kHz, with intermediate results, punctuation prediction, and ITN enabled. Modify the object as needed based on the Real-time speech recognition API reference.

{
  "format": "pcm",
  "sample_rate": 16000,
  "enable_intermediate_result": true,
  "enable_punctuation_prediction": true,
  "enable_inverse_text_normalization": true
}

on(which, handler)

Sets an event callback. Calling on again for the same event replaces its previous callback. Returns no value.

Parameter	Type	Description
`which`	String	The event name.
`handler`	Function	The callback function.

Event	Description	Argument count	Callback argument
`started`	Recognition has started.	1	String: the start message.
`changed`	An intermediate sentence result.	1	String: the intermediate result message.
`begin`	A sentence start was detected.	1	String: the sentence-start message.
`end`	The sentence ended; its final result is available.	1	String: the sentence-result message.
`completed`	The recognition task is complete.	1	String: the task-completion message.
`closed`	The connection is closed.	0	None.
`failed`	A service task error was received.	1	String: the error message.

String messages contain JSON and can be parsed with JSON.parse. end indicates a sentence end, whereas completed indicates task completion. A task can complete without a sentence result if there is no valid speech.

async start(param)

Starts real-time speech recognition. param is an Object that can be created with defaultStartParams() and modified as needed. Returns a Promise that resolves with the start message string when started occurs.

Connection setup failures can reject the Promise. If TaskFailed is received during startup, the error is reported through the failed callback, but the Promise returned by start() may remain pending. Handle failed and closed and set an application-side timeout. For recognition parameters, see Real-time speech recognition API reference.

async close(param)

Sends a request to complete real-time speech recognition. param is an optional Object and is omitted in the example. Returns a Promise. After receiving task completion, the SDK triggers completed, closes the connection, and resolves the Promise with the completion message string.

The Promise rejects if the client is null or a TaskFailed message arrives while waiting. If the connection closes while waiting, handle it through the closed callback. The Promise may remain pending, so set an application-side timeout. Supply all audio before calling this method; do not replace normal completion with shutdown().

shutdown()

Forcibly disconnects without waiting for task completion. Takes no arguments and returns no value. Use it to clean up after errors or when leaving the page.

sendAudio(data)

Sends audio. data is an ArrayBuffer containing binary audio whose format and sample rate match the recognition parameters.

Returns a Boolean: true if the client object exists, or false otherwise. This return value does not indicate whether the server received the audio.