Key point extraction (action items, keywords, and highlights)

更新时间:
复制 MD 格式

This topic describes the key point extraction feature and how to implement it. This feature uses multiple Artificial Intelligence (AI) models and algorithms to extract results such as keywords, highlights, action items, and scene classification from audio files, video files, and real-time meetings.

Request parameters

Parameter name

Type

Required

Description

MeetingAssistanceEnabled

boolean

No

The default value is false.

MeetingAssistance

list[]

No

The default value is empty. Add algorithm types as needed. The following types are supported:

  • Actions: Action items

  • KeyInformation: Key information (includes keywords and highlights)

For descriptions of other request parameters, see the document for your scenario:

Note:

  • If you set `MeetingAssistanceEnabled` to `true` but do not add any algorithm types, the model processes all supported algorithms, namely `Actions` and `KeyInformation`.

  • If you need only one type of output, such as action items, you can enable `MeetingAssistanceEnabled` and add only `Actions`.

Example settings

{
    "Input":{
        ...
    },
    "Parameters":{
        ...
        "MeetingAssistanceEnabled":true,
        "MeetingAssistance":{
            "Types":[
                "Actions",
                "KeyInformation"
            ]
        },
        ...
    }
}

Code examples

#!/usr/bin/env python
#coding=utf-8

import os
import json
import datetime
from aliyunsdkcore.client import AcsClient
from aliyunsdkcore.request import CommonRequest
from aliyunsdkcore.auth.credentials import AccessKeyCredential

def create_common_request(domain, version, protocolType, method, uri):
    request = CommonRequest()
    request.set_accept_format('json')
    request.set_domain(domain)
    request.set_version(version)
    request.set_protocol_type(protocolType)
    request.set_method(method)
    request.set_uri_pattern(uri)
    request.add_header('Content-Type', 'application/json')
    return request

def init_parameters():
    root = dict()
    root['AppKey'] = 'Enter the AppKey that you created in the Tingwu console'

    # Basic request parameters
    input = dict()
    input['SourceLanguage'] = 'cn'
    input['TaskKey'] = 'task' + datetime.datetime.now().strftime('%Y%m%d%H%M%S')
    input['FileUrl'] = 'Enter the URL of the audio file to test'
    root['Input'] = input

    # AI-related parameters. Set them as needed.
    parameters = dict()
    # Key point extraction
    parameters['MeetingAssistanceEnabled'] = True
    meetingAssistance = dict()
    meetingAssistance['Types'] = ['Actions', 'KeyInformation']
    parameters['MeetingAssistance'] = meetingAssistance
    root['Parameters'] = parameters
    return root

body = init_parameters()
print(body)

# TODO: Set your AccessKeyId and AccessKeySecret using environment variables.
credentials = AccessKeyCredential(os.environ['ALIBABA_CLOUD_ACCESS_KEY_ID'], os.environ['ALIBABA_CLOUD_ACCESS_KEY_SECRET'])
client = AcsClient(region_id='cn-beijing', credential=credentials)

request = create_common_request('tingwu.cn-beijing.aliyuncs.com', '2023-09-30', 'https', 'PUT', '/openapi/tingwu/v2/tasks')
request.add_query_param('type', 'offline')

request.set_content(json.dumps(body).encode('utf-8'))
response = client.do_action_with_exception(request)
print("response: \n" + json.dumps(json.loads(response), indent=4, ensure_ascii=False))
package com.alibaba.tingwu.client.demo.aitest;

import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.CommonRequest;
import com.aliyuncs.CommonResponse;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.MethodType;
import com.aliyuncs.http.ProtocolType;
import com.aliyuncs.profile.DefaultProfile;
import org.junit.Test;

/**
 * @author tingwu2023
 */
public class MeetingAssistanceTest {

    @Test
    public void testMeetingAssistance() throws ClientException {
        CommonRequest request = createCommonRequest("tingwu.cn-beijing.aliyuncs.com", "2023-09-30", ProtocolType.HTTPS, MethodType.PUT, "/openapi/tingwu/v2/tasks");
        request.putQueryParameter("type", "offline");

        JSONObject root = new JSONObject();
        root.put("AppKey", "Enter the AppKey that you created in the Tingwu console");

        JSONObject input = new JSONObject();
        input.fluentPut("FileUrl", "Enter the URL of the audio file to test")
                .fluentPut("SourceLanguage", "cn")
                .fluentPut("TaskKey", "task" + System.currentTimeMillis());
        root.put("Input", input);

        JSONObject parameters = new JSONObject();
        parameters.put("MeetingAssistanceEnabled", true);
      	JSONObject meetingAssistance = new JSONObject();
        JSONArray mTypes = new JSONArray().fluentAdd("Actions").fluentAdd("Information");
        meetingAssistance.put("Types", mTypes);
        parameters.put("MeetingAssistance", meetingAssistance);
        root.put("Parameters", parameters);
        System.out.println(root.toJSONString());
        request.setHttpContent(root.toJSONString().getBytes(), "utf-8", FormatType.JSON);

        // TODO: Set your AccessKeyId and AccessKeySecret using environment variables.
        DefaultProfile profile = DefaultProfile.getProfile("cn-beijing", System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"), System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"));
        IAcsClient client = new DefaultAcsClient(profile);
        CommonResponse response = client.getCommonResponse(request);
        System.out.println(response.getData());
    }
    public static CommonRequest createCommonRequest(String domain, String version, ProtocolType protocolType, MethodType method, String uri) {
        // Create an API request and set the parameters.
        CommonRequest request = new CommonRequest();
        request.setSysDomain(domain);
        request.setSysVersion(version);
        request.setSysProtocol(protocolType);
        request.setSysMethod(method);
        request.setSysUriPattern(uri);
        request.setHttpContentType(FormatType.JSON);
        return request;
    }
}

Example output

{
    "Code":"0",
    "Data":{
        "TaskId":"8b78c180e0994e9097e9135*********",
        "TaskStatus":"COMPLETED",
        "Result":{
            "MeetingAssistance":"http://speech-swap-hangzhou.oss-cn-hangzhou.aliyuncs.com/tingwu/output/1503864348104017/8b78c180e0994e9097e913577ecba19b/8b78c180e0994e9097e913577ecba19b_MeetingAssistance_20231029212015.json?Expires=1698672195&OSSAccessKeyId=LTAI****************&Signature=********ZkK%2BC%2FS%2F0Fl9r35s%2Ffc%3D",
            "Transcription":"http://speech-swap-hangzhou.oss-cn-hangzhou.aliyuncs.com/tingwu/output/1503864348104017/8b78c180e0994e9097e913577ecba19b/8b78c180e0994e9097e913577ecba19b_Transcription_20231029212000.json?Expires=1698672195&OSSAccessKeyId=LTAI****************&Signature=********k5QRCL%2BTCfwF2Zrb0%3D"
        }
    },
    "Message":"success",
    "RequestId":"784E7780-E6EB-1F55-A3R9-23*********"
}

The `MeetingAssistance` field contains the HTTP URL to download the key point extraction results.

Protocol parsing

The URL for the key point extraction result contains a JSON message. The following example shows the message format.

{
    "TaskId":"8b78c180e034fe9097e9135s7ebba1fa",
    "MeetingAssistance":{
        "Keywords":[
            "DAMO Academy",
            "Alibaba",   
            "speech"
        ],
        "KeySentences":[
            {
                "Id":1,
                "SentenceId":1,
                "Start":31680,
                "End":36582,
                "Text":"First, I will introduce our work and job requirements."
            },
            {
                "Id":2,
                "SentenceId":45,
                "Start":1452950,
                "End":1462184,
                "Text":"The company mainly focuses on speech, from the speech lab, and primarily develops speech-to-text and speech-related Alibaba Cloud services."
            }
        ],
        "Actions":[
            {
                "Id":1,
                "SentenceId":8,
                "Start":39654,
                "End":52117,
                "Text":"Confirm if there are any issues with the content in the PPT template."
            },
            {
                "Id":2,
                "SentenceId":18,
                "Start":84693,
                "End":86786,
                "Text":"Follow the trial status and upcoming release of Tingwu public cloud."
            }
        ],
        "Classifications":{
            "Interview":0.6549709,
            "Lecture":0.18346232,
            "Meeting":0.16156682
        }
    }
}

The following describes the fields.

Parameter name

Type

Description

TaskId

string

The ID of the task that is generated when the task is created.

MeetingAssistance

object

The key point extraction result object. It can contain zero or more types of results.

MeetingAssistance.Keywords

list[]

The keyword extraction result.

MeetingAssistance.KeySentences

list[]

The key sentence extraction result, also known as highlights.

MeetingAssistance.KeySentences[i].Id

long

The ordinal number of the key sentence.

MeetingAssistance.KeySentences[i].SentenceId

long

The ID of the corresponding sentence in the original Automatic Speech Recognition (ASR) transcription.

MeetingAssistance.KeySentences[i].Start

long

The start time relative to the beginning of the audio. This is a relative timestamp in milliseconds.

MeetingAssistance.KeySentences[i].End

long

The end time relative to the beginning of the audio. This is a relative timestamp in milliseconds.

MeetingAssistance.KeySentences[i].Text

string

The key sentence text.

MeetingAssistance.Actions

list[]

A collection of action items and summaries.

MeetingAssistance.Actions[i].Id

long

The ordinal number of the action item.

MeetingAssistance.Actions[i].SentenceId

long

The ID of the corresponding sentence in the original ASR transcription.

MeetingAssistance.Actions[i].Start

long

The start time relative to the beginning of the audio. This is a relative timestamp in milliseconds.

MeetingAssistance.Actions[i].End

long

The end time relative to the beginning of the audio. This is a relative timestamp in milliseconds.

MeetingAssistance.Actions[i].Text

string

The action item text.

MeetingAssistance.Classifications

object

Scene classification. Currently, only three scene categories are available.

MeetingAssistance.Classifications.Interview

float

The confidence level score for the interview scene.

MeetingAssistance.Classifications.Lecture

float

The confidence level score for the lecture scene.

MeetingAssistance.Classifications.Meeting

float

The confidence level score for the meeting scene.

FAQ

When can I call the key point extraction feature?

  • You can set it directly when you create an offline transcription task for an audio or video file, or when you create a real-time meeting.

  • After an offline transcription or real-time meeting ends, you can also submit a request to rerun the task to generate the key points. You must use the same TaskId.

Why are the key point extraction results not generated or empty after I call the feature?

  • The request parameters may be disabled or configured incorrectly. Compare your settings with the documentation and ensure they are configured correctly.

  • The language model you selected may not support key point extraction. For example, this feature is not currently supported for Japanese.

  • The audio or video file may not contain enough information. For example, if the ASR result contains too little text, the model cannot generate key points. This can happen because of excessive background noise or poor audio quality. To resolve this, test with an audio or video file that contains more useful information.

Why do the generated key points not include highlights or action items?

  • The key point extraction feature is designed to produce several types of results, but it may not generate all types for every task. For example, if the transcribed text contains too little information, the output might only include keywords, or no results at all. Similarly, even with sufficient information, if no valid action items are detected, the action item results will not be generated.