This topic describes the key point extraction feature and how to implement it. This feature uses multiple Artificial Intelligence (AI) models and algorithms to extract results such as keywords, highlights, action items, and scene classification from audio files, video files, and real-time meetings.
Request parameters
Parameter name | Type | Required | Description |
MeetingAssistanceEnabled | boolean | No | The default value is false. |
MeetingAssistance | list[] | No | The default value is empty. Add algorithm types as needed. The following types are supported:
|
For descriptions of other request parameters, see the document for your scenario:
Note:
If you set `MeetingAssistanceEnabled` to `true` but do not add any algorithm types, the model processes all supported algorithms, namely `Actions` and `KeyInformation`.
If you need only one type of output, such as action items, you can enable `MeetingAssistanceEnabled` and add only `Actions`.
Example settings
{
"Input":{
...
},
"Parameters":{
...
"MeetingAssistanceEnabled":true,
"MeetingAssistance":{
"Types":[
"Actions",
"KeyInformation"
]
},
...
}
}Code examples
#!/usr/bin/env python
#coding=utf-8
import os
import json
import datetime
from aliyunsdkcore.client import AcsClient
from aliyunsdkcore.request import CommonRequest
from aliyunsdkcore.auth.credentials import AccessKeyCredential
def create_common_request(domain, version, protocolType, method, uri):
request = CommonRequest()
request.set_accept_format('json')
request.set_domain(domain)
request.set_version(version)
request.set_protocol_type(protocolType)
request.set_method(method)
request.set_uri_pattern(uri)
request.add_header('Content-Type', 'application/json')
return request
def init_parameters():
root = dict()
root['AppKey'] = 'Enter the AppKey that you created in the Tingwu console'
# Basic request parameters
input = dict()
input['SourceLanguage'] = 'cn'
input['TaskKey'] = 'task' + datetime.datetime.now().strftime('%Y%m%d%H%M%S')
input['FileUrl'] = 'Enter the URL of the audio file to test'
root['Input'] = input
# AI-related parameters. Set them as needed.
parameters = dict()
# Key point extraction
parameters['MeetingAssistanceEnabled'] = True
meetingAssistance = dict()
meetingAssistance['Types'] = ['Actions', 'KeyInformation']
parameters['MeetingAssistance'] = meetingAssistance
root['Parameters'] = parameters
return root
body = init_parameters()
print(body)
# TODO: Set your AccessKeyId and AccessKeySecret using environment variables.
credentials = AccessKeyCredential(os.environ['ALIBABA_CLOUD_ACCESS_KEY_ID'], os.environ['ALIBABA_CLOUD_ACCESS_KEY_SECRET'])
client = AcsClient(region_id='cn-beijing', credential=credentials)
request = create_common_request('tingwu.cn-beijing.aliyuncs.com', '2023-09-30', 'https', 'PUT', '/openapi/tingwu/v2/tasks')
request.add_query_param('type', 'offline')
request.set_content(json.dumps(body).encode('utf-8'))
response = client.do_action_with_exception(request)
print("response: \n" + json.dumps(json.loads(response), indent=4, ensure_ascii=False))package com.alibaba.tingwu.client.demo.aitest;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.CommonRequest;
import com.aliyuncs.CommonResponse;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.MethodType;
import com.aliyuncs.http.ProtocolType;
import com.aliyuncs.profile.DefaultProfile;
import org.junit.Test;
/**
* @author tingwu2023
*/
public class MeetingAssistanceTest {
@Test
public void testMeetingAssistance() throws ClientException {
CommonRequest request = createCommonRequest("tingwu.cn-beijing.aliyuncs.com", "2023-09-30", ProtocolType.HTTPS, MethodType.PUT, "/openapi/tingwu/v2/tasks");
request.putQueryParameter("type", "offline");
JSONObject root = new JSONObject();
root.put("AppKey", "Enter the AppKey that you created in the Tingwu console");
JSONObject input = new JSONObject();
input.fluentPut("FileUrl", "Enter the URL of the audio file to test")
.fluentPut("SourceLanguage", "cn")
.fluentPut("TaskKey", "task" + System.currentTimeMillis());
root.put("Input", input);
JSONObject parameters = new JSONObject();
parameters.put("MeetingAssistanceEnabled", true);
JSONObject meetingAssistance = new JSONObject();
JSONArray mTypes = new JSONArray().fluentAdd("Actions").fluentAdd("Information");
meetingAssistance.put("Types", mTypes);
parameters.put("MeetingAssistance", meetingAssistance);
root.put("Parameters", parameters);
System.out.println(root.toJSONString());
request.setHttpContent(root.toJSONString().getBytes(), "utf-8", FormatType.JSON);
// TODO: Set your AccessKeyId and AccessKeySecret using environment variables.
DefaultProfile profile = DefaultProfile.getProfile("cn-beijing", System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"), System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"));
IAcsClient client = new DefaultAcsClient(profile);
CommonResponse response = client.getCommonResponse(request);
System.out.println(response.getData());
}
public static CommonRequest createCommonRequest(String domain, String version, ProtocolType protocolType, MethodType method, String uri) {
// Create an API request and set the parameters.
CommonRequest request = new CommonRequest();
request.setSysDomain(domain);
request.setSysVersion(version);
request.setSysProtocol(protocolType);
request.setSysMethod(method);
request.setSysUriPattern(uri);
request.setHttpContentType(FormatType.JSON);
return request;
}
}Example output
{
"Code":"0",
"Data":{
"TaskId":"8b78c180e0994e9097e9135*********",
"TaskStatus":"COMPLETED",
"Result":{
"MeetingAssistance":"http://speech-swap-hangzhou.oss-cn-hangzhou.aliyuncs.com/tingwu/output/1503864348104017/8b78c180e0994e9097e913577ecba19b/8b78c180e0994e9097e913577ecba19b_MeetingAssistance_20231029212015.json?Expires=1698672195&OSSAccessKeyId=LTAI****************&Signature=********ZkK%2BC%2FS%2F0Fl9r35s%2Ffc%3D",
"Transcription":"http://speech-swap-hangzhou.oss-cn-hangzhou.aliyuncs.com/tingwu/output/1503864348104017/8b78c180e0994e9097e913577ecba19b/8b78c180e0994e9097e913577ecba19b_Transcription_20231029212000.json?Expires=1698672195&OSSAccessKeyId=LTAI****************&Signature=********k5QRCL%2BTCfwF2Zrb0%3D"
}
},
"Message":"success",
"RequestId":"784E7780-E6EB-1F55-A3R9-23*********"
}The `MeetingAssistance` field contains the HTTP URL to download the key point extraction results.
Protocol parsing
The URL for the key point extraction result contains a JSON message. The following example shows the message format.
{
"TaskId":"8b78c180e034fe9097e9135s7ebba1fa",
"MeetingAssistance":{
"Keywords":[
"DAMO Academy",
"Alibaba",
"speech"
],
"KeySentences":[
{
"Id":1,
"SentenceId":1,
"Start":31680,
"End":36582,
"Text":"First, I will introduce our work and job requirements."
},
{
"Id":2,
"SentenceId":45,
"Start":1452950,
"End":1462184,
"Text":"The company mainly focuses on speech, from the speech lab, and primarily develops speech-to-text and speech-related Alibaba Cloud services."
}
],
"Actions":[
{
"Id":1,
"SentenceId":8,
"Start":39654,
"End":52117,
"Text":"Confirm if there are any issues with the content in the PPT template."
},
{
"Id":2,
"SentenceId":18,
"Start":84693,
"End":86786,
"Text":"Follow the trial status and upcoming release of Tingwu public cloud."
}
],
"Classifications":{
"Interview":0.6549709,
"Lecture":0.18346232,
"Meeting":0.16156682
}
}
}The following describes the fields.
Parameter name | Type | Description |
TaskId | string | The ID of the task that is generated when the task is created. |
MeetingAssistance | object | The key point extraction result object. It can contain zero or more types of results. |
MeetingAssistance.Keywords | list[] | The keyword extraction result. |
MeetingAssistance.KeySentences | list[] | The key sentence extraction result, also known as highlights. |
MeetingAssistance.KeySentences[i].Id | long | The ordinal number of the key sentence. |
MeetingAssistance.KeySentences[i].SentenceId | long | The ID of the corresponding sentence in the original Automatic Speech Recognition (ASR) transcription. |
MeetingAssistance.KeySentences[i].Start | long | The start time relative to the beginning of the audio. This is a relative timestamp in milliseconds. |
MeetingAssistance.KeySentences[i].End | long | The end time relative to the beginning of the audio. This is a relative timestamp in milliseconds. |
MeetingAssistance.KeySentences[i].Text | string | The key sentence text. |
MeetingAssistance.Actions | list[] | A collection of action items and summaries. |
MeetingAssistance.Actions[i].Id | long | The ordinal number of the action item. |
MeetingAssistance.Actions[i].SentenceId | long | The ID of the corresponding sentence in the original ASR transcription. |
MeetingAssistance.Actions[i].Start | long | The start time relative to the beginning of the audio. This is a relative timestamp in milliseconds. |
MeetingAssistance.Actions[i].End | long | The end time relative to the beginning of the audio. This is a relative timestamp in milliseconds. |
MeetingAssistance.Actions[i].Text | string | The action item text. |
MeetingAssistance.Classifications | object | Scene classification. Currently, only three scene categories are available. |
MeetingAssistance.Classifications.Interview | float | The confidence level score for the interview scene. |
MeetingAssistance.Classifications.Lecture | float | The confidence level score for the lecture scene. |
MeetingAssistance.Classifications.Meeting | float | The confidence level score for the meeting scene. |
FAQ
When can I call the key point extraction feature?
You can set it directly when you create an offline transcription task for an audio or video file, or when you create a real-time meeting.
After an offline transcription or real-time meeting ends, you can also submit a request to rerun the task to generate the key points. You must use the same TaskId.
Why are the key point extraction results not generated or empty after I call the feature?
The request parameters may be disabled or configured incorrectly. Compare your settings with the documentation and ensure they are configured correctly.
The language model you selected may not support key point extraction. For example, this feature is not currently supported for Japanese.
The audio or video file may not contain enough information. For example, if the ASR result contains too little text, the model cannot generate key points. This can happen because of excessive background noise or poor audio quality. To resolve this, test with an audio or video file that contains more useful information.
Why do the generated key points not include highlights or action items?
The key point extraction feature is designed to produce several types of results, but it may not generate all types for every task. For example, if the transcribed text contains too little information, the output might only include keywords, or no results at all. Similarly, even with sufficient information, if no valid action items are detected, the action item results will not be generated.