This topic describes the AI summarization features of the Large Language Model (LLM) and how to implement them.
The summarization feature is based on the Qwen LLM. It extracts key information from conversations across multiple dimensions.
The feature currently supports full summaries, speaker summaries, Q&A reviews, and mind maps.
Full summary: Extracts a summary that is faithful to the original text. A full summary presents the most important information in 200 to 300 words. This helps you quickly understand the main content and purpose of the recording.
Speaker summary: Meetings often involve discussions among multiple people. Tingwu can distinguish between different speakers and summarize each person's views. The speaker summary feature organizes and presents the key points made by each speaker.
Q&A review: Extracts and refines questions and answers from multi-person interactions into concise Q&A pairs. This helps you quickly identify the key questions and answers from a meeting.
Mind map: Summarizes the audio or video content and generates the data structure required to render a mind map. You must pass the result to a frontend framework to render the mind map image. The resulting tree structure has a maximum depth of four levels.
You can retrieve all summary results at once, or retrieve one or more types as needed.
Request parameters
Parameter Name | Type | Description |
SummarizationEnabled | boolean | The default value is false. |
Summarization | list[] | The default value is empty. Add the summary types you want to process as needed. The following types are supported:
|
For descriptions of other request parameters, see the document for your scenario:
Example settings
{
"Input":{
...
},
"Parameters":{
"SummarizationEnabled":true,
"Summarization":{
"Types":[
"Paragraph",
"Conversational",
"QuestionsAnswering",
"MindMap"
]
}
}
}Code examples
#!/usr/bin/env python
#coding=utf-8
import os
import json
import datetime
from aliyunsdkcore.client import AcsClient
from aliyunsdkcore.request import CommonRequest
from aliyunsdkcore.auth.credentials import AccessKeyCredential
def create_common_request(domain, version, protocolType, method, uri):
request = CommonRequest()
request.set_accept_format('json')
request.set_domain(domain)
request.set_version(version)
request.set_protocol_type(protocolType)
request.set_method(method)
request.set_uri_pattern(uri)
request.add_header('Content-Type', 'application/json')
return request
def init_parameters():
root = dict()
root['AppKey'] = 'Enter the AppKey that you created in the Tingwu console'
# Basic request parameters
input = dict()
input['SourceLanguage'] = 'cn'
input['TaskKey'] = 'task' + datetime.datetime.now().strftime('%Y%m%d%H%M%S')
input['FileUrl'] = 'Enter the URL of the audio file to test'
root['Input'] = input
# AI-related parameters. Set them as needed.
parameters = dict()
# Summarization control, including full summary, speaker summary, and Q&A review
parameters['SummarizationEnabled'] = True
summarization = dict()
summarization['Types'] = ['Paragraph', 'Conversational', 'QuestionsAnswering', 'MindMap']
parameters['Summarization'] = summarization
root['Parameters'] = parameters
return root
body = init_parameters()
print(body)
# TODO: Set your AccessKeyId and AccessKeySecret as environment variables
credentials = AccessKeyCredential(os.environ['ALIBABA_CLOUD_ACCESS_KEY_ID'], os.environ['ALIBABA_CLOUD_ACCESS_KEY_SECRET'])
client = AcsClient(region_id='cn-beijing', credential=credentials)
request = create_common_request('tingwu.cn-beijing.aliyuncs.com', '2023-09-30', 'https', 'PUT', '/openapi/tingwu/v2/tasks')
request.add_query_param('type', 'offline')
request.set_content(json.dumps(body).encode('utf-8'))
response = client.do_action_with_exception(request)
print("response: \n" + json.dumps(json.loads(response), indent=4, ensure_ascii=False))package com.alibaba.tingwu.client.demo.aitest;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.CommonRequest;
import com.aliyuncs.CommonResponse;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.MethodType;
import com.aliyuncs.http.ProtocolType;
import com.aliyuncs.profile.DefaultProfile;
import org.junit.Test;
/**
* @author tingwu2023
*/
public class SummarizationTest {
@Test
public void testSummarization() throws ClientException {
CommonRequest request = createCommonRequest("tingwu.cn-beijing.aliyuncs.com", "2023-09-30", ProtocolType.HTTPS, MethodType.PUT, "/openapi/tingwu/v2/tasks");
request.putQueryParameter("type", "offline");
JSONObject root = new JSONObject();
root.put("AppKey", "Enter the AppKey that you created in the Tingwu console");
JSONObject input = new JSONObject();
input.fluentPut("FileUrl", "Enter the URL of the audio file to test")
.fluentPut("SourceLanguage", "cn")
.fluentPut("TaskKey", "task" + System.currentTimeMillis());
root.put("Input", input);
JSONObject parameters = new JSONObject();
parameters.put("SummarizationEnabled", true);
JSONObject summarization = new JSONObject();
// Full summary, speaker summary, and Q&A summary (Q&A review)
JSONArray types = new JSONArray()
.fluentAdd("Paragraph")
.fluentAdd("Conversational")
.fluentAdd("QuestionsAnswering")
.fluentAdd("MindMap");
summarization.put("Types", types);
parameters.put("Summarization", summarization);
root.put("Parameters", parameters);
System.out.println(root.toJSONString());
request.setHttpContent(root.toJSONString().getBytes(), "utf-8", FormatType.JSON);
// TODO: Set your AccessKeyId and AccessKeySecret as environment variables
DefaultProfile profile = DefaultProfile.getProfile("cn-beijing", System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"), System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"));
IAcsClient client = new DefaultAcsClient(profile);
CommonResponse response = client.getCommonResponse(request);
System.out.println(response.getData());
}
public static CommonRequest createCommonRequest(String domain, String version, ProtocolType protocolType, MethodType method, String uri) {
// Create an API request and set its parameters
CommonRequest request = new CommonRequest();
request.setSysDomain(domain);
request.setSysVersion(version);
request.setSysProtocol(protocolType);
request.setSysMethod(method);
request.setSysUriPattern(uri);
request.setHttpContentType(FormatType.JSON);
return request;
}
}Example output
{
"Code":"0",
"Data":{
"TaskId":"5a7343ad75e64z3da121ce65********",
"TaskStatus":"COMPLETED",
"Result":{
"AutoChapters":"http://speech-swap-hangzhou.oss-cn-hangzhou.aliyuncs.com/tingwu/output/1503864348104017/5a7343ad75e6493da121ce653f9389eb/5a7343ad75e6493da121ce653f9389eb_AutoChapters_20231029224651.json?Expires=1698677343&OSSAccessKeyId=LTAI****************&Signature=********TYeZfkvr3mgEK%2Bseo1U%3D",
"Transcription":"http://speech-swap-hangzhou.oss-cn-hangzhou.aliyuncs.com/tingwu/output/1503864348104017/5a7343ad75e6493da121ce653f9389eb/5a7343ad75e6493da121ce653f9389eb_Transcription_20231029224625.json?Expires=1698677343&OSSAccessKeyId=LTAI****************&Signature=Kliw%2Fp2xCKHpKR06********Y8%3D",
"Summarization":"http://speech-swap-hangzhou.oss-cn-hangzhou.aliyuncs.com/tingwu/output/1503864348104017/5a7343ad75e6493da121ce653f9389eb/5a7343ad75e6493da121ce653f9389eb_Summarization_20231029224656.json?Expires=1698677343&OSSAccessKeyId=LTAI****************&Signature=rY5v4CbGjKnh0Nu*********Xhs%3D"
}
},
"Message":"success",
"RequestId":"4EEBD53F-BCC9-1A0E-B17E-C459********"
}The value of the Summarization field is the HTTP URL to download the summary result.
Protocol parsing
The summary result URL provides a JSON message. The following example shows the message format.
{
"TaskId": "5a7343ad75e6493da121ce65*********",
"Summarization": {
"ParagraphSummary": "This text introduces the work and job requirements at Alibaba DAMO Academy. It mentions that DAMO Academy primarily focuses on text-to-speech and speech-to-text cloud services. It also answers questions about essay grading standards for different grades and introduces the multimodal project, Tingwu.",
"ConversationalSummary": [
{
"SpeakerId": "1",
"SpeakerName": "Speaker 1",
"Summary": "Introduced the work and job requirements at Alibaba DAMO Academy, focusing on speech-to-text and text-to-speech cloud services. He mentioned that DAMO Academy aims to provide an API-based service sold on the cloud. He also described the features of the Tingwu product, including multi-modal capabilities such as voiceprint recognition, agenda segmentation, and keyword extraction. The product will be released at the end of the month for user access."
},
{
"SpeakerId": "2",
"SpeakerName": "Speaker 2",
"Summary": "He is responsible for the production of NLP-related AI capabilities and their integration with business needs. He introduced three of the company's projects. He also mentioned some difficulties and solutions encountered in grading, and the company's future exploration of multimodal projects."
}
],
"QuestionsAnsweringSummary": [
{
"Question": "What kind of department is DAMO Academy?",
"SentenceIdsOfQuestion": [
207,
208,
209,
210
],
"Answer": "DAMO Academy is a department within Alibaba Group that primarily handles cloud services such as speech-to-text, text-to-speech, text translation, and image recognition.",
"SentenceIdsOfAnswer": [
207,
208,
209,
210
]
}
],
"MindMapSummary": [
{
"Title": "Minutes of the On-site Visit to Alibaba DAMO Academy for Voice Technology and Smart Devices",
"Topic": [
{
"Title": "1. Introduction to DAMO Academy",
"Topic": [
{
"Title": "Headquartered in Hangzhou, with branches in multiple global locations (USA, Israel, Singapore, etc.)",
"Topic": []
},
{
"Title": "Main Research Areas and Achievements",
"Topic": [
{
"Title": "Prediction of the Top Ten Future Technology Trends",
"Topic": []
},
{
"Title": "Ongoing High-Risk Scientific Research Projects",
"Topic": []
}
]
}
]
},
{
"Title": "2. Discussion on Voice Technology",
"Topic": [
{
"Title": "Laboratory Environment Challenges",
"Topic": [
{
"Title": "High noise, multiple speakers, and interference from various electrical appliances",
"Topic": []
}
]
},
{
"Title": "Technical Challenges",
"Topic": [
{
"Title": "Target speaker identification",
"Topic": []
},
{
"Title": "Semantic understanding: Differentiating emotions for homophones",
"Topic": []
}
]
},
{
"Title": "Current Applications",
"Topic": [
{
"Title": "Automated responses in call centers",
"Topic": []
},
{
"Title": "Optimization of speech recognition in meeting scenarios",
"Topic": []
}
]
},
{
"Title": "Future Outlook",
"Topic": [
{
"Title": "AI participation in meetings to improve decision quality",
"Topic": []
},
{
"Title": "Widespread application of voice technology in various fields",
"Topic": []
}
]
}
]
}
]
}
]
}
}The fields are defined as follows.
Parameter Name | Type | Description |
TaskId | string | The ID of the task, generated when the task is created. |
Summarization | object | The summary result object. It may contain results for zero or more summary types. |
Summarization.ParagraphSummary | string | The full summary result. |
Summarization.ConversationalSummary | list[] | A list of speaker summary results. |
Summarization.ConversationalSummary[i].SpeakerId | string | The speaker ID. |
Summarization.ConversationalSummary[i].SpeakerName | string | The speaker name. |
Summarization.ConversationalSummary[i].Summary | string | The summary for the speaker. |
Summarization.QuestionsAnsweringSummary | list[] | A list of Q&A review summary results. |
Summarization.QuestionsAnsweringSummary[i].Question | string | The question. |
Summarization.QuestionsAnsweringSummary[i].SentenceIdsOfQuestion | list[] | A list of sentence IDs from the original speech transcription that correspond to the question. |
Summarization.QuestionsAnsweringSummary[i].Answer | string | The answer to the question. |
Summarization.QuestionsAnsweringSummary[i].SentenceIdsOfAnswer | list[] | A list of sentence IDs from the original speech transcription that correspond to the answer. |
Summarization.MindMapSummary | list[] | A list of mind map results. |
Summarization.MindMapSummary[i].Title | string | The text content of a single node in the mind map. |
Summarization.MindMapSummary[i].Topic | list[] | A list of child nodes for a single node in the mind map. |
FAQ
When can I call the summarization feature?
You can set the parameters directly when you create an offline transcription task for an audio or video file, or when you create a real-time meeting.
You can also rerun the task after an offline transcription or real-time meeting ends. You must use the same TaskId for the rerun request.
Why are no summary results generated, or why are the results empty after I make a call?
The call parameters may not be enabled or may be set incorrectly. You can check the developer documentation to ensure the parameters are set correctly.
The specified language model may not support summarization. For example, summarization is not currently supported for Japanese audio or video files.
The audio or video file may not contain enough information. For example, if the file has too little Automatic Speech Recognition (ASR) information after transcription (possibly due to excessive background noise or poor quality), the model cannot generate chapter information. In this case, you can try testing with a different audio or video file that contains more useful information.
You may not have set a summary type, or you may have set an unsupported type. You can refer to the types described in this topic and set them correctly.
Why does the summarization process produce no results?
Although the feature can produce multiple types of summaries, a result is not guaranteed. For example, a summary may not be generated if the transcription contains too little useful information.
If a Q&A summary is not generated, it may be because the original audio or video does not contain any question-and-answer content.
I did not set parameters for a chapter overview, so why was a chapter overview result returned in the summary?
A full summary is calculated based on the original audio or video information, the ASR transcription result, and the chapter title information. Therefore, when you enable the full summary feature, Tingwu automatically calls the chapter overview feature and includes its result in the output.