PPT extraction and summary

Feature description

This feature extracts PPT images from video files and summarizes the corresponding spoken content.

This feature is supported only for videos where the PPT is the main focus, such as in screen sharing scenarios or when a speaker is visible next to the slides. It does not support scenarios where a person walks or speaks in front of the PPT. You can test the performance on the Tingwu website. Click here to test

Request parameters

Example settings

{
    "Input":{
        ...
    },
    "Parameters":{
        ...
        "PptExtractionEnabled":true,
        ...
    }
}

Code examples

#!/usr/bin/env python
#coding=utf-8

import os
import json
import datetime
from aliyunsdkcore.client import AcsClient
from aliyunsdkcore.request import CommonRequest
from aliyunsdkcore.auth.credentials import AccessKeyCredential

def create_common_request(domain, version, protocolType, method, uri):
    request = CommonRequest()
    request.set_accept_format('json')
    request.set_domain(domain)
    request.set_version(version)
    request.set_protocol_type(protocolType)
    request.set_method(method)
    request.set_uri_pattern(uri)
    request.add_header('Content-Type', 'application/json')
    return request

def init_parameters():
    root = dict()
    root['AppKey'] = 'Enter the AppKey that you created in the Tingwu console'

    # Basic request parameters
    input = dict()
    input['SourceLanguage'] = 'cn'
    input['TaskKey'] = 'task' + datetime.datetime.now().strftime('%Y%m%d%H%M%S')
    input['FileUrl'] = 'Enter the URL of the audio file to test'
    root['Input'] = input

    # AI-related parameters. Set as needed.
    parameters = dict()
    # PPT extraction and summary
    parameters['PptExtractionEnabled'] = True
    root['Parameters'] = parameters
    return root

body = init_parameters()
print(body)

# TODO: Set your AccessKeyId and AccessKeySecret as environment variables.
credentials = AccessKeyCredential(os.environ['ALIBABA_CLOUD_ACCESS_KEY_ID'], os.environ['ALIBABA_CLOUD_ACCESS_KEY_SECRET'])
client = AcsClient(region_id='cn-beijing', credential=credentials)

request = create_common_request('tingwu.cn-beijing.aliyuncs.com', '2023-09-30', 'https', 'PUT', '/openapi/tingwu/v2/tasks')
request.add_query_param('type', 'offline')

request.set_content(json.dumps(body).encode('utf-8'))
response = client.do_action_with_exception(request)
print("response: \n" + json.dumps(json.loads(response), indent=4, ensure_ascii=False))

package com.alibaba.tingwu.client.demo.aitest;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.CommonRequest;
import com.aliyuncs.CommonResponse;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.MethodType;
import com.aliyuncs.http.ProtocolType;
import com.aliyuncs.profile.DefaultProfile;
import org.junit.Test;

/**
 * @author tingwu2023
 */
public class PptExtractionTest {

    @Test
    public void testPpt() throws ClientException {
        CommonRequest request = createCommonRequest("tingwu.cn-beijing.aliyuncs.com", "2023-09-30", ProtocolType.HTTPS, MethodType.PUT, "/openapi/tingwu/v2/tasks");
        request.putQueryParameter("type", "offline");

        JSONObject root = new JSONObject();
        root.put("AppKey", "Enter the AppKey that you created in the Tingwu console");

        JSONObject input = new JSONObject();
        input.fluentPut("FileUrl", "Enter the URL of the audio file to test")
                .fluentPut("SourceLanguage", "cn")
                .fluentPut("TaskKey", "task" + System.currentTimeMillis());
        root.put("Input", input);

        JSONObject parameters = new JSONObject();
        parameters.put("PptExtractionEnabled", true);
        root.put("Parameters", parameters);
        System.out.println(root.toJSONString());
        request.setHttpContent(root.toJSONString().getBytes(), "utf-8", FormatType.JSON);

        // TODO: Set your AccessKeyId and AccessKeySecret as environment variables.
        DefaultProfile profile = DefaultProfile.getProfile("cn-beijing", System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"), System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"));
        IAcsClient client = new DefaultAcsClient(profile);
        CommonResponse response = client.getCommonResponse(request);
        System.out.println(response.getData());
    }
    public static CommonRequest createCommonRequest(String domain, String version, ProtocolType protocolType, MethodType method, String uri) {
        // Create an API request and set its parameters.
        CommonRequest request = new CommonRequest();
        request.setSysDomain(domain);
        request.setSysVersion(version);
        request.setSysProtocol(protocolType);
        request.setSysMethod(method);
        request.setSysUriPattern(uri);
        request.setHttpContentType(FormatType.JSON);
        return request;
    }
}

Example output

{
    "Code":"0",
    "Data":{
        "TaskId":"d9cea51c1fezg647af24e815********",
        "TaskStatus":"COMPLETED",
        "Result":{
            "PptExtraction":"http://speech-swap.oss-cn-zhangjiakou.aliyuncs.com/tingwu_data/output/1738248129743478/d9cea51c1fef4647af24e8154d6ae0c0/d9cea51c1fef4647af24e8154d6ae0c0_PptExtraction_20231030213209.json?Expires=1698759180&OSSAccessKeyId=LTAI****************&Signature=4XY2*******Yra72pue4g1vetQ%3D",
            "Transcription":"http://speech-swap.oss-cn-zhangjiakou.aliyuncs.com/tingwu_data/output/1738248129743478/d9cea51c1fef4647af24e8154d6ae0c0/d9cea51c1fef4647af24e8154d6ae0c0_Transcription_20231030213158.json?Expires=1698759180&OSSAccessKeyId=LTAI****************&Signature=hV4C*******1aMauJnlg8djxZCQ%3D"
        }
    },
    "Message":"success",
    "RequestId":"3c347cdc-7e54-456b-9db3-c23********"
}

The PptExtraction field provides the HTTP URL for downloading the extraction and summary results.

Protocol parsing

The URL for the extraction and summary results points to a JSON message. The following example shows the structure of the message.

{
    "TaskId":"d9cea51c1fef464zaf21e815*******",
    "PptExtraction":{
        "AvailableForSummary":true,
        "PdfPath":"https://speech-swap.oss-cn-zhangjiakou.aliyuncs.com/tingwu/output/1738248129743478/61556f87f76a4ee285cb87629f8b0394/ppt_20231031110531/61556f87f76a4ee285cb87629f8b0394.pdf?Expires=1698731046&OSSAccessKeyId=yourAccessKeyID&Signature=URg%2BgSN******t2%2FUfiniGdYRGE%3D",
        "KeyFrameList":[
            {
                "Id":1,
                "Start":1160,
                "End":12320,
                "FileUrl":"https://speech-swap.oss-cn-zhangjiakou.aliyuncs.com/tingwu/output/1738248129743478/61556f87f76a4ee285cb87629f8b0394/ppt_20231031110531/61556f87f76a4ee285cb87629f8b0394_001.png?Expires=1698730995&OSSAccessKeyId=yourAccessKeyID&Signature=aO******TxzY%2FQFo9glABLwmnnY%3D",
                "Summary": "This section introduces the work and job requirements at Alibaba DAMO Academy, focusing on text-to-speech and speech-to-text Alibaba Cloud services. It also mentions that DAMO Academy aims to provide a single-interface service sold on the cloud. The features of the Tingwu product are also described, including multimodal capabilities such as voiceprint recognition, agenda segmentation, and keyword extraction. The product will be released at the end of the month for user access."
            },
            {
                "Id":2,
                "Start":14880,
                "End":50389,
                "FileUrl":"https://speech-swap.oss-cn-zhangjiakou.aliyuncs.com/tingwu/output/1738248129743478/61556f87f76a4ee285cb87629f8b0394/ppt_20231031110531/61556f87f76a4ee285cb87629f8b0394_002.png?Expires=1698731023&OSSAccessKeyId=yourAccessKeyID&Signature=2N*******vM8dYfJYNaXG59HA%3D",
                "Summary": "He is responsible for the production of AI capabilities in the NLP field and their integration with business needs. He introduced three company projects. He also mentioned some difficulties and solutions encountered during review and the company's future attempts at multimodal projects."
            }
        ]
    }
}

The following describes the fields.

FAQ

When can I use the PPT extraction and summary feature?

• You can set the parameters when you create an offline transcription task for an audio or video file.

• After an offline transcription task is complete, you can also rerun the task with the same TaskId to generate the results.

• This feature is designed to extract PPT slides and summaries from videos, which limits its use cases. It cannot be used for real-time meetings that use audio streams without video, or for offline transcription if the source file is not a video.

Why are the PPT extraction results empty after I call the API?

• The request parameters may be incorrect. Check your settings by referring to this document.

• The feature cannot extract PPT slides from real-time meetings or non-video files. This results in empty extraction and summary results.

• Even if you use a video file, no results are generated if the service does not detect any PPT slides in the video.

Why did I also receive speech transcription results?

• PPT extraction does not depend on Automatic Speech Recognition (ASR) results. However, the summary generation process requires the transcribed text from ASR.

Why do my results sometimes contain extracted images but no summaries?

• This feature works in two steps. First, it extracts PPT slides from the video. Second, it uses the timestamps and the transcribed text to generate a summary for each slide. If the transcribed text for a slide is too short, the service cannot generate a meaningful summary. In this case, the summary field for that slide is empty.