DashScopeParse is a tool for efficiently parsing and representing files-Alibaba Cloud Model Studio(Model Studio)-阿里云帮助中心

DashScopeParse is a parser designed to efficiently parse and represent files. This enables effective retrieval and context enhancement through the LlamaIndex framework. It uses the Alibaba Cloud Document Mind service for parsing.

Get started

Prerequisites

Log on to https://bailian.console.aliyun.com/ to obtain your API key. If you want to specify a workspace, you must also obtain the workspace ID.

For more information, see Obtain and configure an API key.
A workspace has been created. For more information, see Workspace Management.

Install the DashScopeParse package (Python >=3.8 and <=3.12).

pip install llama-index-core
pip install llama-index-readers-dashscope

Sample code

Use the following code to retrieve the parsing results for a PDF file:

import os

os.environ['DASHSCOPE_API_KEY'] = "<Your API Key>"
os.environ['DASHSCOPE_WORKSPACE_ID'] = "<Your workspace ID. The default workspace is empty.>"

from llama_index.readers.dashscope.base import DashScopeParse
from llama_index.readers.dashscope.utils import ResultType

file = ['aiayn.pdf', 'not_exist.pdf']
parse = DashScopeParse(result_type=ResultType.DASHSCOPE_DOCMIND, category_id="<category id>")
documents = parse.load_data(file_path=file)

Note:

Currently, only .doc, .docx, and .pdf files can be uploaded and parsed. Files must be smaller than 100 MB and have fewer than 1000 pages.

Note:

DASHSCOPE_API_KEY: Your API key. This parameter is required.
DASHSCOPE_WORKSPACE_ID: Optional. The ID of the workspace that you want to use. If you do not specify this parameter, the default workspace is used.

Output

The result is a list of Document objects. The following JSON example shows only the key fields. The text field is truncated because it is too long.

[
    {
        "id_": "doc_ab3d344971084f288248d69ba9bada1310022601",
        "metadata": {
            "parse_fmt_type": "DASHSCOPE_DOCMIND"
        },
        "text": "\"{\\\"logics\\\":{\\\"docTree\\\":[{\\\"backlink\\\":{\\\"parent\\\":[\\\"ROOT\\\"]},\\\"level\\\":0,\\\"link\\\":{\\\"child\\\":[],\\\"contains\\\":[]},\\\"uniqueId\\\":\\\"171c391e5e7c68a253eb0adf59fa476d\\\"},{\\\"backlink\\\":{\\\"parent\\\":[\\\"171c391e5e7c68a253eb0adf59fa476d\\\"]},\\\"level\\\":1,\\\"link\\\":{\\\"child\\\":[],\\\"contains\\\":[]},\\\"uniqueId\\\":\\\"397257823ee2878fc4b4a5759fe9afe9\\\"},{\\\"backlink\\\":{\\\"parent\\\":[\\\"ROOT\\\"]},\\\"level\\\":0,\\\"link\\\":{\\\"child\\\":[],\\\"contains\\\":[]},\\\"uniqueId\\\":\\\"08ab57d41ea73966c49191292e1abb5b\\\"},{\\\"backlink\\\":{\\\"parent\\\":[\\\"08ab57d41ea73966c49191292e1abb5b\\\"]},\\\"level\\\":1,\\\"link\\\":{\\\"child\\\":[],\\\"contains\\\":[]},\\\"uniqueId\\\":\\\"2f5ac27b37e3cb65f14799d4811bb633\\\"},......",
    }
]

Parameters

Input parameters

Parameter	Type	Default value	Description
result_type	Enum	ResultType.DASHSCOPE_DOCMIND	The type of the parser result.
num_workers	int	4	The number of workers to use for API requests when processing multiple files.
check_interval	int	5	The interval for querying the parsing status, in seconds.
max_timeout	int	3600	The parsing timeout period, in seconds.
workspace	str	None	The DashScope workspace ID.
api_key	str	None	The DashScope API key. You can set this using an environment variable or other methods.
category_id	str	default	The ID of the category for the upload.

Output Document object fields

Field name	Field type	Field description
id_	str	The system-generated document ID.
metadata.parse_fmt_type	str	The format type of the parsing result.
text	str	The parsing result string. This is a JSON-escaped string. In Python, use `json.loads(text)` to parse it.