Getting started

Updated at:

The Intelligent Document Processing Software Development Kit (IDP-SDK) is a Python SDK for the Alibaba Cloud Document Mind service. It converts documents into a structured object format. The SDK helps you quickly clean data and integrate with frameworks such as LangChain and LlamaIndex.

Introduction to the IDP-SDK

What is the IDP-SDK?

The Intelligent Document Processing Software Development Kit (IDP-SDK) is a Python SDK for the Alibaba Cloud Document Mind service. It converts documents into a structured object format. The SDK helps you quickly clean data and integrate with frameworks such as LangChain and LlamaIndex.

Features

  • Provides objects to deserialize doc-json results from the structured output of Document Mind and includes helper functions.

Scenarios

Document Mind parsing calls

Version number

Description

1.0.0

Provides a unified Python package for calling the Document Mind parsing, electronic document parsing, and document parsing (Large Language Model (LLM)) APIs.

Environment selection

  • To configure environment variables for cloud-based invocation, see the Service Endpoints document:

    • ALIBABA_CLOUD_ACCESS_KEY_ID=YOUR_ALIBABA_CLOUD_ACCESS_KEY_ID

    • ALIBABA_CLOUD_ACCESS_KEY_SECRET=YOUR_ALIBABA_CLOUD_ACCESS_KEY_SECRET

Integration method

  • Python 3.9 or later

pip install https://docmind-api-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/sdk/doc_json_sdk-1.0.2-py3-none-any.whl

Usage instructions

The main workflow for using the SDK is to load a file and process it with various handler methods. This process creates a document object. The object contains information about the layout, text/Markdown content, images, tables, text styles, and doctree hierarchy. You can then use the render method to output the parsed result as a Markdown file.

image

Calling in a public cloud environment

Calls in a public cloud environment.

Configure the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables.

from doc_json_sdk.loader.document_model_loader import DocumentModelLoader
from doc_json_sdk.handler.document_handler import DocumentExtractHandler, DocumentDigitalExtractHandler
def test_document_hander():
    file_path = "" # your local file
    file_url = None
    # DocumentExtractHandler: Document Mind parsing
    # DocumentDigitalExtractHandler: Electronic document parsing
    loader = DocumentModelLoader(handler=DocumentExtractHandler())
    document = loader.load(file_path=file_path, file_url=file_url) 
    
    

Calling with formula enhancement in a public cloud environment

from doc_json_sdk.loader.document_model_loader import DocumentModelLoader
from doc_json_sdk.handler.document_handler import DocumentExtractHandler
def test_render_formula_markdown():
    file_path = "" # your local file
    file_url = None
    handler = DocumentExtractHandler()
    loader = DocumentModelLoader(handler=handler)
    document = loader.load(file_path=file_path,file_url=file_url,
                           formula_enhancement=True,
                           reveal_markdown=True,
                           save_json_path="docmind_result.json")

Loading from a doc-json string

from doc_json_sdk.loader.document_model_loader import DocumentModelLoader
def test_local_json_document():
    file_path = "" # your local file
    loader = DocumentModelLoader()
    document = loader.load(doc_json_fp=open(file_path, "r"))

Processing a file into a Markdown string

from doc_json_sdk.loader.document_model_loader import DocumentModelLoader
from doc_json_sdk.handler.document_handler import DocumentExtractHandler,DocumentDigitalExtractHandler
from doc_json_sdk.render.document_model_render import DocumentModelRender
def test_render_markdown():
    file_path = "" # your local file
    file_url = None
    loader = DocumentModelLoader(handler=DocumentExtractHandler())
    document = loader.load(file_path=file_path, file_url=file_url, reveal_markdown=True)
    render = DocumentModelRender(document_model=document)
    with open("docmind_result.md", "w") as f:
        f.write(render.render_reveal_markdown())

Filtering based on layout objects

from doc_json_sdk.model.enums.layout_type_enum import LayoutTypeEnum

for layout in document:
    type_enum = layout.get_layout_type_enum()
    if (type_enum == LayoutTypeEnum.Elements.FOOTER or
            type_enum == LayoutTypeEnum.Elements.HEADER or
            type_enum == LayoutTypeEnum.Elements.NOTE):
        #  header and footer notes
        pass
    elif type_enum == LayoutTypeEnum.Elements.IMAGE:
        # image with head_line or split_line
        if layout.type.find("_line")!=-1:
            continue
    elif type_enum == LayoutTypeEnum.Elements.TABLE:
        #table
        pass
    else:
        # paragraph or note(table or figure)
        pass