DashScopeJsonNodeParser

DashScopeJsonNodeParser is a text chunking model from Alibaba's Tongyi Lab that allows developers to use the LlamaIndex framework to chunk documents parsed by DashScopeParse.

Get started

Prerequisites

First, log on to https://bailian.console.aliyun.com/cn-beijing/ to obtain your API key. To specify a workspace, you must also obtain the workspace ID.

• An API key has been created. For more information, see Obtain and configure an API key.
• A workspace has been created. For more information, see Workspace management.

Then, install the DashScopeJsonNodeParser package (Python >=3.9 and <=3.12).

pip install llama-index-core
pip install llama-index-node-parser-dashscope

Sample code

import json
import os
from llama_index.node_parser.dashscope import (
    DashScopeJsonNodeParser,
)
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.schema import Document

# Set your Dashscope API key in the environment
os.environ["DASHSCOPE_API_KEY"] = "your_api_key_here"

documents = [
    # Prepare your documents obtained from the Dashscope reader
]

# Initialize the DashScope JsonNodeParser
node_parser = DashScopeJsonNodeParser(
    chunk_size=100, overlap_size=0, separator=" |,|,.|?|!|\n|?|!"
)

# Set up the ingestion pipeline with the node parser
pipeline = IngestionPipeline(transformations=[node_parser])

# Process the documents and print the resulting nodes
nodes = pipeline.run(documents=documents, show_progress=True)
for node in nodes:
    print(node)

• DashScopeJsonNodeParser is primarily used with DashScopeParse.

Parameters