how to batch export documents using scroll-OpenSearch(Open Search)-阿里云帮助中心

Use the scroll method to export documents in batches beyond the 5,000-document limit of the standard search method.

Scenarios

Traditional search returns the most relevant results quickly but limits the number of retrievable documents. For example, the `search` method can retrieve a maximum of 5,000 documents. To retrieve a larger number of documents for analysis, use the scroll API.

Parameters

Search parameters:

Parameter	Type	Required	Valid values	Default value	Description
scroll	STRING	Yes	Week, day, hour, minute, second		The validity period for the next scroll request. Must be set for each request. For example, `1m` indicates 1 minute. Supported time units are: w for Week, d for Day, h for Hour, m for minute, and s for second.
search_type	STRING	Yes	scan		Required for the first query. For subsequent queries, specify the `scroll_id` to perform the next query.
scroll_id	string	Yes			The first scroll call returns a `scroll_id` but no data. Each subsequent search must include the `scroll_id` from the previous response. Subsequent results return a new `scroll_id` along with the matching data.
fetch_fields	string				Specifies the application fields to retrieve.

Response parameters:

Parameter	Type	Description
status	string	The execution result. `OK` indicates success. `FAIL` indicates failure. Troubleshoot based on the returned error code.
request_id	string	The query record ID, used primarily for troubleshooting.
result	string	The query result, including the query time (`searchtime`), total number of results (`total`), number of results in the current response (`num`), maximum number of results for the query (`viewtotal`), query result items (`items`), facet statistics (`facet`), and the `scroll_id`.
errors	string	Error details. `error_message` contains the error information. For the meaning of an `error_code`, see the Error codes.

Note

The scroll response currently supports only the `fulljson` and `json` formats.

Results

Response to the first request:

{
    "status": "OK",
    "request_id": "150150574119953661605242",
    "result": {
        "searchtime": 0.005029,
        "total": 1,
        "num": 0,
        "viewtotal": 1,
        "scroll_id": "eJxtUMtuhDAM/BrvOYQC5cABdulvRFFIirsm2TpBavv1Ndut1EMlS36NZ0Y2ZHMxbueceAjIuWCMnrPjRITLyfzZm83y9V QVGT8x80U3PxQNUqieVZV1/an4ItbTUBPSx5wgXqKdvOSbmuKR8ZYjGWWirB4tvToAiX7u3G2eCNK77vnz8GlGPAV6suKBeqxAn0OiTd7NGEnesspyoyFLF6hecn4JUKjVgp0K3FnkfMfIyPoDuYWegX9GeYOpicY9TG8gwOSuBL04X1 MMg3ROwCesLlG6X7a2o=",
        "items": [],
        "facet": []
    },
    "errors": [],
    "tracer": ""
}

Response to a subsequent request:

{
    "status": "OK",
    "request_id": "150150574119952551519970",
    "result": {
        "searchtime": 0.006293,
        "total": 1,
        "num": 1,
        "viewtotal": 1,
        "scroll_id": "eJxNT9tugzAM/RrznIRC4YEHaNlvRFFIhteQtE6Qtn39TNdJk2z5dnx8rIPJRdudcqKhl60Uir2Vp06ISv8b6s3QbZCVzpaCdp93XXBzg2wEW9MJ2dWq8q7YVXt0YckDLlBP0WyOw31N8YgYizZEnAUsjkx4VT4k8zexpjiNS/XYHX0NNkWP71BfVyxQjxLUxSfazFH4PYSPnCL3iMniDZq3jN98aFRCgGrZniy8/itkBHWGuYVeQH+B+QzTCUZ1NJ9gj4FVMfrQPr8Y+Hk+dgU14fIDVhtfTw==",
        "items": [
            {
                "fields": {
                    "cate_id": "0",
                    "float_arr": "0",
                    "id": "1",
                    "int_arr": "0",
                    "literal_arr": "search",
                    "name": "search",
                    "phone": "1381111****",
                    "index_name": "app_schema_demo"
                },
                "property": {},
                "attribute": {},
                "variableValue": {},
                "sortExprValues": [
                    "1"
                ]
            }
        ],
        "facet": []
    },
    "errors": [],
    "tracer": ""
}

Notes

The `sort` clause only supports single fields of the INT type. This applies only to the v3 API and software development kit (SDK).
The scroll feature only supports exporting all data. It does not support Aggregate or Distinct clauses, coarse or fine sort expressions, or query analysis.
The `start` parameter in the `config` clause has no effect in a scroll query. The default value is 0, which means pagination is not supported. The value of `hits` must be in the range of [0, 500].
A scroll query uses the `hits` value from the first query, which is the query that returns the `scroll_id`. Changes to the `hits` value in subsequent queries do not take effect.
The first query returns only a `scroll_id` and no document data. The second query, which includes the `scroll_id`, returns the data.
To identify search errors, check the `code` and `message` fields for exceptions. Do not rely on the `status` field. For more information, see Error codes.
If the `Scroll_id is expired` error is returned, the scroll request has expired. Adjust the `scroll` parameter to extend the validity period.

SDK sample demos

Note

1. The `start` parameter in the `config` clause has no effect. Instead, use the `hits` value to set the number of documents to retrieve for each request.

2. Features such as `aggregate`, `distinct`, and coarse and fine sort expressions are not supported. The `sort` clause only supports sorting by a single field of the INT type.

3. Scroll queries across multiple applications are not supported.

4. If you pass an invalid `scroll_id`, the query returns an error.

5. The supported formats for retrieval results are fulljson and json.

6. The first query returns only a `scroll_id` and no document data. To retrieve data, you must perform a second query and include the `scroll_id` from the first response.