ReadPageScrape-Enhanced-Information Query Service(IQS)-阿里云帮助中心

This topic describes the ReadPageScrape API, which uses a headless browser to dynamically render and access target pages. This topic also covers the features, parameters, return values, and calling methods of the API.

1. API description

The features of the API are as follows:

Reads HTML and parses webpage content in a browser sandbox environment.
The API starts parsing after the resources on the target page are fully loaded. You can adjust the maximum wait time using the pageTimeout parameter. The overall API processing time is significantly affected by the resource loading status of the target site.
If the Content-Type in the response header of the target address is application/pdf, the system automatically triggers PDF parsing. The parsed content is extracted from the PDF file displayed in the current browser window.

2. API definition

2.1 Request parameters

Field

Parameter value

Description

url

<string>

required

The target address to parse. It must start with http:// or https://.

pageTimeout

<int>

[0, 100000]

default: 10000

The timeout period for waiting for the target site to fully load. The API will complete within pageTimeout + 6000 ms.

Most pages can be loaded within 5000 ms.
For pages with many resources, set this to 10000 ms.

formats

<array>

rawHtml

html

markdown

text

screenshot

default: [

"html", "markdown", "text"

]

The format of the parsing result.

rawHtml: The HTML of the target site.
html: The page content processed according to readabilityMode.
markdown: The markdown content converted from the HTML.
text: The text content from the HTML.
screenshot: A screenshot of the page. If enabled, the API will take an additional 2-5 s on top of pageTimeout.

maxAge

<int>

[0, ∞]

default: 1296000

The maximum cache time, in seconds.

If the cache time is less than maxAge, the cached content is returned.
If maxAge is 0, the cache is not used.

actions

<array>

actions.type: "wait"
Waits for a specified time or for an element to appear.

actions.duration<int>: The number of milliseconds to wait.
actions.selector<string>: A CSS selector. Waits for the element to appear.

[]

Executes specified operations before getting the main content.

A maximum of 20 steps.
Execution timeout is 60000 ms.

Example:

[
  {
    "type": "wait",
    "selector": "#list",
    "timeout": 10000
  },
  {
    "type": "wait",
    "duration": 800
  },
  {
    "type": "eval",
    "scripts": "document.querySelector('#cookie-accept')?.click()"
  }
]

actions.type: "eval"

actions.scripts<string>: The JavaScript code to execute. Supports return values.

readability

<map>

readabilityMode

<string>

none

normal

article

default: none

normal: Based on a proprietary algorithm, removes irrelevant information (such as headers, footers, and navigation) and returns the main content.

article: Based on a proprietary algorithm, gets the main content of the site. Suitable for blogs and news sites, but not for directory or navigation pages.

excludeAllImages

<bool>

default: false

Specifies whether to remove all images.

excludeAllLinks

<bool>

default: false

Specifies whether to remove all links.

excludedTags

<array>

[]

Specifies the tags to exclude, such as:

["form", "header", "footer", "nav"]

Return parameters

Field			Field description	Example
requestId <string>			The request ID. Provide this ID when troubleshooting issues.
errorCode <string \| null>			Error code.	Error codes
errorMessage <string \| null>			Error message.
data <map>	statusCode <int>		If the request to the target site is successful, returns the HTTP code of the target URL. If the request to the target site fails, returns a custom IQS error code. 4030: Security restrictions on the target site (such as robots.txt or security policies). 4080: Request timeout. 4290: The site's rate limiting policy was triggered. 5010: Unknown exception.
	rawHtml <string \| null>		The original HTML of the target site.
	html <string \| null>		The readable HTML of the target site.
	text <string \| null>		Content in text format.
	markdown <string \| null>		Content in markdown format.
	screenshot <string \| null>		A screenshot of the target site.
	actions <array \| null>		The execution results corresponding to the actions.	`{ "actions": [ {}, {}, { "ret": "" } ] }`
	links <map \| null>	internal <map\|null>	Link information from the target URL (internal links). href: The link address. text: The display text of the link. title: Additional tooltip text.	`{ "href": "https://www.alibabagroup.com/cn/global/home", "text": "Alibaba Group", "title": "" }`
	links <map \| null>	external <map\|null>	Link information from the target URL (external links).
	media <map \| null>	images <map\|null>	Image information from the target URL. type: image src: The image address. data: Embedded or additional data. alt: Alternative text. desc: Additional description. format: The image format.	`{ "type": "image", "src": "https://img.alicdn.com/tfs/TB1AOdINW6qK1RjSZFmXXX0PFXa-258-258.jpg", "data": "", "alt": "Alibaba Cloud WeChat", "desc": null, "format": "jpg" }`
		audios <array\|null>	Audio information from the target URL. type: audio src: The audio address. data: Embedded or additional data. alt: Alternative text. desc: Additional description. format: The format.
		videos <array\|null>	Video information from the target URL. type: video src: The video address. data: Embedded or additional data. alt: Alternative text. desc: Additional description. format: The format.	`[ { "type": "video", "src": "blob:https://xxxxx.com/xxx", "data": "", "alt": null, "desc": "", "format": null } ]`
	metadata <map>	url <string>	Target address.	{ "metadata": { "url": "https://example.com/article/123", "title": "Example Article Title", "hostLogo": "https://example.com/favicon.ico", "lastModified": "2026-06-20T10:00:00Z", "redirectedUrl": null, "contentType": "text/html; charset=utf-8", "pdfParse": false, "description": "A brief description of the article.", "author": "John Doe", "publishedDate": "2026-06-15T08:30:00Z", "siteName": "Example Site", "imageUrl": "https://example.com/images/og-image.jpg", "canonicalUrl": "https://example.com/article/123", "language": "en", "schemaType": "Article", "pageType": "article" } }
		redirectedUrl	The actual URL after redirection.
		title <string\|null>	Site title.
		hostLogo <string\|null>	Site favicon or logo URL.
		lastModified <string\|null>	Last modified time.
		publishedDate <string\|null>	Published date (`article:published_time / JSON-LD datePublished`)
		contentType <string\|null>	The `Content-Type` header of the HTTP response.
		description <string\|null>	Page description (`og:description` → `<meta name="description">`).
		author <string\|null>	Author information (`article:author` / JSON-LD author).
		siteName <string\|null>	Site name (`og:site_name`).
		imageUrl <string\|null>	Main image URL (`og:image`).
		canonicalUrl <string\|null>	Canonical URL (`<link rel="canonical">`).
		language <string\|null>	Page language (`<html lang="...">`).
		schemaType <string\|null>	JSON-LD `@type` (such as `Article`, `Product`).
		pageType <string\|null>	Page classification based on a proprietary algorithm. article: A single article, news story, blog post, or long-form content. forum: A forum, Q&A pair, or discussion thread. product: An E-commerce product page. listing: A search result, list, or category index page. collection: A home page, landing page, or navigation page. documentation: Technical documentation (such as API Reference, Tutorials, or FAQ). service: A marketing, pricing, contact, about, or privacy policy page. empty: A blank page (such as a 404 or a page with no content). login: A logon or authentication page. video: A page where video is the primary content. paid: A paywall. Other
		pdfParse <bool>	Indicates whether PDF parsing was triggered.

Error codes

HTTP Code	Error code	Error message	Solution
404	InvalidAccessKeyId.NotFound	Specified access key is not found.	Check and ensure that your AccessKey and Secret are correct.
403	Retrieval.NotActivate	Please activate AI search service	Place an order or contact your account manager to activate the service.
403	Retrieval.NotAuthorised	Please authorize the AliyunIQSFullAccess privilege to the sub-account.	The RAM user is not authorized. For more information, see Create a RAM user and grant permissions.
429	Retrieval.Throttling.User	Request was denied due to user flow control.	The rate limit was exceeded.
429	Retrieval.TestUserQueryPerDayExceeded	The query per day exceed the limit.	The test quota was exceeded (1000 times/30 days).
403	ReadPage.SecurityRestrict	Security restrictions on the target site (e.g., robots.txt).
400	ReadPage.RequestTimeout	Request target url timeout.	The request timed out. Extend the timeout period as needed.
429	ReadPage.RateLimitByDomain	The domain has reached the rate limit.
500	ReadPage.UnknownError	Unknown error.

Additional information

1. Stealth mode

This API enables stealth mode by default. This mode optimizes the browser environment's fingerprint to more closely resemble that of a real user's browser. This improves the compatibility and stability of automated access.

Basic fingerprint detection

2. Limitations

This API complies with the target website's robots.txt protocol, user agreements, and relevant laws and regulations.
This API only provides technical support and page parsing capabilities. We are not responsible for the generation, publication, display, or use of content from third-party websites. We make no promises or guarantees of any kind for third-party content that is scraped, accessed, or otherwise obtained through this service.

API calls

Examples

Python SDK

Prerequisites

Ensure that you have Python 3.8 or later installed.

Install the SDK

pip install alibabacloud_iqs20241111==1.6.0

Sample code

import json

from Tea.exceptions import TeaException
from alibabacloud_iqs20241111 import models
from alibabacloud_iqs20241111.client import Client
from alibabacloud_tea_openapi import models as open_api_models


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client() -> Client:
        config = open_api_models.Config(
            # TODO: Replace with your AccessKey and Secret. We recommend loading them from environment variables.
            access_key_id="$YOUR_ACCESS_KEY",
            access_key_secret="$YOUR_ACCESS_SECRET"
        )
        config.endpoint = f"iqs.cn-zhangjiakou.aliyuncs.com"
        return Client(config)

    @staticmethod
    def main() -> None:
        client = Sample.create_client()
        run_instances_request = models.ReadPageScrapeRequest(
            body=models.ReadPageScrapeBody(
                url="http://www.example.com",
                max_age=0,
            )
        )
        try:
            response = client.read_page_scrape(run_instances_request)
            print(f"API call successful. request_id: {response.body.request_id}, result: ")
            print(f"{json.dumps(response.body.data.to_map(), indent=2)}")

        except TeaException as e:
            request_id = e.data.get("requestId")
            code = e.data.get("errorCode")
            message = e.data.get("errorMessage")
            print(f"API call failed. requestId: {request_id}, code: {code}, message: {message}")


if __name__ == "__main__":
    Sample.main()

Java SDK

Prerequisites

Ensure that you have Java 8 or later installed.

Maven dependency

<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>iqs20241111</artifactId>
    <version>1.6.0</version>
</dependency>

Sample code

package com.aliyun.iqs.readpage.example;

import com.aliyun.iqs20241111.Client;
import com.aliyun.iqs20241111.models.*;
import com.aliyun.teaopenapi.models.Config;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class Example {
    public static void main(String[] args) throws Exception {
        Client client = initClient();
        invoke(client, "http://www.example.com");
    }

    private static Client initClient() throws Exception {
        // TODO: Replace with your AccessKey and Secret. We recommend loading them from environment variables.
        String accessKeyId = "$YOUR_ACCESS_KEY";
        String accessKeySecret = "$YOUR_ACCESS_SECRET";

        Config config = new Config()
                .setAccessKeyId(accessKeyId)
                .setAccessKeySecret(accessKeySecret);

        config.setEndpoint("iqs.cn-zhangjiakou.aliyuncs.com");
        return new Client(config);
    }

    private static void invoke(Client client, String url) {
        ReadPageScrapeBody input = new ReadPageScrapeBody();
        input.setUrl(url);

        ReadPageScrapeRequest request = new ReadPageScrapeRequest().setBody(input);

        try {
            ReadPageScrapeResponse response = client.readPageScrape(request);

            printOutput(response.getBody());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static void printOutput(ReadPageBasicResponseBody output) {
        // Create a Gson instance with pretty printing using GsonBuilder.
        Gson gson = new GsonBuilder()
                .setPrettyPrinting()
                .disableHtmlEscaping()
                .create();

        // Print the formatted JSON.
        String prettyJson = gson.toJson(output);
        System.out.println(prettyJson);
    }
}

Go SDK

Prerequisites

Ensure that you have Go 1.10.x or later installed.

Install the SDK

require (
  github.com/alibabacloud-go/iqs-20241111 v1.6.0
)

Sample code

package main

import (
	"fmt"
	"log"

	openapi "github.com/alibabacloud-go/darabonba-openapi/v2/client"
	iqs20241111 "github.com/alibabacloud-go/iqs-20241111/client"
	util "github.com/alibabacloud-go/tea-utils/v2/service"
	"github.com/alibabacloud-go/tea/tea"
)

const endpointURL = "iqs.cn-zhangjiakou.aliyuncs.com"

func createClient() (*iqs20241111.Client, error) {
	// TODO: Replace with your AccessKey and Secret.
	accessKeyID := "YOUR_ACCESS_KEY"
	accessKeySecret := "YOUR_ACCESS_SECRET"

	if accessKeyID == "" || accessKeySecret == "" {
		return nil, fmt.Errorf("ACCESS_KEY or ACCESS_SECRET environment variable is not set")
	}

	config := &openapi.Config{
		AccessKeyId:     tea.String(accessKeyID),
		AccessKeySecret: tea.String(accessKeySecret),
		Endpoint:        tea.String(endpointURL),
	}

	return iqs20241111.NewClient(config)
}

func runReadPage(client *iqs20241111.Client) error {
	body := &iqs20241111.ReadPageScrapeBody{
		Url: tea.String("http://www.example.com"),
	}
	request := &iqs20241111.ReadPageScrapeRequest{
		body,
	}
	runtime := &util.RuntimeOptions{}

	resp, err := client.ReadPageScrapeWithOptions(request, nil, runtime)
	if err != nil {
		return fmt.Errorf("readpage failed: %w", err)
	}

	fmt.Printf("[%s] response: %s\n", *resp.Body.RequestId, resp.Body)
	return nil
}

func main() {
	client, err := createClient()
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	if err := runReadPage(client); err != nil {
		log.Fatalf("Error running readpage: %v", err)
	}
}

HTTP call

Request parameters (Request Body)

curl --location "https://cloud-iqs.aliyuncs.com/readpage/scrape" \
--header "Content-Type: application/json" \
--header "X-API-Key: <YOUR-IQS-API-KEY>" \
--data '{
    "url": "https://www.example.com",
    "maxAge": 0
}'

Back

{
  "data": {
    "html": "<html>\n<head><title>Example Domain</title></head>\n<body>\n<div>\n<h1>Example Domain</h1>\n<p>This domain is for use in documentation examples without needing permission. Do not use it in operations.</p>\n<p><a href=\"https://iana.org/domains/example\">Learn more</a></p>\n</div>\n</body>\n</html>",
    "links": {
      "internal": "[]",
      "external": "[{\"href\":\"https://iana.org/domains/example\",\"text\":\"Learn more\",\"title\":\"\"}]"
    },
    "markdown": "# Example Domain\nThis domain is for use in documentation examples without needing permission. Do not use it in operations.\n[Learn more](https://iana.org/domains/example)\n",
    "media": {
      "images": "[]",
      "audios": "[]",
      "videos": "[]"
    },
    "metadata": {
      "hostname": "www.example.com",
      "pdfParse": false,
      "title": "Example Domain",
      "url": "https://www.example.com"
    },
    "statusCode": 200,
    "text": "# Example Domain\nThis domain is for use in documentation examples without needing permission. Do not use it in operations.\nLearn more\n"
  },
  "requestId": "1d0ac13a-8c73-4134-a835-35d0126f733c"
}