边缘函数HTMLStream API-Edge Security Acceleration(ESA)-阿里云帮助中心

Background information

Frontend modification is a common use case for EdgeRoutine. You often need to alter HTML responses at the edge based on dynamic request data, such as User-Agent headers, geographic information, or IP addresses. Traditional regular-expression-based methods are error-prone and unsuitable for streaming. Open-source JavaScript parsers like parse5 or htmlparser2 can perform this task but consume significant memory and incur high performance overhead. To address these challenges, EdgeRoutine includes a built-in streaming HTML parser that efficiently modifies HTML code and pages.

Note

The parser is built in EdgeRoutine and is not based on web standards.

Example

Scenario

Suppose you need to modify all <a/> tags in an HTML page to point to http://www.taobao.com. You can use the following code in EdgeRoutine to do this.

Sample code

addEventListener('fetch', (event) => {
  event.respondWith(handle(event));
});

async function handle(event) {
  // 1. Fetch the response that contains the HTML page to modify.
  const response = await fetch("http://www.example.com");
  // 2. Set up the streaming HTML parser. You can register rewriters 
  //    for various CSS selectors to modify the content.
  const htmlStream = new HTMLStream(
    response.body, // The HTML stream to rewrite.
    [[
      "a",         // An element selector that matches all `a` tags.
      {  
        // Register a handler object. The `element` handler is called for each `a` tag found.
        element: function(e) {
          // Change the 'href' attribute.
          e.setAttribute("href", "http://www.taobao.com");
        }
      }
    ]]);
  
  // 3. Return the modified stream to the browser. HTMLStream is a ReadableStream,
  //    so it can be used anywhere a ReadableStream is accepted.
  return new Response(htmlStream);
}

Result analysis
The preceding sample code modifies an HTML flow in real time by using the HTMLStream API. The following section describes how HTMLStream works:
- The Fetch API retrieves a flow expression for the request. However, EdgeRoutine may not have retrieved the response body from the network layer, which reduces the frequency of garbage collection caused by data buffering.
- HTMLStream acts as a TransformStream that takes an input stream and uses your registered rewrite handlers to modify the HTML page in real time. To modify an HTML page, obtain its original data stream and pass it as input to a new HTMLStream, as shown in step 2 of the code example.
  - The first parameter of HTMLStream is a stream, which represents the raw data stream of the HTML page.
  - The second parameter of HTMLStream is an array of rewriters. A rewriter is a two-element array containing a selector and a handler object. The properties of this handler object are callback functions. For example, ["a" , {....}] declares a rewriter. The first element, "a", is an element selector that finds all a tags in the document. The second element is the handler object. When using an element selector, this object can contain the following three functions:
    - The element function, with the signature function(e), is called when a matched element is parsed.
    - The comments function, with the signature function(e), is called when a comment nested within the element is parsed.
    - The text function, with the signature function(e), is called when text within the element is parsed. This function can be called multiple times, as the streaming parser may be processing a fragment of a string.
- You can directly respond to HTMLStream. However, HTMLStream does not buffer data. Unlike parse5 and htmlparser2, HTMLStream does not generate a DOM tree, which significantly reduces processing time and memory consumption. This enables high throughput and concurrency when parsing HTML content.

Rewriter

A rewriter registers the object that you want to rewrite. It is an array of two elements.

The first element in the array must be of the string type or null.
- String: specifies an element selector that is used to locate an element or tag.
- null: specifies that the rewriter applies to an entire document.
  
  Note
  In most cases, you do not need to apply the rewriter to an entire document. If the rewriter is applied to an entire document, it cannot locate elements.
The second element in the array must be a JavaScript object. This object is returned to the callback function that you have registered.

If you use an element selector, this object is named as the element callback function. If you use a document selector, this object is named as the document callback function.

Note

You can specify one or more rewriters for an HTMLStream operation. You can specify multiple element selectors but only one document selector.

Syntax of element selectors

The element selector syntax is a subset of CSS selector syntax. The programming language of an element selector may be different from the programming language of a CSS selector. The following section describes the supported patterns:

*: Matches all elements or tags.
div: Matches tags named div. This pattern applies to other tag names, including standard HTML tags and custom tags.
E#id: Matches an element E with an id attribute value of id.
E.Class: Matches an element E that has a class attribute containing the class name Class.
E[attr]: Matches an element E that has an attribute named attr.
Element attributes:
- E[attr="a"]: Matches element E, where the attribute attr has a value of "a". The match is case-sensitive.
- E[attr^="a"]: Matches element E, where the value of the attr attribute starts with "a".
- E[attr$="a"]: Matches element E, where the value of the attr attribute ends with "a".
- E[attr^="a"]: Selects an E element whose attr attribute value starts with "a".
- E[attr*="a"]: Matches element E, where the value of the attr attribute contains "a".
- E[attr|="a"]: Matches element E, where the attr attribute value is either exactly "a" or begins with "a-". For example, [lang|="en"] matches elements with lang="en" or lang="en-US".
Order between elements:
- E F: Matches an element F that is a descendant of an element E.
- E > F: Matches an element F that is a direct child of a parent element E.
E:not(S): Matches an element E that does not match the selector S.

Callback functions for element selectors

The following table describes the callback functions supported by element selectors.

Callback function	Description	Callback function signature
element	A non-asynchronous callback function that is called after the selected elements are completely parsed.	The signature of the callback function is function(e). This signature is carried in the Element object. For more information, see Element.
comments	A non-asynchronous callback function that is called when the selected elements have comments.	The signature of the callback function is function(e). This signature is carried in the Comments object. For more information, see Comments.
text	A non-asynchronous callback function that is called when the text returned to the callback function is parsed.	The signature of the callback function is function(e). This signature is carried in the TextChunk object. For more information, see TextChunk. Note This callback function may be called multiple times. When HTMLStream reads chunks of text from the raw HTML data, this callback function is called each time a chunk is parsed. If you want to view the complete text, you must load and merge all the text chunks.

Note

An element selector can ignore all the preceding callback functions. In this case, relevant elements are printed directly without processing. If you want to modify a specific chunk of the text, you need only register the required callback function.

Document selector

A document selector selects an entire document. To use a document selector, set the first element in the rewriter array to null. Only one document selector is allowed per HTMLStream operation.

Callback functions for document selectors

Document selector callback functions are similar to element selector callback functions. The following table describes the supported callbacks.

Callback function	Description	Callback function signature
doctype	A non-asynchronous callback function that is called when the document type declaration (DOCTYPE) in the specified document is parsed.	The signature of the callback function is function(e). This signature is carried in the Doctype object. For more information, see Doctype.
comments	A non-asynchronous callback function that is called when the specified document has comments.	The signature of the callback function is function(e). This signature is carried in the Comments object. For more information, see Comments.
text	A non-asynchronous callback function that is called when the specified document has text nodes.	The signature of the callback function is function(e). This signature is carried in the TextChunk object. For more information, see TextChunk. Note This callback function may be called multiple times. When HTMLStream reads chunks of text from the raw HTML data, this callback function is called each time a chunk is parsed. If you want to view the complete text, you must load and merge all the text chunks.
docend	A non-asynchronous callback function that is called after the specified document is completely parsed. This callback function appends content such as debugging information to the end of the HTML document as comments. You can use this information for troubleshooting.	The signature of the callback function is function(e). This signature is carried in the Docend object. For more information, see Docend.

Error handling

EdgeRoutine catches all JavaScript exceptions thrown by the preceding callback functions. HTMLStream then stops processing HTML streams and propagates the exception to the outer layers.

If the reader.read method is triggered in JavaScript, the exceptions are thrown again.
If the reader.read method is called in HTMLStream when EdgeRoutine is running, EdgeRoutine hides the exceptions. For example, if the exceptions occur when EdgeRoutine returns a response to a client, the response is interrupted and the client receives only a part of the response. This is because HTMLStream treats data as streams. In this case, the stream may be interrupted before all the data is returned to the client. The method used by HTMLStream to process exceptions is similar to the method used by TransformStream. TransformStream also writes and reads data as streams.

Callback parameters

Each callback function receives an object that represents the selected HTML tags or other relevant information. This topic describes the callback parameters: Element, TextChunk, and Comments.

Note

All parameters must be passed into callback functions. If the methods or attributes of a parameter are invoked outside of a callback function, JavaScript exceptions are thrown. To avoid this problem, you can pass the desired parameters into other JavaScript objects or data structures.
Each option in this document represents an object. The html property of this object can be set to true or false to specify whether the content is interpreted as HTML content or plain text. If the html property is set to false, HTMLStream performs html encoding/escaping on the content.

Element

Definition

This object is returned when the Element callback function is called. This object represents the selected HTML tags.
Attributes
- tagName (string): The name of the tag.
- attributes (iterator): An iterator that yields all the element's attributes as [name, value] pairs.
- removed (boolean, read-only): Indicates whether the element has been removed. Use the remove() method to remove an element. You should typically check this property to skip elements that have already been removed.
- namespaceURI (string, read-only): The namespace URI of the element, such as for svg or math elements.
Methods
- Modify attributes
  - getAttribute(name): queries an attribute name of a specified element.
  - setAttribute(name, value): Sets or modifies the value of an attribute.
  - hasAttribute(name): queries whether an attribute name exists in a specified element.
  - removeAttribute(name): deletes an attribute name from a specified element.
  Note
  Both the attribute name and value must be of string type.
- Modify content
  - before(data, option): inserts content before the specified element (element tag).
  - after(data, option): inserts content after the specified element (element tag).
  - prepend(data, option): Inserts content right after the element's opening tag. For example: <div>(inserted content) ... </div>.
  - append(data, option): Inserts content right before the element's closing tag. For example: <div>... (inserted content)</div>.
  - replace(data, option): replaces the entire element, including the tags and nested tags.
  - setInnerContent(data, option): specifies the element content and retains the tags and attributes.
  - remove(): deletes the specified element. After the element is deleted, the value of the removed attribute changes to true.
  - removeAndKeepContent(): deletes tags and attributes of the specified element and retains the content.

TextChunk

Definition

This object is returned when the Text callback function is called. This object represents a chunk of the selected HTML text.
Attributes
- removed (boolean, read-only): Indicates whether the text chunk has been removed.
- text (string, read-only): The content of the text chunk. This may be only a part of a larger text node. If this string is empty, it indicates the end of the text node.
- lastInTextNode (boolean, read-only): Indicates if this is the last chunk of the text node. If true, the text property returns an empty string.
Methods
Modify content
- before(data, option): inserts content before the specified element (element tag).
- after(data, option): inserts content after the specified element (element tag).
- replace(data, option): replaces the entire element, including the tags and nested tags.
- remove(): deletes the specified element. After the element is deleted, the value of the removed attribute changes to true.

Comments

Definition

This object is returned when the Comments callback function is called. This object represents the comments in the selected HTML content.
Attributes
- removed (boolean, read-only): Indicates whether the comment has been removed.
- text (string): The content of the comment. This property is readable and writable.
Methods
Modify content
- before(data, option): inserts content before the specified element (element tag).
- after(data, option): inserts content after the specified element (element tag).
- replace(data, option): replaces the entire element, including the tags and nested tags.
- remove(): deletes the specified element. After the element is deleted, the value of the removed attribute changes to true.

Doctype

Definition

This object is returned when the DOCTYPE callback function is called. This object represents the DOCTYPE of the selected HTML content.
Attributes
- name (string, read-only): The name of the doctype.
- publicId (string, read-only): The public identifier, or null if not present.
- systemId (string, read-only): The system identifier, or null if not present.

Docend

Definition

This object is returned when the Docend callback function is called. This object represents the end of an HTML document.
Method

append(string, option): appends content to the end of the HTML document.