HTTP node-DataWorks(DataWorks)-阿里云帮助中心

Permissions

The RAM account used for node development must be added to the corresponding workspace and have the developer or workspace administrator role. For more information, see Add workspace members.

Limitations

Important

This feature is currently in beta testing. If a downstream node needs to consume parameters from an HTTP node, contact technical support to upgrade the downstream node's image.

Resource group: HTTP nodes can run only on a Serverless resource group. Ensure that your workspace is associated with an available Serverless resource group.
Protocol support: Only the HTTP and HTTPS protocols are supported. It automatically parses the protocol from the URL you provide.
Network access: If the target endpoint is protected by an IP allowlist, you must add the egress IP addresses of the Serverless resource group to the endpoint's allowlist. For more information, see Add the IP addresses of a Serverless resource group to an allowlist.
Outputs: The HTTP node writes the entire response to the outputs parameter. The maximum transfer size is 2 MB. When a downstream node consumes data by referencing outputs, we recommend that the response content be in JSON format for structured parsing.
Synchronous execution: The HTTP node is a synchronous node and is not suitable for long-lived connections or long-polling scenarios. For external tasks that require polling, use a workflow orchestration pattern of "trigger + multiple HTTP queries".

Quick start

This example shows how to create, debug, configure, and deploy an HTTP node using the public endpoint https://httpbin.org/get.

Develop the node

Log on to the DataWorks console. After switching to the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select the target workspace from the drop-down list and click Enter.
On the Data Studio page, create an HTTP node.

On the HTTP node editing page, complete the request configuration. The following table lists the main properties. For a complete description, see Node properties.

Parameter	Example value
Request URL	`https://httpbin.org/get`
Request method	`GET`
Authentication	No Auth
Validation condition	Default status code (a `2xx` response code is considered a success)

After completing the configuration, click Run Configuration on the right side. Select the target Serverless debugging resource group and other required execution parameters. Then click the Run button to trigger a local debug run. After the run completes, view the HTTP response status code and response body in the Result panel at the bottom.
After debugging is successful, click Properties on the right side. Configure production-level scheduling policies such as the schedule, dependencies, and parameters to enable the node to run automatically on a periodic basis.
After completing the task scheduling configuration, you must click Save before proceeding to the next step.

Deploy and operate the node

After completing the schedule settings, you can submit and deploy the HTTP node to the production environment for formal deployment from development to production. For detailed steps and the deployment process, see Deploy nodes.
After the deployment succeeds, the task runs periodically according to your schedule settings. Go to Operation Center > Scheduled Task Operations > Scheduled Tasks to view the deployed HTTP node and check the request and response details in the instance logs. Operation Center displays HTTP nodes in the same way as assignment nodes, with a focus on output parameters. For more information, see View scheduled instances.

Node properties

HTTP node properties fall into five categories: basic information, request parameters, timeout settings, authentication, and validation conditions.

Basic information

Parameter	Required	Description
Request URL	Yes	The complete HTTP or HTTPS request URL, such as `https://api.example.com/v1/users`. You can reference scheduling parameters in the URL, such as `https://api.example.com/v1/logs?date=${bizdate}`.
Request method	Yes	The following six request methods are supported: `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, and `HEAD`. The request parameter fields dynamically change based on the selected request method. For example, the Body configuration is not displayed for `GET` requests.

Request parameters

Request parameters support three types of key-value pairs, all of which can reference scheduling parameters.

Parameter type	Description
Query parameters	URL query string parameters, appended to the end of the URL in the format `?key=value&key2=value2`. Applicable to all request methods.
Headers	Request headers. You can specify key-value pairs such as `Content-Type`, `Accept`, and custom business headers. Applicable to all request methods.
Body	The request body content. You can specify JSON, form data, or plain text. The body is available only for `POST`, `PUT`, and `PATCH` requests.

Timeout settings

Parameter	Default value	Description
Connection timeout	30000 ms	The maximum time to wait for a TCP connection to be established. If the connection is not established within this period, the request is considered failed.
Socket timeout	60000 ms	The maximum time to wait for data to be read after the connection is established. If no response data is received within this period, the request is considered failed.

Authentication

The HTTP node supports four authentication methods. Select the one that matches your target endpoint's requirements.

Authentication method	Description
No Auth	No authentication information is included. This option is suitable for public endpoints or internal endpoints that are authorized through an IP allowlist.
Basic Auth	HTTP basic authentication based on a username and password. After you configure the username and password, the HTTP node automatically adds `Authorization: Basic <base64(user:password)>` to the request header.
Bearer Token	A static token that the HTTP node includes in the request header. This option is suitable for scenarios that use pre-issued long-lived tokens.
API-Key	Authentication through an API key. Add a custom header (such as X-API-Key) in the request headers.

Note

Sensitive credentials such as passwords, tokens, and client secrets should be injected through DataWorks workspace secret parameters to avoid storing them in plaintext in the node configuration.

Validation conditions

Validation conditions determine whether a request succeeded. The node runs successfully only when all specified conditions are met. The following validation methods can be combined.

Validation type	Description
Status code validation	Default behavior: An HTTP response status code of `2xx` is considered a success, and any other response code is considered a failure. When this option is selected, a response code input field appears. You can specify the expected response codes (such as `200,201,204`). The node is considered successful only when the actual response code matches one of the specified codes.
Content contains	When this option is selected, a content input field appears. You can specify a string that is expected to appear in the response. The node is considered successful only when the response body contains the specified string. This option is suitable for endpoints whose response codes are unreliable but whose response content contains fixed identifiers.
Content does not contain	When this option is selected, a content input field appears. You can specify a string that should not appear in the response (such as `"error"` or `"failed"`). The node is considered successful only when the response body does not contain the specified string.

Schedule settings and parameters

HTTP node schedule settings work the same as other scheduled node types such as MaxCompute SQL nodes. You can configure the schedule, dependencies, scheduling parameters, and other settings. For more information, see Configure schedule settings.

Scheduling parameters can be referenced in the Request URL, Query Parameters, Headers, and Body by using the ${parameter_name} syntax. Examples:

Reference the business date in the request URL to retrieve data by partition: https://api.example.com/v1/logs?date=${bizdate}.
Construct a JSON body by using scheduling parameters: {"date":"${bizdate}","region":"cn-hangzhou"}.

Note

Only a Serverless resource group can be selected as the scheduling resource group.
To reference an output parameter from an upstream node, add a parameter in Schedule Settings > Node Context > Input Parameters of This Node, and then inject it into the request by using the ${parameter_name} syntax.

Output parameters and downstream consumption

The HTTP node writes the entire response to the output parameter named outputs. Downstream nodes can reference this value through node context parameters. This mechanism works the same as the assignment node.

For easier structured parsing, we recommend that the target endpoint return responses in JSON format.

Example: Pass an HTTP response to a downstream Shell node

Upstream HTTP node: Calls https://httpbin.org/get. The response body is in JSON format.
Downstream Shell node configuration:
1. In the right-side panel of the Shell node, click Schedule Settings and select the Node Context tab.
2. In the Input Parameters of This Node section, click Add Parameter.
3. In the dialog that appears, select the output parameter of the upstream node as the outputs of the HTTP node, and specify a custom Parameter Name for the input parameter of the current node, such as resp.
  
  Note
  After the configuration is complete, the downstream node automatically establishes a dependency on the upstream HTTP node.
4. After completing the parameter configuration, you can use the response data from the upstream HTTP node in the Shell node code by using the ${resp} syntax.
Shell node sample code: Directly print the HTTP response content.
```
echo "HTTP response content: ${resp}"
```

FAQ

Q: What is the difference between an HTTP node and an HTTP trigger node?

A: The two serve completely different purposes.
- HTTP node (this topic): An execution node that actively sends HTTP requests. It is run by DataWorks scheduling, sends requests to external endpoints, and retrieves responses.
- HTTP trigger node: A virtual node that passively waits to be triggered. It waits for an external system to trigger it through the DataWorks OpenAPI, which then starts the scheduling of its downstream nodes.
Q: Why can HTTP nodes run only on Serverless resource groups?

A: HTTP nodes depend on the runtime environment of Serverless resource groups. Exclusive resource groups for scheduling and shared resource groups do not yet support this node type. Ensure that your workspace is associated with an available Serverless resource group.
Q: The response is not in JSON format. How can downstream nodes consume it?

A: When the response is not in JSON format, outputs stores the raw response content as a string. Downstream nodes can retrieve the entire string by referencing ${parameter_name} and then parse it in the downstream node (Shell, Python, etc.) as needed.
Q: The target endpoint has an IP allowlist restriction and the request is rejected. What should I do?

A: Add the egress IP addresses of the Serverless resource group bound to your workspace to the IP allowlist of the target endpoint. For more information, see Add the IP addresses of a Serverless resource group to an allowlist.
Q: Request timeouts occur frequently. How can I optimize the configuration?

A: Adjust the connection timeout and socket timeout values based on the actual response time of the called endpoint. If the endpoint itself takes a long time to respond, we recommend that you use an asynchronous task orchestration pattern: use an HTTP node to trigger the task, and then use a do-while node combined with HTTP nodes to poll the task status, instead of having a single HTTP node wait for an extended period.

References

Assignment node: Learn about the general mechanism for output parameters and downstream consumption.
HTTP trigger node: Learn about the virtual node that is triggered by an external OpenAPI call, which complements the HTTP node.
Configure schedule settings: Learn about the common schedule settings in DataWorks.