Data push node

更新时间:
复制 MD 格式

The data push node sends query results from other nodes in your Data Studio workflow to destinations such as DingTalk, Lark, WeCom, Teams, or email by creating a data push destination. This allows your team members to receive and stay informed of the latest data in a timely manner.

Background

A data push node uses Node Context Parameters to retrieve the output parameters from upstream SQL query nodes, such as an assignment node, Hologres SQL node, or ClickHouse SQL node. You can then edit these parameters in the push body as placeholders and send the content to the destination.

image
  • After an upstream SQL query node finishes running, it generates a node output parameter named outputs containing the query results.

  • In the downstream data push node, use Node Context Parameters to retrieve the outputs parameter from the upstream node and bind it as an input parameter of the data push node.

  • In the data push node configuration, reference the input parameter as the push content, set the destination, and push the content.

Supported upstream SQL query nodes:

Prerequisites

Limitations

  • Data size limits for different push destinations are as follows.

    Push destination

    Push limits

    DingTalk

    The push data size cannot exceed 20 KB.

    Lark

    The push data size cannot exceed 20 KB, and images must be smaller than 10 MB.

    WeCom

    Each bot can send no more than 20 messages per minute.

    Teams

    The push data size cannot exceed 28 KB.

    Email

    Each data push task supports only one email body. If an email body has already been added, you cannot add another one. For more limits, refer to the SMTP limits of your email service.

  • Supported regions for Data Push:

    China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), China (Chengdu), China (Hong Kong), Singapore, Japan (Tokyo), US (Silicon Valley), US (Virginia), and Germany (Frankfurt).

Go to Data Studio

  1. Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.

  2. In the left-side navigation pane, click the image icon to go to the Data Studio page.

Create a push pipeline

Create and configure a pipeline that consists of an SQL query node and a data push node.

  1. Create a workflow, and then create an SQL query node and a data push node in the workflow.

  2. Create an SQL query node and add the SQL code for the data you want to query and push.

    Note
    • The data push node does not support directly retrieving query data from ODPS SQL. You can create an assignment node and query the data to be pushed in the assignment node.

    • Other SQL query nodes can be created as usual.

  3. Configure the SQL query node.

    You must configure the Compute Resource, Resource Group for Scheduling, and Same-cycle Dependency in the Scheduling Settings panel on the right side of the node. If no upstream node exists, you can select Add Root Node. The output parameter configuration differs as follows:

    Node type

    Configuration notes

    Assignment node

    An assignment node has a default node output parameter. No additional output parameter configuration is required.

    Other SQL query nodes

    Other SQL query nodes do not have default node output parameters. You must go to Node Context Parameters in the Scheduling Settings panel on the right side of the node, find Node output parameters, and click Add Assignment Parameter to pass the current query results to the downstream data push node through parameters.

  4. You have created a data push node and configured the SQL query node as the upstream node.

    1. Click Scheduling Settings. In the Scheduling Dependency > Same-cycle Dependency section, click Add and search by node name to add the SQL query node you created as the upstream node of the data push node.

    2. In the Scheduling Policy > Resource Group for Scheduling section, select the serverless resource group you created.

    3. In the Node Context Parameters > Input Parameters section, click Add parameters to add the output parameter of the SQL query node as the value source and bind it as the input parameter of the data push node.

Configure push destinations and content

On the data push node editing page, customize the push title and configure the push destination and body content.

Configure the push destination

In the Destination section, select a destination.

  1. If the destination does not exist, click Create Destination. You can also set a data push destination in Data Service.

    Note

    To modify a data push destination, go to the Data Push destination management page in Data Service.

  2. On the Create Destination page, create a data push destination by configuring the following parameters.

    Create a DingTalk, Lark, WeCom, or Teams push destination

    Parameter

    Description

    Destination

    Select DingTalk, Flying Book, WeCom, or Teams.

    Destination Name

    You can customize the name based on your business needs.

    Webhook

    Obtain and configure the Webhook URL based on the selected push destination.

    Create an email push destination

    Parameter

    Description

    Destination

    Select Mail.

    SMTP Host

    The address of the SMTP server used to send emails.

    SMTP Port

    The port number of the SMTP server used to connect to the mail server. Default value: 465. You can manually change this value.

    Sender Address

    The email address from which emails are sent.

    Sender Nickname

    Optional. A custom display name for the sender.

    SMTP Account

    The email account used to log on to the SMTP server.

    SMTP Password

    The password used to authenticate the SMTP account.

    Receiver Address

    The email addresses of the recipients. Separate multiple addresses with commas (,).

Configure the push body

You can configure the body content in three formats: Markdown, table, and email body.

  • In the Markdown body of a data push, you can directly use the parameter name of the Input Parameters of the data push node as a placeholder in the format ${input_parameter_name} to retrieve the output parameter from the upstream node.

  • In the table body of a data push, you can use the column names of the upstream SQL query node as Parameters to retrieve data.

  • DataWorks Data Push supports adding an email body to the push content. When editing the email body, note the following:

    • Each data push task supports only one email body.

    • The email body is rendered only when the push destination is email. When the push destination is not email, the email body is hidden in the message pushed via Webhook.

Debug the data push

After the configuration is complete, save the SQL query node and data push node. Then, click the image icon in the toolbar above the workflow to debug the data push task.

Next steps

  • Configure schedule settings: If you want nodes in the project directory to be executed on a periodic schedule, configure the Scheduling Policy in the Scheduling Settings panel on the right side of the node.

  • Deploy a node: If you want to deploy tasks to the production environment, click the image icon on the page to initiate the deployment process. Nodes in the project directory are periodically scheduled only after they are deployed to the production environment.