DocsICP FilingConsole
官方文档
首页

AI data inference solution: AI-powered data processing

更新时间：
复制 MD 格式
产品详情
我的收藏

Manual deployment

25

Overview

EventBridge integrates with AI services to perform intelligent inference on real-time data flowing through an event bus. This enhances existing data processing capabilities and enables innovative new use cases.

By using EventBridge as a central data hub, you can distribute real-time event streams from various data sources to different AI models for inference. The results are then routed back through EventBridge to downstream business systems or data storage. Incorporating AI inference into the transform step of your ETL process enables more intelligent data transformation, increases processing efficiency and accuracy, and provides more powerful data integration and analytics capabilities.

Architecture

The following diagram shows the system architecture that you will build on Alibaba Cloud by using this solution's default settings. Although you can modify the settings based on your resource plan, the final environment will be similar to the one shown.

image

This guide uses sensitive information filtering as an example to demonstrate data cleaning and masking of downstream data. By automatically identifying sensitive business keywords with AI, you can ensure data compliance for downstream services.

The expected outcome is as follows: an input containing sensitive information, such as ["Customer John Doe (13812345678) reported an issue..."], will produce a masked output like ["Customer *** (138*****5678) reported an issue..."]. Because large model outputs can vary, the format of your test results may differ. In a production environment, you can use prompt engineering to achieve more consistent outputs.

This solution uses the following infrastructure and cloud services:

Resources

Description

1 event stream

Used to build the data pipeline.

2 Message Service (MNS) queues

Serve as the data source and sink.

Alibaba Cloud Model Studio (Bailian)

Processes data intelligently by invoking a large model.

Prerequisites

Before you begin, set up your account and ensure it is adequately funded.

Prepare your account

  1. 如果您还没有阿里云账号，请访问阿里云账号注册页面，根据页面提示完成注册。阿里云账号是您使用云资源的付费实体，因此是部署方案的必要前提。

  2. 为阿里云账号充值。本方案的云资源支持按量付费，且默认设置均采用按量付费引导操作。如果确定任何一个云资源采用按量付费方式部署，账户余额都必须大于等于100元。

Activate services

  1. Log on to Model Studio. After you read and agree to the Terms of Service, Model Studio is automatically activated. If the Terms of Service do not appear, this indicates that you have already activated Model Studio.

    If you are prompted to complete identity verification when activating the service, you must first complete individual identity verification.

    Note

    After you activate Model Studio for the first time, the platform automatically grants you a free quota for new users for each model. This quota is valid for 30 to 90 days and can be used for model inference services. You are charged for any usage that exceeds the free quota or its validity period. For pricing details, see the prices on the console and your final bill. For more information about how to claim the free quota, see Free quota for new users.

  2. If this is your first time using Message Service (MNS), follow these steps to activate it.

    1. Log on to the Message Service (MNS) console.

    2. On the Overview page, click Activate for Free. Follow the on-screen instructions to complete the activation.

  3. If this is your first time using EventBridge, follow these steps to activate it.

    1. Log on to the EventBridge console.

    2. On the Overview page, find the Activate EventBridge wizard and click Activate for Free. Follow the on-screen instructions to complete the activation.

    3. EventBridge requires authorization to receive events. On the Overview page, find the Access Authorization wizard and click One-Click Authorization to grant the necessary permissions.

Obtain an API key

Model Studio API key

  1. Go to the Key Management page of Model Studio.

  2. On the API-Key tab, you can create or view API keys.

    Note

    1. A RAM user requires authorization from the Alibaba Cloud account before creating an API key.

    2. Do not expose your API key. Unauthorized use can cause security risks and financial losses.

  3. Click the Copy button to copy the API Key.

Create a source and a sink

10

In this guide, you will create two Message Service (MNS) queues to serve as the source and sink for the event stream.

MNS queues

  1. Log on to the Message Service (MNS) console.

  2. In the left-side navigation pane, click Queue List. For Region, select China (Shenzhen).

  3. Click Create Queue. For Name, enter eventbridge-dev. Keep the default settings for the other options.

  4. Click Confirm and wait for the queue to be created.

  5. Repeat the steps to create another queue named eventbridge-online.

Important

If you experience duplicate message consumption, we recommend changing the Message Visibility Timeout to a value between 60 and 300 seconds.

Build the data pipeline

10

Create and configure an EventBridge event stream to act as the data pipeline.

Event stream and source

  1. Log on to the EventBridge console.

  2. In the left-side navigation pane, click Event Streams. For Region, select China (Shenzhen).

  3. Click Create Event Stream and configure the following parameters:

    1. Enter a custom Task Name.

    2. For Metering Method, select Pay-by-event.

  4. On the Task Creation page, in the Source section, configure the following parameters:

    1. For Data Provider, select Message Service (MNS).

    2. For Queue Name, select the eventbridge-dev queue that you created. Keep the default settings for the other options.

  5. Click Next.

Filtering

  1. In the Filtering section, configure the following parameter:

    1. For Pattern Content, select All Events Match. For real-world use cases, you can also define flexible filtering rules by selecting other options or by creating a custom pattern.

  2. Click Next.

Transform

  1. In the Transformation section, configure the following parameters:

    1. For Alibaba Cloud Service, select Model/Agent Call (Alibaba Cloud Model Studio). This guide uses Model Call as an example.

    2. For Model Name, enter qwen-max.

    3. Configure the Model Context as shown in the following table:

      SYSTEM

      Fixed value

      You are a data compliance officer responsible for ensuring that all data entering the analytics system complies with GDPR/CCPA regulations and does not contain any personally identifiable information (PII). Your task is to automatically identify and mask names, phone numbers, and ID card numbers in text during the ETL process. You also need to block sensitive content related to religion, sex, and politics.

      USER

      Partial event

      $.data.messageBody

    4. For Structured Output, select Enabled. Click Add Field and add the fields shown in the following table:

      data

      string

      The masked data content.

      Required

      cause

      string

      The reason why the data was filtered, returned in English.

      Optional

    5. For API key, enter the Model Studio API key that you obtained.

  2. Click Next.

Sink

  1. In the Sink section, configure the following parameters:

    1. For Service type, select SMQ.

    2. For Queue Name, select the eventbridge-online queue that you created.

    3. For Message Body, select Partial Event and enter $.transform0.structuredoutput.

  2. Click Save. Wait for the event stream to be created, after which its Status changes to Running.

Verification

3

Send a message containing sensitive information from the source queue and check for the processed message in the sink queue to verify that the large model has correctly masked the data.

Send message

  1. Log on to the Message Service (MNS) console, and in the left-side navigation pane, click Queue List.

  2. Find the eventbridge-dev queue and click Send and Receive Messages in the Actions column.

  3. In the Send message section, configure the following parameters:

    1. For Message Content, enter Customer John Doe (13812345678) reported an issue.

    2. Set Message delay time to 0s.

  4. Click Send Message.

Receive message

  1. Log on to the Message Service (MNS) console, and in the left-side navigation pane, click Queue List.

  2. Find the eventbridge-online queue and click Send and Receive Messages in the Actions column.

  3. In the Receive message section, click Receive Message.

  4. The received message appears in the list. Click Details in the Actions column and view the Base64-decoded message content. A result similar to the one shown in the figure indicates that the solution has successfully processed the data.

Clean up resources

2

In this solution, you created one EventBridge event stream and two Message Service (MNS) queues. After you finish testing, follow these steps to delete the resources that you created and avoid further charges:

  1. Delete the EventBridge event stream: Log on to the EventBridge console, and in the left-side navigation pane, click Event Streams. Find the event stream you created, click Delete in the Actions column, and then follow the on-screen instructions.

  2. Delete the Message Service (MNS) queues: Log on to the Message Service (MNS) console, and in the left-side navigation pane, click Queue List. Find the queues you created, click the icon at the far right of the Actions column, click Delete, and then follow the on-screen instructions.

One-click deployment (Free trial)

Overview

EventBridge integrates with AI services to perform intelligent inference on real-time data flowing through an event bus. This enhances existing data processing capabilities and enables innovative new use cases.

By using EventBridge as a central data hub, you can distribute real-time event streams from various data sources to different AI models for inference. The results are then routed back through EventBridge to downstream business systems or data storage. Incorporating AI inference into the transform step of your ETL process enables more intelligent data transformation, increases processing efficiency and accuracy, and provides more powerful data integration and analytics capabilities.

Architecture

The following diagram shows the system architecture that you will build on Alibaba Cloud by using this solution's default settings. Although you can modify the settings based on your resource plan, the final environment will be similar to the one shown.

image

This guide uses sensitive information filtering as an example to demonstrate data cleaning and masking of downstream data. By automatically identifying sensitive business keywords with AI, you can ensure data compliance for downstream services.

The expected outcome is as follows: an input containing sensitive information, such as ["Customer John Doe (13812345678) reported an issue..."], will produce a masked output like ["Customer *** (138*****5678) reported an issue..."]. Because large model outputs can vary, the format of your test results may differ. In a production environment, you can use prompt engineering to achieve more consistent outputs.

This solution uses the following infrastructure and cloud services:

Resources

Description

1 event stream

Used to build the data pipeline.

2 Message Service (MNS) queues

Serve as the data source and sink.

Alibaba Cloud Model Studio (Bailian)

Processes data intelligently by invoking a large model.

One-click deployment

  1. Click the Try Now button on the right side of the page to start creating resources. The deployment will complete in a few minutes.

    Note

    If the page indicates that your account balance is less than the estimated trial points, you must acquire enough trial points before proceeding. To get more trial points, visit Free Trials.

  2. After the deployment is complete, in the RAM User Logon page on the right, click Next.

  3. In the left panel, under Cloud Resources > Trial Account, copy the Logon Password. Then, in the right panel, paste the password into the User Password field and click Logon.

  4. After logging on, on the Select an MFA Binding Method page, click Skip Binding in the upper-right corner.

Important

  • After the deployment is complete, click the Cloud Resources menu on the left to view information about all created cloud resources.

  • During the trial, if your trial period is approaching its limit, the system will prompt you to renew. You can check the Remaining Duration in the upper-right corner of the page and click Renew to extend the trial period. When the remaining time reaches zero, the system automatically releases all created resources.

  • In the following steps, if you need to access a cloud resource, copy its address from the Cloud Resources panel on the left and paste it into the browser on the right.

Activate services

  1. Log on to Model Studio. After you read and agree to the Terms of Service, Model Studio is automatically activated. If the Terms of Service do not appear, this indicates that you have already activated Model Studio.

    If you are prompted to complete identity verification when activating the service, you must first complete individual identity verification.

    Note

    After you activate Model Studio for the first time, the platform automatically grants you a free quota for new users for each model. This quota is valid for 30 to 90 days and can be used for model inference services. You are charged for any usage that exceeds the free quota or its validity period. For pricing details, see the prices on the console and your final bill. For more information about how to claim the free quota, see Free quota for new users.

  2. If this is your first time using Message Service (MNS), follow these steps to activate it.

    1. Log on to the Message Service (MNS) console.

    2. On the Overview page, click Activate for Free. Follow the on-screen instructions to complete the activation.

  3. If this is your first time using EventBridge, follow these steps to activate it.

    1. Log on to the EventBridge console.

    2. On the Overview page, find the Activate EventBridge wizard and click Activate for Free. Follow the on-screen instructions to complete the activation.

    3. EventBridge requires authorization to receive events. On the Overview page, find the Access Authorization wizard and click One-Click Authorization to grant the necessary permissions.

Obtain an API key

Model Studio API key

  1. Go to the Key Management page in the Model Studio console.

  2. On the API-Key tab, create or view your API key.

    Important

    1. A RAM user must be granted permission by the main account before creating an API key.

    2. Do not expose your API key publicly to avoid security risks or financial losses from unauthorized use.

  3. In the API Key column, click the copy icon image to copy the API key.

Create a source and a sink

10

In this guide, you will create two Message Service (MNS) queues to serve as the source and sink for the event stream.

MNS queues

  1. Log on to the Message Service (MNS) console.

  2. In the left-side navigation pane, click Queue List. For Region, select China (Shenzhen).

  3. Click Create Queue. For Name, enter eventbridge-dev. Keep the default settings for the other options.

  4. Click Confirm and wait for the queue to be created.

  5. Repeat the steps to create another queue named eventbridge-online.

Important

If you experience duplicate message consumption, we recommend changing the Message Visibility Timeout to a value between 60 and 300 seconds.

Build the data pipeline

10

Create and configure an EventBridge event stream to act as the data pipeline.

Event stream and source

  1. Log on to the EventBridge console.

  2. In the left-side navigation pane, click Event Streams. For Region, select China (Shenzhen).

  3. Click Create Event Stream and configure the following parameters:

    1. Enter a custom Task Name.

    2. For Metering Method, select Pay-by-event.

  4. On the Task Creation page, in the Source section, configure the following parameters:

    1. For Data Provider, select Message Service (MNS).

    2. For Queue Name, select the eventbridge-dev queue that you created. Keep the default settings for the other options.

  5. Click Next.

Filtering

  1. In the Filtering section, configure the following parameter:

    1. For Pattern Content, select All Events Match. For real-world use cases, you can also define flexible filtering rules by selecting other options or by creating a custom pattern.

  2. Click Next.

Transform

  1. In the Transformation section, configure the following parameters:

    1. For Alibaba Cloud Service, select Model/Agent Call (Alibaba Cloud Model Studio). This guide uses Model Call as an example.

    2. For Model Name, enter qwen-max.

    3. Configure the Model Context as shown in the following table:

      SYSTEM

      Fixed value

      You are a data compliance officer responsible for ensuring that all data entering the analytics system complies with GDPR/CCPA regulations and does not contain any personally identifiable information (PII). Your task is to automatically identify and mask names, phone numbers, and ID card numbers in text during the ETL process. You also need to block sensitive content related to religion, sex, and politics.

      USER

      Partial event

      $.data.messageBody

    4. For Structured Output, select Enabled. Click Add Field and add the fields shown in the following table:

      data

      string

      The masked data content.

      Required

      cause

      string

      The reason why the data was filtered, returned in English.

      Optional

    5. For API key, enter the Model Studio API key that you obtained.

  2. Click Next.

Sink

  1. In the Sink section, configure the following parameters:

    1. For Service type, select SMQ.

    2. For Queue Name, select the eventbridge-online queue that you created.

    3. For Message Body, select Partial Event and enter $.transform0.structuredoutput.

  2. Click Save. Wait for the event stream to be created, after which its Status changes to Running.

Verification

3

Send a message containing sensitive information from the source queue and check for the processed message in the sink queue to verify that the large model has correctly masked the data.

Send message

  1. Log on to the Message Service (MNS) console, and in the left-side navigation pane, click Queue List.

  2. Find the eventbridge-dev queue and click Send and Receive Messages in the Actions column.

  3. In the Send message section, configure the following parameters:

    1. For Message Content, enter Customer John Doe (13812345678) reported an issue.

    2. Set Message delay time to 0s.

  4. Click Send Message.

Receive message

  1. Log on to the Message Service (MNS) console, and in the left-side navigation pane, click Queue List.

  2. Find the eventbridge-online queue and click Send and Receive Messages in the Actions column.

  3. In the Receive message section, click Receive Message.

  4. The received message appears in the list. Click Details in the Actions column and view the Base64-decoded message content. A result similar to the one shown in the figure indicates that the solution has successfully processed the data.

Clean up resources

Click imageEnd Trial in the upper-right corner of the solution page, and then click OK in the confirmation dialog box to end the trial and release all resources.

Note

When the trial ends, the system deletes the trial account and automatically releases all resources.

Previous：Multi-source RAG: An efficient data pipelineNext：Tutorial