Add a Cloud Service Data Source

Billing Information

Synchronizing data from a cloud service data source to IoT Platform incurs data processing unit (DPU) costs.

Limits

• Currently, only files in an OSS bucket can be synchronized to DataService Studio on IoT Platform.

• Each file in an OSS bucket must not exceed 1 GB. Daily files must be organized in folders named by date.

  Supported date formats:

  • Day: YYYYMMDD, for example: 20230302.

  • Hour: YYYYMMDDHH, for example 2023030213.

Create a Cloud Service Data Source

1. In the IoT Platform console, go to the Overview page and click the ID or alias of the target Enterprise Edition instance.

2. In the left navigation bar, select DataService Studio > Data Ingestion, and click the Cloud Service tab.

3. On the Cloud Service tab, click Create Cloud Service Data Source.

4. In the right panel, follow these steps to complete the cloud service data source configuration.

Basic Information

1. On the Basic Information wizard page, configure the following parameters.

   Parameter	Description
   Name	The name of the cloud service data source. The name can contain Chinese characters, letters, digits, periods (.), underscores (_), and parentheses (). Maximum length: 32 characters.
   Description	The description of the cloud service data source.
   Cloud Service Type	Currently, only OSS is supported.
   Bucket	Select an OSS bucket. For instructions on creating a bucket in OSS, see Create a bucket. Important Before you configure the bucket, click Create next to the role-not-created prompt to create a service-linked role. This role authorizes DataService Studio to access and synchronize data from your OSS bucket. For more information about service-linked roles, see Service-linked role for cloud product data ingestion.
   Parent Directory	The parent directory of the file in the bucket. For more information about managing file directories, see Manage directories.

2. Click Next.

Field Mapping

1. On the Field Mapping wizard page, configure the data format for the data source.

   Parameter	Description
   Data Format	Select one of the following options: JSON: JSON format. ProtoBuf: ProtoBuf format. CSV: CSV format. Customize: A custom script to parse the data format.
   Row Delimiter	Select one of the following options:System Default Delimiter, Comma, Vertical Line, Space, Tab Character, Customize. If you select System Default Delimiter, the system delimiter is used automatically. If you select Customize, enter a visible character. Note When Data Format is JSON, Row Delimiter cannot be Comma. For Data Format set to ProtoBuf, a protocol-specific line break is used. When writing data to a file, use writeDelimitedTo (Java) or delimited_message_util.h (C++). Custom row delimiters are not supported. When Data Format is CSV, Row Delimiter defaults to System Default Delimiter and cannot be changed.
   Column Delimiter	This parameter is available only when Data Format is set to CSV. Options: Comma, Vertical Line, Space, Tab Character, or Customize.

2. Click Configure in the Field Mapping dialog box to configure the field mapping for the data source.

   1. Configure sample data for format parsing based on the selected data format.

      Data Format	Operation
      JSON	In the Sample Data box, enter sample data in JSON format. Only single-line data entries are supported. Important Array format data entries are not supported.
      ProtoBuf	Upload a .desc file by clicking Upload .desc File to parse the ProtoBuf format data. For instructions on generating a .desc file, see Appendix: Generating .desc Files. Select the message type and upload sample data by clicking Upload Binary Data File. Important Array format data entries are not supported.
      CSV	Enter sample data directly in the Sample Data box. Sample data must include column names and contain only one line of data. For example, with Column Delimiter set to Comma: productKey,deviceName,time,temperature,humidity g9rp0*****,device1,20230102,23,30
      Customize	Select a scripting language and enter the script in the Edit Script box. The supported scripting language is Javascript (ecmascript 5), with the function name defined as parse_line. For script templates and examples, see Write custom scripts. On the Analog Input tab, enter analog device reporting data based on the row delimiter.

   2. Validate data parsing:

      • If the Data Format is JSON, ProtoBuf, or CSV, click Validate Parsing:

        • If parsing succeeds, the parsed field information is displayed under Data Storage Type.

        • If parsing fails, adjust the sample data as prompted and try again.

      • If the Data Format is Customize, click Execute:

        • If execution succeeds, the parsed data appears on the Execution Result tab.

        • If execution fails, revise the script based on the feedback and try again.

   3. In the parsed field list, assign the Field Display Name and designate a Primary Key.

      The Field Display Name can contain Chinese characters, letters, digits, underscores (_), hyphens (-), and parentheses (). Maximum length: 30 characters.

   4. Click Confirm.

3. Click Next.

Synchronization Cycle

1. Select the frequency for the Synchronization Cycle: Hour or Day.

2. Choose the Earliest Synchronization Time.

3. Click Finish.

After you create a cloud service data source:

• On the Data Ingestion > Cloud Service tab, the new cloud service data source synchronization task is listed. The Synchronization Status is Stopped by default. You must manually enable synchronization. For more information, see Enable Cloud Service Data Source Synchronization.

• On the Data Storage > Cold Data Storage > Custom Storage Table tab, a storage table is created with the cloud service data source name as the table name.

Enable Cloud Service Data Source Synchronization

To enable data synchronization and allow cloud service data to be ingested into DataService Studio:

1. Go to the Cloud Service tab and find the cloud service data source synchronization task that you want to enable.

2. Toggle the Synchronization Status switch in the corresponding Stopped column

After synchronization is enabled, the Synchronization Status of the task changes to Synchronizing. The system synchronizes cloud service data to the custom storage table in DataService Studio based on the configured cycle.

Optional Operations

Edit Cloud Service Data Source Configuration

1. In the Cloud Service tab, find the cloud service data source that you want to edit.

2. Click the Edit button in the corresponding Operation column.

3. In the Edit Cloud Service Data Source panel, modify the configurations as needed.

Delete Cloud Service Data Source

You can delete a cloud service data source only when its synchronization status is stopped.

1. Go to the Cloud Service tab and find the cloud service data source that you want to delete.

2. Click the Delete button in the corresponding Operation column.

3. Click Confirm to confirm the deletion.

What to Do Next

To manage synchronization tasks and data storage tables, see Manage data synchronization tasks and storage tables.