DocsICP FilingConsole
官方文档
首页

Obtain data from an ApsaraDB RDS for MySQL database for data enrichment

更新时间：
复制 MD 格式
产品详情
我的收藏

Simple Log Service data transformation can pull data from ApsaraDB RDS for MySQL databases and enrich log data based on transformation rules.

Background information

Data is often distributed across different sources. For example, user operation and behavior data might be stored in Simple Log Service, while user profiles and registration information are stored in an ApsaraDB RDS for MySQL instance. You can use Simple Log Service data transformation to fetch data from the ApsaraDB RDS for MySQL instance and store the enriched data in a Logstore.

Use the res_rds_mysql function to obtain data from an ApsaraDB RDS for MySQL database, and then use the e_table_map function to enrich the data.

Note

Use the e_table_map function to enrich data

The following example uses the e_table_map and res_rds_mysql functions to enrich data.

  • Raw data

    • Sample data records in a table of an ApsaraDB RDS for MySQL database

      province

      city

      population

      cid

      eid

      Shanghai

      Shanghai

      2000

      1

      00001

      Tianjin

      Tianjin

      800

      1

      00002

      Beijing

      Beijing

      4000

      1

      00003

      Henan

      Zhengzhou

      3000

      2

      00004

      Jiangsu

      Nanjing

      1500

      2

      00005

    • Sample logs from a Logstore in Simple Log Service:

      time:"1566379109"
data:"test-one"
cid:"1"
eid:"00001"

time:"1566379111"
data:"test_second"
cid:"1"
eid:"12345"

time:"1566379111"
data:"test_three"
cid:"2"
eid:"12345"

time:"1566379113"
data:"test_four"
cid:"2"
eid:"12345"

  • Transformation rule

    The cid field in the Logstore is matched against the cid field in the ApsaraDB RDS for MySQL table. A match succeeds only when the cid values are identical. After a successful match, the province, city, and population fields are retrieved from the database table and joined with the Logstore data.

    Note

    • If multiple rows match in the table, the e_table_map function returns only the first matching record. In this example, the cid field in the table has multiple values of 1.

    e_table_map(res_rds_mysql(address="rds-host", username="mysql-username",password="xxx",database="xxx",table="xx",refresh_interval=60),"cid",["province","city","population"])

    For more information about how to configure an ApsaraDB RDS for MySQL database in the res_rds_mysql function, see res_rds_mysql.

  • Transformation results 

    time:"1566379109"
data:"test-one"
cid:"1"
eid:"00001"
province:"Shanghai"
city:"Shanghai"
population:"2000"

time:"1566379111"
data:"test_second"
cid:"1"
eid:"12345"
province:"Shanghai"
city:"Shanghai"
population:"2000"

time:"1566379111"
data:"test_three"
cid:"2"
eid:"12345"
province:"Henan"
city:"Zhengzhou"
population:"3000"

time:"1566379113"
data:"test_four"
cid:"2"
eid:"12345"
province:"Henan"
city:"Zhengzhou"
population:"3000"
Previous：Pull a CSV file from OSS to enrich dataNext：Enrich data from ApsaraDB RDS for MySQL