空值规则

配置示例

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt=$[yyyymmdd]/hh=$[hh24-1/24]
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - missing_count(birthday) = 0
  - missing_percent(gender) < 5%
  - missing_count(first_name) = 0
    missing:
      regex: "(?:N/A)"
  - missing_count(first_name) = 0
    missing:
      values:
        - n/a
        - NA
        - none
  - missing_percent(email_address) = 0%
computeResource:
  id: 2001

定义空值规则

执行空值规则校验时，系统会扫描数据中的指定字段，统计出指定字段值为空的数据行数或者数据行数占比。

• 系统会将NULL值当作空值统计。

• 可以通过missing_percent来校验空值行数的占比。

示例一：校验birthday字段为NULL的数据。

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt=$[yyyymmdd]/hh=$[hh24-1/24]
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - missing_count(birthday) = 0
computeResource:
  id: 2001

示例二：定义number_employees字段为NULL的行数不能超过监控总行数的5%。

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt=$[yyyymmdd]/hh=$[hh24-1/24]
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - missing_percent(number_employees) < 5%
computeResource:
  id: 2001

保留问题数据

如果规则的collectFailedRows设置为true，则在校验不通过（状态为warn或者fail）时，会将指定字段为NULL的数据自动保存到问题数据表中供后续问题排查使用。

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt=$[yyyymmdd]/hh=$[hh24-1/24]
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - missing_percent(number_employees) < 5%
    collectFailedRows: true
computeResource:
  id: 2001