Defining and using data standards

Field standards

A field standard normalizes a field's name, data type, and value range. Standardizing fields with the same meaning prevents confusion from naming or type variations. For example, a field standard named member_id ensures that all member ID fields across tables share a consistent definition.

Lookup tables

A lookup table defines the valid value range and allowed content for a field. For example, the values for a gender field can be restricted to Male, Female, and Unknown.

Measurement units

A measurement unit defines business-relevant units of measure, such as currency, object quantifiers, and time units. For example, the measurement unit for product quantity is pieces.

Naming dictionaries

A naming dictionary provides standardized translations for business terms, physical tables, and fields, including root words and morphemes. It serves as an enterprise-level library for naming conventions. For example, a company's annual revenue can be standardized as Annual Gross Revenue.

Hierarchy

• A field standard must be placed under the root directory, a directory, or a standard set. The hierarchy is as follows:

  • Root directory: The top-level directory. All other directories, standard sets, and standards must be placed in the root directory.

  • Directory: A folder used to store standards and standard sets, similar to a folder in an operating system.

  • Standard set: Similar to a directory, but can only contain standards.

• Field standards can have inheritance relationships. For example, both buyer ID standards and seller ID standards can inherit from a member ID standard.

Define a field standard

1. In the left navigation bar of the Data Standard page, click Field Standard to go to the Field Standard page.

2. In the directory tree on the left, right-click the target directory or standard set and select Create Standard.

   You can create directories or standard sets to organize your field standards as needed.

3. In the Create Standard dialog box, the key parameters are as follows.

   Parameter	Description
   Abbreviation	The name of the associated field in a logical model.
   Display Name	The display name of the associated field in a logical model.
   Length	A parameter related to the data type. For example, if you select the DECIMAL data type, the length corresponds to 20 in DECIMAL(20,4).
   Precision	A parameter related to the data type. For example, if you select the DECIMAL data type, the precision corresponds to 4 in DECIMAL(20,4).
   Not Null	Specifies whether a field referencing this standard can be null. Null values are allowed by default.
   Default Value	The default value for the field if no value is provided. The value cannot exceed 2,048 characters in length.
   Parent Standard	Select an existing standard as a parent to establish an inheritance relationship and better identify field associations. For example, both buyer ID and seller ID are types of member ID. Therefore, the parent standard for both the buyer ID standard and seller ID standard is the member ID standard.
   Referenced Lookup Table	Reference an existing lookup table to constrain the value range of a field. For more information, see Data standards: Lookup tables.

   Important

   Before you can delete a field standard, you must remove all of its references.

Use a field standard

Field standards can define specific fields in logical models, including source tables, dimension tables, fact tables, aggregate tables, and application tables. For example, in the dimension table for member information dim_ec_con_member_df, you can associate the member ID field with the field standard member_id. The field name is inherited from the abbreviation of the field standard, the display name from the display name, and properties such as Type and Not Empty are also inherited. For more information, see Configure dimension table fields.

Define a lookup table

1. In the left navigation pane on the Data Standard page, click Lookup Table to go to the Lookup Table page.

2. Right-click the created directory name, and click Create Lookup Table.

   You can create directories to organize your lookup tables as needed.

3. In the Create Lookup Table dialog box, configure the following parameters and add enumerated values.

   For example, set the ID to gender, the display name to Gender, and the name to gender. The enumerated values are shown in the following table.

   Code value	Code name	Name	Description
   0	Unknown	unknown	Gender not specified
   1	Male	male	Male
   2	Female	female	Female

   Important

   Before you can delete a lookup table, you must remove all of its references.

Publish a lookup table

On the lookup table details page, click Publish in the upper-right corner to materialize the lookup table as a physical table or materialized view.

Use a lookup table

Lookup tables can define specific fields in logical models, including source tables, dimension tables, and fact tables. For example, in the dimension table for member information dim_ec_con_member_df, you can associate the gender field with the lookup table gender. The field name is inherited from the ID of the lookup table, and the display name from the display name of the lookup table. For more information, see Configure dimension table fields.

Associating a lookup table automatically unifies field names across multiple tables.

Apply standards based on lookup tables

When a logical model fact table or dimension table is published as a physical table, you can generate a quality rule for a field associated with a lookup table. This rule can be used to create a quality monitoring task that enforces the standard on the physical table. For more information, see Data labeling.

An example of a generated quality rule is as follows: The rule name is Field Rule-Lookup Table-(gender), the quality dimension is Consistency, the scope is Field-level(gender), and the rule template is Number of rows with mismatched enumerated values is 0. For the monitoring threshold, !=0 indicates an exception and =0 indicates normal. The severity is Weak Rule.

Define a measurement unit

1. On the Data Standard page, in the left navigation bar, click Measurement Unit to enter the Measurement Unit page.

2. On the Measurement Unit page, right-click the target measurement category, and select Create Measurement Unit.

   The system provides you with Currency Unit, Object Quantifier, Ratio Unit, Ranking Unit, and Time Unit measure categories.

3. In the Create Measurement Unit dialog box that appears, configure the parameters to complete the setup.

   For example, set the abbreviation to m, the name to meter, and the display name to Meter. For the category, select Object Quantifier.

Define a naming dictionary

1. On the Data Standard page, in the left navigation bar, click Naming Dictionary to go to the Naming Dictionary page.

2. Click Create. In the Create Naming Dictionary dialog box, configure the parameters and click OK.

   For example, set the display name to Engine, the name to engine, and the abbreviation to eng..

Use a naming dictionary

A naming dictionary validates whether table names comply with naming conventions for data warehouse layers. This feature is supported for source tables, dimension tables, fact tables, aggregate tables, and application tables. For example, if no naming dictionary entry with the abbreviation trade exists, the table name dwd_trade_order does not comply with the DWD layer naming convention.

On the Checker Management page, click Add Rule. Select Table Name as the rule type. You can then define the rule by using the fixed text dwd and selections from the Root Word Abbreviation and Business Process Abbreviation dropdowns to match your naming convention.

To use this feature, you can configure it in the data warehouse layer checker. For more information, see Configure data warehouse layer checkers and Use checkers.

Batch import standards

For a large number of data standards, you can import them in batches. Download the import template provided by DataWorks, fill it in, and upload it.

1. On the Data Standard page, in the left navigation bar, click Naming Dictionary to go to the Naming Dictionary page.

   The details pages for field standards and lookup tables also provide Import and Export buttons.

2. Click Import, and on the Import page, select the Import Type.

3. In the Template Preview section, click Download Template and fill in the required fields.

4. Click Next Step. On the Data Import tab, upload and preview the data file.

   Note

   • Import Mode: If an object with the same name as the imported file already exists in DataWorks, you can choose to skip the object or overwrite it with the content of the imported file.

   • Only files in the .xlsx format are supported for batch import. You can import up to 30,000 records at a time. The file size cannot exceed 10 MB.

5. On the Complete tab, you can view the import result details. Click View More Details in the details list to go to the edit page, where you can perform additional operations. If the import status is Failed, you need to resolve the error based on the error details and re-import the data.

Batch export standards

To reuse data standards across workspaces, export them in batches from the details pages for field standards, lookup tables, or naming dictionaries.