Troubleshoot and manage node development issues

Choose a check method

• Syntax errors while you type — Real-time syntax diagnosis (LSP) flags them in real time.

• Built-in governance rules — A live scan applies efficient built-in rules to the active node when you save it.

• Full rule coverage — Trigger a deep check manually to apply all rules, including custom AI rules. You can scan a single file or a batch of open files.

Enable configuration items

Adjust issue diagnosis settings to fit your development workflow. All configuration items take effect only for the current Alibaba Cloud account.

DataStudio Governance Check Module Enablement controls the check capability of the Data Asset Governance plugin, and SyntaxErrorEnable controls real-time syntax diagnosis. The two options check code quality at different layers. Enable both to get full issue diagnosis coverage.

1. Go to the DataWorks workspace list page, select the target workspace, and choose Quick Start > DataStudio in the Actions column.

2. Click Setup at the bottom of the left-side navigation pane, and click the User tab on the Setup page.

   The following table describes the configuration items.

Manage the built-in rule library

Rule libraries drive the check behavior of the Data Asset Governance plugin and consist of built-in rules and custom rules. Enable or disable a specific rule as needed.

Rule library management takes effect only for the current Alibaba Cloud account.

1. In the left-side navigation pane of the DataStudio page, click the governance icon to open the Data Asset Governance plugin configuration.

2. In the Built-in Rule Library section of the Data Asset Governance plugin panel, find the target governance rule and click the toggle next to it to enable or disable the rule.

   • Enabled (default): The rule is checked when you save a node.

   • Disabled: The rule is no longer checked when you save a node.

Custom rule library

When built-in rules cannot cover the specific business logic or coding standards of your team, use the custom rule library of the Data Asset Governance plugin to define new check rules. You describe each rule in natural language and provide correct and incorrect examples.

Supported custom rule scenarios

Procedure

1. In the left-side navigation pane of the DataStudio page, click the governance icon to open the Data Asset Governance plugin configuration.

2. In the Custom rule library section of the Data Asset Governance plugin panel, click + to create a rule.

   To generate a governance rule with DataWorks Copilot, click AI Generation.

3. Configure the rule fields. This example creates the following custom rule:

   • Rule Name: Fact table updates must include WHERE.

   • Severity: The warning level prompts you during development only, and the error level triggers an alert and blocks the release before deployment. In this example, select Warning.

   • Scope: Select the node scope that the rule applies to, such as MaxCompute > MaxCompute SQL.

   • Effective scope: The user level, workspace level, and tenant level are supported. Only a tenant administrator can see the tenant level, only a workspace administrator can see the workspace level, and a regular member can see the user level option only.

   • Rule Description: Check UPDATE statements in MaxCompute SQL. If a statement updates a fact table whose name ends with _f but has no WHERE clause, flag it as a high-risk issue.

   • Valid example:

     • UPDATE my_project.order_detail_f SET status='shipped' WHERE order_id='123';

     • UPDATE my_project.order_detail_dim SET status='shipped';

   • Invalid example: UPDATE my_project.order_detail_f SET status='expired';

4. Click Save. The rule takes effect in deep check.

   To stop using a custom governance rule, hover over it and click the disable icon that appears.

deep check

A deep check is a comprehensive, time-consuming check that you trigger manually. It applies all governance rules, including custom AI rules. By contrast, the live scan that runs when you save a node applies only to efficient built-in rules. To check your code against the full rule set, run a deep check.

To scan a single file

To scan the file that is currently open in the editor, click deep check above the editor. The editor scans the file in depth and reports the issues that remain.

For example, in the sample code from Quick start, if you fix only the Partitioned table queries must include a partition issue, the scan reports the unresolved Creating tables in SQL is not allowed issue.

To scan multiple files in a batch

Note the following limits before you start a batch scan:

• A batch check covers only the file tabs that are currently open in the editor. Files that are not open cannot be selected.

• You can select up to 5 files.

• Only one deep check can run at a time during a batch check.

1. In the left-side navigation pane of the DataStudio page, click the governance icon to open the Data Asset Governance plugin configuration.

2. In the deep check section of the Data Asset Governance plugin panel, select the files and the rules for the batch check, and then start the deep check.

   After the deep check finishes, the deep check tab at the bottom displays the results. For example, the check of the file testassetrule returns two warnings: Creating tables in SQL is not allowed [my_partitioned_table] (rule ForbidUseCreateTable, line 7, column 28) and Partitioned table queries must include a partition (rule MissPartitionKeyFilter, line 17, column 18). Click a rule name to view its details.

In the deep check result pane, click View Log in the upper-right corner to review how the large language model reasoned about each check rule.

Partitioned table queries must include a partition

Risk: Querying a MaxCompute partitioned table without specifying a partition triggers a full table scan, which consumes large amounts of compute resources and generates high computing costs.

Incorrect code example:

SELECT user_id, order_amount
FROM user_orders
WHERE status = 'paid';

Correct code example:

SELECT user_id, order_amount
FROM user_orders
WHERE status = 'paid'
AND pt = '${bizdate}'; -- Add a partition filter

Automatic fix logic: Quick Fix is supported. The system adds a partition filter to the WHERE clause, such as AND pt = '${bizdate}'.

INSERT INTO with the rerun property is not allowed

Risk: If an SQL task contains only INSERT INTO logic and its scheduling configuration allows reruns, every rerun appends data to the target table. This easily causes duplicate data and compromises data accuracy.

Incorrect code example:

-- The task property is set to "rerunnable"
INSERT INTO target_table SELECT * FROM source_table;

Correct code example:

-- The task property is set to "rerunnable"
INSERT OVERWRITE TABLE target_table SELECT * FROM source_table;

Automatic fix logic: Quick Fix is supported. The system changes INSERT INTO to INSERT OVERWRITE so that a rerun overwrites data instead of appending it.

JOIN field types must match

Risk: In MaxCompute SQL, if the foreign key fields of a JOIN operation have inconsistent types, implicit type conversion occurs. This can cause computing errors and performance degradation, and it affects data quality.

Incorrect code example:

-- a.user_id is BIGINT, b.uid is STRING
SELECT * FROM table_a a JOIN table_b b ON a.user_id = b.uid;

Correct code example:

-- a.user_id is BIGINT, b.uid is STRING
SELECT * FROM table_a a JOIN table_b b ON a.user_id = CAST(b.uid AS BIGINT);

Automatic fix logic: Quick Fix is supported. The system detects the mismatch and applies the CAST function to one of the fields to convert its type explicitly so that it matches the other field.

INSERT into a table that does not belong to the current project

Risk: Writing data from a task in project A to a table in project B is a high-risk operation. It breaks the isolation between projects and can lead to unauthorized data access and data leaks.

Incorrect code example:

-- Run in a task of project_A
INSERT INTO project_B.some_table SELECT * FROM my_table;

Correct code example:

-- Recommended approach: let a task in the project that owns the table (project_B) write the data.
-- Avoid this operation in project_A.

Automatic fix logic: Automatic fix is not supported. Adjust the data synchronization link based on your business requirements so that the target project writes the data with its own task.

Online auto triggered tasks must not write to tables in the development environment

Risk: When an auto triggered task in the production environment writes data to a table in the development environment, the protection level of the data drops and data security risks arise.

Incorrect code example:

-- Run in a task of the production environment (PROD)
INSERT OVERWRITE TABLE user_dev.temp_data SELECT * FROM user_prod.source_data;

Correct code example:

-- A production task must write to a table in the production environment
INSERT OVERWRITE TABLE user_prod.result_data SELECT * FROM user_prod.source_data;

Automatic fix logic: Automatic fix is not supported. Modify the target table manually so that the data flow complies with the environment isolation standards.

Creating tables in SQL is not allowed

Risk: Using CREATE TABLE directly in a scheduled SQL task makes table ownership unclear, because the table usually belongs to the Alibaba Cloud account or the scheduling account. This increases management costs and creates a risk of accidental data deletion.

Incorrect code example:

CREATE TABLE my_temp_table (id INT);
INSERT INTO my_temp_table VALUES (1);

Correct code example:

-- Create the table in the DataWorks table management module first, and then use it directly in the SQL task.
INSERT INTO my_temp_table VALUES (1);

Automatic fix logic: Automatic fix is not supported. Create the table in DataWorks metadata management, or use CREATE TABLE IF NOT EXISTS to avoid errors in some scenarios.

Missing scheduling parameter checker

Risk: An auto triggered task processes data incrementally by time. If the WHERE condition omits a scheduling parameter such as ${bizdate}, the task might process full data every day or process an incorrect date range. This causes missing or incorrect data and wastes large amounts of resources.

Incorrect code example:

-- Incorrect: the scheduling parameter is missing, so daily incremental processing is impossible
INSERT OVERWRITE TABLE users_active_today PARTITION (pt = '${bizdate}')
SELECT    user_id  FROM    login_log; -- WHERE pt = '...' is missing

Correct code example:

-- Correct: use the scheduling parameter as a filter to process data incrementally every day
INSERT OVERWRITE TABLE users_active_today PARTITION (pt = '${bizdate}')
SELECT    user_id FROM    login_log
WHERE    pt = '${bizdate}';

Automatic fix logic: Automatic fix is partially supported. The plugin highlights where a scheduling parameter filter might be missing. Because business logic varies, add the correct filter manually based on your requirements.

Do these checks affect editor performance or the time it takes to save a task?

Real-time syntax diagnosis (LSP) has a slight but acceptable impact on editor performance. The Data Asset Governance plugin runs when you save a node, which adds a small amount of time to the save operation. The exact duration depends on the complexity of the code and the number of rules. If you notice obvious lag, check your network environment or contact technical support.

Is issue diagnosis billed separately, and is DataWorks Copilot required?

Issue diagnosis is a basic capability of DataWorks and is not charged separately. However, if you use DataWorks Copilot fix suggestions, the billing rules described in Data Agent Overview apply. You can also fix issues manually without Copilot.

I enabled the feature, but the issues in my code are not detected. Why?

Troubleshoot as follows:

• Check the main switches: Make sure that DataStudio Governance Check Module Enablement and SyntaxErrorEnable are enabled in Setup > User. For the navigation path and the valid values, see Enable configuration items.

• Check the rule switches: Go to the Built-in Rule Library and make sure that the rules you want to apply are enabled. For instructions, see Manage the built-in rule library.

• Confirm the node type: Check whether the current node type is within the scope of the target rule.

• Check the network: Check whether any network request failed in the developer tools of your browser. The LSP service or the governance service might have failed to load because of network problems.

What happens if the LSP service or the governance plugin service is unavailable?

If the backend service is temporarily unavailable, issue diagnosis fails silently. You do not see real-time syntax prompts or governance issues when you save, but code editing and saving still work as usual. Refresh the page and trigger the checks again.