ODPS-0010000: System internal error

Error message

ODPS-0010000:System internal error - fuxi job failed, caused by:
kWorkerOutOfMemory(errCode:256) at Odps/<job_id>/<stage_task>.
Detail error msg: KILL_NAKILL_NA:
plannedResource({Memory=<planned_mem>, CPU=<planned_cpu>})
usedResource({Memory=<used_mem>, CPU=<used_cpu>})

Where:

• <job_id>: The MaxCompute job identifier.

• <stage_task>: The stage and task identifier (for example, R2_1 indicates reducer stage 2, task 1).

• <planned_mem> and <planned_cpu>: The resources planned for the task.

• <used_mem> and <used_cpu>: The resources consumed at runtime.

Cause

The job consumed more resources than planned, which caused an OOM error.

Solution

1. Identify the planned and used resource amounts from plannedResource and usedResource in the error message.

2. Determine the task type from the error message:

   • M: mapper

   • R: reducer

   • J: joiner

3. Use SET operations to adjust parameters based on the task type. For example, for a mapper task: Set the memory value higher than <used_mem> reported in the error message.

      SET odps.stage.mapper.mem=<value>;
      SET odps.stage.mapper.split.size=<value>;

Hologres direct-read mode optimization

If the OOM error occurs while a MaxCompute job reads Hologres data through an external table in direct-read mode, increasing mapper memory alone may not resolve the issue. Configure the following Hologres-specific parameters in addition to the general memory settings:

Example:

SET odps.stage.mapper.mem=12288;
SET odps.sql.mapper.split.size=512;
SET hg_foreign_table_executor_max_dop=32;
SET hg_experimental_query_batch_size=4096;
SET odps.external.holo.mapper.instances=50000;
SET odps.external.holo.read.table.rules='{get_snapshot_by_shard:10}';

Error message

ODPS-0010000:System internal error - fuxi job failed, caused by:
kWorkerOutOfMemory(errCode:256) at
Odps/autonavi_location_20230831212329151gio81xoniqh_SQL_0_1_0_job_0/R2_1@68d01417.na62#4533.
Detail error msg: KILL_NAKILL_NA:
plannedResource({Memory=4096, CPU=100})
usedResource({Memory=5751, CPU=164}).

Cause

The segment SQL_0_1_0_job_0/R2_1 indicates the OOM error occurred in the R2_1 task of the reducer stage.

• plannedResource({Memory=4096, CPU=100}): Resources planned for this task.

• usedResource({Memory=5751, CPU=164}): Resources the task consumed at runtime.

The task required more memory and CPU than planned.

Solution

Set the memory and CPU for the stage where the OOM error occurred. In this example, configure the reducer stage:

SET odps.sql.reducer.memory=6144;  -- Must be greater than the Memory value in usedResource
SET odps.sql.reducer.cpu=200;      -- Must be greater than the CPU value in usedResource

Error message

ODPS-0010000:System internal error - fuxi job failed, caused by:
alloc too large size: <size>

Cause

The job requested too much memory at runtime. Possible causes:

• A bug in MaxCompute.

• Abnormal user data, such as an ARRAY type field with too many elements.

Solution

1. Check for oversized ARRAY data and reduce its volume.

2. If the issue persists, submit a ticket for technical support.

Error message

ODPS-0010000:System internal error - failed to get error message of java
exception because of outofmemoryerror

Cause

An OOM error caused the program to fail.

Solution

Increase memory by setting the following parameters:

SET odps.sql.mapper.cpu=400;
SET odps.sql.mapper.memory=8192;

This example increases the mapper memory. If the OOM error occurs in a different stage, adjust the corresponding stage parameters instead. See Error 1: kWorkerOutOfMemory (basic) for how to identify the failing stage.

Error message

ODPS-0010000:System internal error - Shortcut optimize failed because the
script is too complex

Cause

The SQL statement is too complex, which causes the shortcut optimization to fail.

Solution

Optimize the SQL statement:

• Split and simplify complex SQL statements.

• Check for:

  • Deeply nested subqueries

  • Excessive JOIN operations

  • Complex aggregate or window functions

• Reduce data processing volume:

  • Add WHERE clauses to limit query scope.

  • Query only the required partitions of partitioned tables.

Error messages

Error 9:

ODPS-0010000:System internal error - fuxi job failed, caused by:
codeGenCallBuiltinFunction fail in CopmareBase for function: NEQ, <details>

Error 10:

ODPS-0010000:System internal error - fuxi job failed, caused by:
process core dumped

Cause

Both errors occur when a NEQ (not equal) operation is performed on a field or expression of the ARRAY type. MaxCompute does not support this operation. For example, this can happen when the COLLECT_SET function is used.

Solution

Use a function that supports comparison operations on ARRAY type fields. For example, use SIZE(ARRAY)!=0 instead of a direct NEQ comparison. For more information, see SIZE.

Alternatively, upgrade MaxCompute to the latest version.

Error message

ODPS-0010000:System internal error - fuxi job failed, caused by:
Regex compile error

Cause

The regular expression in the SQL statement is invalid.

Solution

Fix the regular expression syntax in the SQL statement.

Cause

Object Storage Service (OSS) experienced temporary service instability. Services were unavailable for a short period.

Solution

Rerun the job.

Cause

The user does not have the required permissions to access the OSS file.

Solution

Grant the user who accesses the external table the permissions on the corresponding OSS external table.

Error message

ODPS-0010000:System internal error - fuxi job failed, caused by:
Merge into statement not support update or delete duplicate row, please check
your query, the query text location is from [line <m>, column <n>] to
[line <i>, column <j>]

Where:

• <m>, <n>: The start position (line and column) of the problematic query text.

• <i>, <j>: The end position (line and column) of the problematic query text.

Cause

Duplicate values exist on both sides of the equal sign (=) from line <m>, column <n> to line <i>, column <j>. This causes duplicate rows due to data bloat. The MERGE INTO statement cannot update or delete duplicate rows, so the operation fails.

Solution

1. Check for duplicate values in the source data from line <m>, column <n> to line <i>, column <j>.

2. Remove duplicates before running the statement again.