Dataphin代码任务报错："Fatal error occurred when node tried to create too many dynamic partitions"

问题描述

Dataphin代码任务运行报错："Fatal error occurred when node tried to create too many dynamic partitions"。

2021-11-11 11:13:12,999 INFO [IPC Server handler 24 on 64040] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1636027603751_126799_r_000113_2: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

问题原因

在每个执行MR的节点上，默认最大可以创建100个动态分区。

解决方案

添加参数进行修改默认配置。

SET hive.exec.max.dynamic.partitions.pernode=10000;

适用于

• Dataphin