DocsICP FilingConsole
官方文档
首页

Troubleshoot common issues

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic describes common issues and troubleshooting methods for the Apsara File Storage for HDFS file system SDK.

The java.lang.ClassNotFoundException message

If a hadoop command or a task fails and returns the java.lang.ClassNotFoundException: Class com.alibaba.dfs.DistributedFileSystem not found message, troubleshoot the issue as follows:

  1. Ensure that the Apsara File Storage for HDFS file system SDK (the aliyun-sdk-dfs-x.x.x.jar file) is in the CLASSPATH of all Hadoop environments.

    ${HADOOP_HOME}/share/hadoop/hdfs/

  2. Ensure that the user who runs the job has read permissions on the SDK.

  3. If the hadoop fs command runs successfully but analysis tasks fail, ensure that task execution frameworks such as YARN were restarted after the Apsara File Storage for HDFS file system SDK was installed. For example, you can run the following commands to restart YARN:

    ${HADOOP_HOME}/sbin/stop-yarn.sh
${HADOOP_HOME}/sbin/start-yarn.sh

The No FileSystem for scheme: dfs message

If a hadoop command or a task fails and returns the No FileSystem for scheme: dfs message, troubleshoot the issue as follows:

  1. Ensure that the following configurations for Apsara File Storage for HDFS are added to the core-site.xml file in all Hadoop environments.

    <configuration>
    <property>
         <name>fs.defaultFS</name>
         <value>dfs://x-xxxxxxxx.cn-xxxxx.dfs.aliyuncs.com:10290</value>
         <!-- Set this value to the mount target address of your Apsara File Storage for HDFS file system. -->
    </property>
    <property>
         <name>fs.dfs.impl</name>
         <value>com.alibaba.dfs.DistributedFileSystem</value>
    </property>
    <property>
         <name>fs.AbstractFileSystem.dfs.impl</name>
         <value>com.alibaba.dfs.DFS</value>
    </property>
</configuration>

  2. Ensure that the user who runs the job has read permissions on the core-site.xml file.

  3. If the hadoop fs command runs successfully but analysis tasks fail, you can load the Apsara File Storage for HDFS configurations directly in your code. For more information, see Application initialization.

The java.lang.NoClassDefFoundError message

If a hadoop command or a task fails and returns a java.lang.NoClassDefFoundError message, troubleshoot the issue as follows:

  1. Check the error log to determine whether the missing class is related to com.alibaba.dfs.

  2. If the missing class is related to com.alibaba.dfs, update the SDK to the latest version.

  3. If the class is not related to com.alibaba.dfs, a dependent class is missing from the Hadoop runtime environment. Check the error log, download the required JAR package, and place it in the CLASSPATH of the Hadoop runtime environment.

    For example, if the following error occurs when you run a hadoop command or a task, it indicates that the org/apache/hadoop/fs/PathFilter class is missing from the Hadoop runtime environment. This class is included in the hadoop-common-x.x.x.jar package. Download the correct version of this JAR package and place it in the CLASSPATH of all Hadoop environments.

    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/PathFilter
Previous：SDK examplesNext：Best Practices