Use TensorFlow with Apsara File Storage for HDFS

Updated at:

This topic describes how to use TensorFlow with Apsara File Storage for HDFS.

Prerequisites

  • You have activated the Apsara File Storage for HDFS service. For more information, see Activate Apsara File Storage for HDFS.

  • You have created a file system, added a mount target, and mounted the file system. For more information, see Quick Start.

  • Java Development Kit (JDK) 1.8 or later is installed on all nodes in the Hadoop cluster.

  • You have downloaded the source code for your version of TensorFlow.

    This topic uses TensorFlow 1.15.0.

Background information

TensorFlow is an open source software library for numerical computation that uses data flow graphs.

Note

TensorFlow requires a specific file system schema. To use TensorFlow with Apsara File Storage for HDFS, you must modify the TensorFlow source code. For more information, see Configure TensorFlow to support Apsara File Storage for HDFS.

Configure TensorFlow to support Apsara File Storage for HDFS

  1. Add support for Apsara File Storage for HDFS to the TensorFlow source code. For more information, see tensorflow support aliyun HDFS.

  2. Compile TensorFlow.

    Compile the TensorFlow source code that you modified in Step 1. For more information, see Build from source.

  3. Verify the configuration.

    1. Create a test file on Apsara File Storage for HDFS.

      1. Create a folder, such as test_tensorflow.

        hadoop fs -mkdir  dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow
      2. Create a file, such as words.

        hadoop fs -touchz  dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow/words
      3. Write data to the words file.

        echo "hello tensorflow" | hadoop fs -appendToFile - dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow/words

      Replace f-xxxx.cn-xxxx.dfs.aliyuncs.com with the domain name of your Apsara File Storage for HDFS mount target.

    2. Use the TensorFlow gfile API to view the data on Apsara File Storage for HDFS.

      1. Enter the Python interactive shell.

        python
      2. Import the TensorFlow module.

        import tensorflow as tf
      3. List the contents of the test_tensorflow folder.

        tf.io.gfile.listdir('dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow')
      4. View the contents of the words file.

        with tf.gfile.Open('dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow/words') as rf:
            rf.read()

        If the output is the same as the test data that you wrote, the configuration is successful.