Use TensorFlow with Apsara File Storage for HDFS
This topic describes how to use TensorFlow with Apsara File Storage for HDFS.
Prerequisites
You have activated the Apsara File Storage for HDFS service. For more information, see Activate Apsara File Storage for HDFS.
You have created a file system, added a mount target, and mounted the file system. For more information, see Quick Start.
Java Development Kit (JDK) 1.8 or later is installed on all nodes in the Hadoop cluster.
You have downloaded the source code for your version of TensorFlow.
This topic uses TensorFlow 1.15.0.
Background information
TensorFlow is an open source software library for numerical computation that uses data flow graphs.
TensorFlow requires a specific file system schema. To use TensorFlow with Apsara File Storage for HDFS, you must modify the TensorFlow source code. For more information, see Configure TensorFlow to support Apsara File Storage for HDFS.
Configure TensorFlow to support Apsara File Storage for HDFS
Add support for Apsara File Storage for HDFS to the TensorFlow source code. For more information, see tensorflow support aliyun HDFS.
Compile TensorFlow.
Compile the TensorFlow source code that you modified in Step 1. For more information, see Build from source.
Verify the configuration.
Create a test file on Apsara File Storage for HDFS.
Create a folder, such as test_tensorflow.
hadoop fs -mkdir dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflowCreate a file, such as words.
hadoop fs -touchz dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow/wordsWrite data to the words file.
echo "hello tensorflow" | hadoop fs -appendToFile - dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow/words
Replace
f-xxxx.cn-xxxx.dfs.aliyuncs.comwith the domain name of your Apsara File Storage for HDFS mount target.Use the TensorFlow gfile API to view the data on Apsara File Storage for HDFS.
Enter the Python interactive shell.
pythonImport the TensorFlow module.
import tensorflow as tfList the contents of the test_tensorflow folder.
tf.io.gfile.listdir('dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow')View the contents of the words file.
with tf.gfile.Open('dfs://f-xxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_tensorflow/words') as rf: rf.read()If the output is the same as the test data that you wrote, the configuration is successful.