Export an OSS-HDFS metadata inventory-Object Storage Service(OSS)-阿里云帮助中心

You can export a metadata inventory for the HDFS service of a bucket. The inventory is exported as a JSON file to a specific path, which you can use for statistical analysis.

Prerequisites

Use the latest version of the Jindo software development kit (SDK). To download the SDK, see GitHub.
Configure the AccessKey pair for the bucket where the HDFS service is enabled.

For more information, see Access OSS and HDFS using Jindo CLI commands.

Procedure

Export the metadata inventory.
1. Log on to the OSS console.
2. In the left-side navigation pane, click Buckets. On the page that appears, click the name of the target bucket.
3. In the navigation pane on the left, choose Data Lake > OSS-HDFS.
4. In the Object Metadata section, click Export.
  
  The export can take from several minutes to several hours, depending on the volume of metadata. The output inventory file is saved to the oss://<hdfs_bucket>.<dls_endpoint>/.sysinfo/inventory/ path by default. Custom output paths are not supported.
  
  Important
  The .sysinfo/inventory/ directory cannot be deleted. You can access and delete the exported files in this directory.

Download the file.

Run the following command to download the inventory file.

jindo fs -get oss://<hdfs_bucket>.<dls_endpoint>/.sysinfo/inventory/1666584461201.2ce40fba-5704-45c4-8720-d92a891d**** /tmp/

Open the inventory file using the vi or vim command.

The following is a sample inventory file:

{"id":163**,"path":"/","type":"directory","size":0,"user":"admin","group":"supergroup","atime":1666581702921,"mtime":1666581702933,"permission":511}
{"id":624668410678950****,"path":"/dls-1000326249","type":"directory","size":0,"user":"hadoop","group":"supergroup","atime":1666581702923,"mtime":1660889124590,"permission":511}
{"id":624668410678950****,"path":"/dls-1000326249/benchmark","type":"directory","size":0,"user":"hadoop","group":"supergroup","atime":166658170293325,"mtime":1660889124590,"permission":511}
{"id":624668410678950****,"path":"/dls-1000326249/benchmark/n1","type":"directory","size":0,"user":"hadoop","group":"supergroup","atime":1666581702930,"mtime":1660889124590,"permission":511}
{"id":624668410678950****,"path":"/dls-1000326249/benchmark/n1/490747449","type":"directory","size":0,"user":"hadoop","group":"supergroup","atime":1666581702931,"mtime":1660895613953,"permission":511}

The metadata checklist result file contains the following fields:

Field	Description
id	The ID of the file or directory.
path	The path of the file or directory.
type	The type of metadata. directory: a directory. file: a file.
size	The data size in bytes. If the metadata type is file, the value of the size field is the file size. If the metadata type is directory, the value of the size field is 0.
user	The owner of the file or directory.
group	The user group of the file or directory.
atime	The access time of the file or directory. The value is a UNIX timestamp.
mtime	The modification time of the file or directory. The value is a UNIX timestamp.
permission	The permissions on the file or directory.

(Optional) Delete the inventory file.

The HDFS metadata inventory file consumes storage space and incurs storage fees. If you no longer need the inventory file, delete it promptly using a Hadoop command to avoid unnecessary fees.

Important
When you delete the inventory file, the file path must be the same as the Data Location in the output of Step 1. This prevents accidental deletion of system data in the .dlsdata and .sysinfo paths.

The following example shows how to run a Jindo command-line interface (CLI) command to delete an inventory file:
```
jindo fs -rm -skipTrash oss://<hdfs_bucket>.<dls_endpoint>/.sysinfo/inventory/1666584461201.2ce40fba-5704-45c4-8720-d92a891d****
```