Filter objects based on object metadata-Object Storage Service(OSS)-阿里云帮助中心

Scenarios

Directly browsing or processing objects in a bucket that contains a large number of objects is time-consuming. For this reason, ossutil 2.0 provides a series of efficient metadata-based filtering options, which allows you to accurately filter objects based on specific metadata.

Prerequisites

Object Storage Service (OSS) is activated.
A bucket is created.
ossutil 2.0 is installed.

Metadata-based object filtering

ossutil 2.0 allows you to filter objects based on object metadata in a fine-grained manner. In most cases, this feature is suitable for multiple advanced commands, such as ls which is used to list objects, cp which is used to upload, download, and copy objects, and rm which is used to delete objects. You can filter objects based on object metadata, such as object type, to improve filtering efficiency.

Parameter description

Parameter	Type	Description
--metadata-include	stringArray	The metadata filtering rules that you want to use to include objects. This way, only objects that meet the metadata rules are displayed.
--metadata-exclude	stringArray	The metadata filtering rules that you want to use to exclude objects. This way, objects that meet the metadata rules are not displayed.
--metadata-filter	stringArray	The metadata filtering rules that are specified directly to support more flexible matching logic.
--metadata-filter-from	stringArray	Specifies that the metadata filtering rules are read from an external rule file to facilitate the management and reuse of the difficult metadata rules.

When you use the --metadata-include parameter, a rule named - ** which excludes other types of objects is automatically implicitly added to the end of the list.
When you use the --metadata-filter "+ rule" parameter, all exclusion rules are not automatically added. That is, --metadata-include "rule" is equal to --metadata-filter "+ rule" + --metadata-filter "- **". For example, to accurately filter out all JPEG image objects in a bucket, you must specify --metadata-filter "+ content-type=image/jpeg" --metadata-filter "- **".
When you process multiple external rule files in which metadata filtering rules are stored at a time, the external files are executed from left to right, whereas the rules within each file are executed from top to bottom, and blank lines and comment lines that start with# or; are ignored.
After the first object metadata rule is matched, subsequent rule checks are stopped.

Supported metadata types

Storage class. To filter objects by storage class, specify x-oss-storage-class=value, in which the value can be Standard, IA, Archive, ColdArchive, or DeepColdArchive.
Object type. To filter objects by object type, specify x-oss-object-type=value, in which the value can be Normal, Multipart, Appendable, or Symlink.
Restoration status. To filter objects by restoration status, specify x-oss-restore=value, in which the value is the actual restoration status that you want to filter by.
Content type. To filter objects by content type, specify content-type=value, in which the value is the actual content type that you want to filter by.
User metadata. To filter objects by user metadata, specify x-oss-meta-aaa=value, in which x-oss-meta- is a fixed prefix, indicating the beginning of the user metadata, aaa indicates the custom key name, and value indicates the value of the user metadata specified based on your business requirements. Example: x-oss-meta-location:hangzhou.

Examples

Filter objects of a specific storage class

The following example describes how to filter objects whose storage class is Standard in the examplebucket bucket:
```
ossutil ls oss://examplebucket --metadata-include "x-oss-storage-class=Standard"
```
The following example describes how to filter objects whose storage class is Archive, Cold Archive, and Deep Cold Archive in the examplebucket bucket:
```
ossutil ls oss://examplebucket --metadata-include "x-oss-storage-class=*Archive"
```

Filter objects in the restored state or the restoring state

Parameter

Type

Example

Description

x-oss-restore

String

Restoring: going going-request="true"
Restored: ongoing-request="false", expiry-date="Thu, 04 Jan 2024 03:28:54 GMT"

You can call the HeadObject operation and check whether the objects are restored based on the x-oss-restore response header.

If no restoration operation is performed or the restoration operation times out, the x-oss-restore response header is not returned.
If a restoration operation is in progress, ongoing-request="true" is displayed to the right of the x-oss-restore response header.

If a restoration operation is complete, ongoing-request="false" is displayed to the right of the x-oss-restore response header. In this case, expiry-date is included in the x-oss-restore response header to indicate the time after which the object cannot be read.

The following example describes how to filter objects in the restoring state in the examplebucket bucket. The two asterisks (**) indicate that the objects can be read at any time.
```
ossutil ls oss://examplebucket --metadata-include "x-oss-restore=ongoing-request=\"true\"**"
```
The following example describes how to filter objects in the restored state in the examplebucket bucket. The two asterisks (**) indicate that the objects can be read at any time.
```
ossutil ls oss://examplebucket --metadata-include "x-oss-restore=ongoing-request=\"false\"**"
```

Filter objects of a specific format by specifying inclusion and exclusion rules

The following example describes how to filter all JPEG image objects in the examplebucket bucket for specific operations or data analysis:
```
ossutil ls oss://examplebucket --metadata-include "content-type=image/jpeg"
```
The following command lists all plain text objects in the examplebucket bucket by matching the content-type metadata attribute:
```
ossutil ls oss://examplebucket --metadata-include "content-type=text/plain"
```
Note
To filter objects by object name using wildcard patterns, use --include instead of --metadata-include. For example: ossutil ls oss://examplebucket --include "*.txt".

Filter objects of a specific format by specifying metadata filtering rules directly

The following example describes how to filter all JPEG image objects in the examplebucket bucket. You can specify the --metadata-filter parameter to specify metadata filtering rules directly and use the plus sign (+) and the minus sign (-) to specify the inclusion and exclusion rules. All objects whose Content-Type is image/jpeg in the examplebucket bucket are listed, and objects whose Content-Type is not image/jpeg are excluded.

ossutil ls oss://examplebucket --metadata-filter "+ content-type=image/jpeg" --metadata-filter "- **"

Filter objects by specifying metadata filtering rules in an external rule file

If the metadata filtering rules are complex or multiple different metadata filtering rules need to be applied at a time, you can create an external rule file in advance to filter objects.

To filter all JPEG image objects whose storage class is Standard by using the external rule file named metadata_filters.txt, add the following sample code to the external rule file:

# rules file: metadata_filters.txt
+ content-type=image/jpeg
+ x-oss-storage-class=Standard
- **

Use the --metadata-filter-from option to call the external rule file to filter objects:

ossutil ls oss://examplebucket --metadata-filter-from metadata_filters.txt