Manage the directory quota feature

更新时间:
复制 MD 格式

Directory quotas let you cap the storage size and file count of OSS-HDFS directories, preventing runaway growth and unexpected charges.

OSS-HDFS supports two quota types:

  • Capacity quota: the maximum total storage a directory can use (including all objects and subdirectories). When the quota is exceeded, object and subdirectory creation in that directory fails.

  • Quantity quota: the maximum number of objects and subdirectories a directory can contain. When the quota is exceeded, object and subdirectory creation in that directory fails.

Limitations

  • Quotas apply to a directory as a whole. Quotas based on individual object storage classes are not supported.

Usage notes

Directory usage statistics are not updated in real time. When a directory contains a large number of objects or has high capacity utilization, actual usage may exceed the configured quota.

Prerequisites

Before you begin, ensure that you have:

  • An OSS bucket with OSS-HDFS enabled

  • An AccessKey pair (AccessKey ID and AccessKey secret) for your Alibaba Cloud account or a RAM user with the required permissions

  • The JindoFS SDK installed on your machine

Configure directory quotas

Step 1: Download and install the JindoFS SDK

Download the JindoFS SDK installation package and extract it.

Step 2: Set up credentials and environment

  1. Go to the bin directory of the extracted JindoFS SDK package. The following example uses the Linux package:

    cd jindofs-sdk-x.x.x-linux/bin/

    Replace jindofs-sdk-x.x.x-linux with your actual package version name.

  2. In the bin directory, create a configuration file named jindofs.cfg and add your AccessKey pair:

    [client]
    fs.oss.accessKeyId = <your-access-key-id>
    fs.oss.accessKeySecret = <your-access-key-secret>
    PlaceholderDescription
    <your-access-key-id>Your AccessKey ID
    <your-access-key-secret>Your AccessKey secret
  3. Set the JINDOSDK_CONF_DIR environment variable to the absolute path of the jindofs.cfg configuration file:

    export JINDOSDK_CONF_DIR=<JINDOSDK_CONF_DIR>

Step 3: Enable the directory quota feature

Run the following command to enable the directory quota feature for your bucket:

./jindofs admin -putConfig -dlsUri oss://<bucket>.<oss-hdfs-endpoint>/ -conf namespace.directory.quota.enable=true
PlaceholderDescriptionExample
<bucket>Name of the bucket with OSS-HDFS enabledmy-bucket
<oss-hdfs-endpoint>OSS-HDFS endpoint for your regioncn-shanghai.oss-dls.aliyuncs.com

To verify that the feature is enabled:

./jindofs admin -getConfig -dlsUri oss://<bucket>.<oss-hdfs-endpoint>/ -name namespace.directory.quota.enable

Step 4: Set quotas on a directory

Replace <bucket>, <oss-hdfs-endpoint>, and <path> with your actual values in all commands below.

Set a quantity quota (maximum number of objects and subdirectories):

./jindofs admin -setQuota -dlsUri oss://<bucket>.<oss-hdfs-endpoint>/<path> -q 1000

The -q 1000 flag sets the limit to 1,000 objects and subdirectories combined.

Set a capacity quota (maximum storage size, in bytes):

./jindofs admin -setSpaceQuota -dlsUri oss://<bucket>.<oss-hdfs-endpoint>/<path> -q 1000

The -q 1000 flag sets the limit to 1,000 bytes.

Verify the configured quotas:

./jindofs fs -count -q -v -dlsUri oss://<bucket>.<oss-hdfs-endpoint>/<path>

Step 5: Remove quotas (optional)

If your business requirements change or the storage capacity of a directory exceeds your initial plan, remove the quotas from a directory.

Remove the quantity quota:

./jindofs admin -clearQuota -dlsUri oss://<bucket>.<oss-hdfs-endpoint>/<path>

Remove the capacity quota:

./jindofs admin -clearSpaceQuota -dlsUri oss://<bucket>.<oss-hdfs-endpoint>/<path>