DocsICP FilingConsole
官方文档
首页

Create a dataset

更新时间：
复制 MD 格式
产品详情
我的收藏

A dataset is a container for storing and managing metadata. This topic describes how to create a dataset.

Usage notes

  • Store unrelated files in different datasets and related files in the same dataset.

  • The number of datasets in a project cannot exceed the project's limit.

  • The number of files in a dataset cannot exceed its limit. The total number of files across all datasets in a project cannot exceed the project's limit.

  • The number of OSS buckets bound to a dataset cannot exceed its limit. The total number of buckets bound to all datasets in a project cannot exceed the project's limit.

  • When building a metadata index, a dataset's workflow template takes precedence over the project's workflow template. If the dataset does not have a workflow template, IMM uses the project's template. For more information about workflow templates, see Workflow templates and operators.

Prerequisites

  • You have created an AccessKey pair. For more information, see Create an AccessKey pair.

  • You have activated OSS, created a bucket, and uploaded files to the bucket. For more information, see Upload files.

  • You have activated IMM. For more information, see Activate IMM.

  • You have created a project in the IMM console. For more information, see Create a project.

    Note

    • You can also call the CreateProject operation to create a project. For more information, see Create a project.

    • You can call the ListProjects operation to list all projects in a specified region.

Procedure

Step 1: Create a dataset

This example calls the CreateDataset operation to create a dataset named test-dataset with the description Dataset 1 and the template Official:ImageManagement in the test-project project.

  • Sample request

    {
 "ProjectName": "test-project",
 "DatasetName": "test-dataset",
 "Description": "Dataset 1",
 "TemplateId": "Official:ImageManagement"
}

  • Sample response

    {
    "RequestId": "9AB4BD43-C4E5-06AA-A7AB-****",
    "Dataset": {
        "FileCount": 0,
        "BindCount": 0,
        "ProjectName": "test-project",
        "CreateTime": "2022-07-05T10:43:32.429344821+08:00",
        "DatasetMaxTotalFileSize": 90000000000000000,
        "DatasetMaxRelationCount": 100000000000,
        "DatasetMaxFileCount": 100000000,
        "DatasetName": "test-dataset",
        "DatasetMaxBindCount": 10,
        "UpdateTime": "2022-07-05T10:43:32.429344821+08:00",
        "DatasetMaxEntityCount": 10000000000,
        "TotalFileSize": 0,
        "TemplateId": "Official:ImageManagement"
    }
}

  • Complete sample code (Python SDK v1.27.3)

    # -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint.
        config.endpoint = f'imm.cn-shenzhen.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # An Alibaba Cloud account AccessKey pair can perform all API operations. For security, we recommend using a RAM user for API access or routine O&M.
        # Do not hard-code your AccessKey ID and AccessKey secret in the code. This practice can lead to security risks if the keys are leaked.
        # This example retrieves an AccessKey pair from environment variables for authentication. For more information about how to configure environment variables, see https://help.aliyun.com/document_detail/2361894.html.
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        create_dataset_request = imm_20200930_models.CreateDatasetRequest(
            project_name='test-project',
            dataset_name='test-dataset',
            description='Dataset 1',
            template_id='Official:ImageManagement'
        )
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response of the API operation.
            response = client.create_dataset_with_options(create_dataset_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if an error occurs.
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

(Optional) Step 2: Query a dataset

This example calls the GetDataset operation to query the dataset named test-dataset in the test-project project.

  • Sample request

    {
 "ProjectName": "test-project",
 "DatasetName": "test-dataset"
}

  • Sample response

    {
    "RequestId": "9AB4BD43-C4E5-06AA-E4B2-****",
    "Dataset": {
        "FileCount": 0,
        "BindCount": 0,
        "ProjectName": "test-project",
        "CreateTime": "2022-07-05T10:43:32.429344821+08:00",
        "DatasetMaxTotalFileSize": 90000000000000000,
        "DatasetMaxRelationCount": 100000000000,
        "DatasetMaxFileCount": 100000000,
        "DatasetName": "test-dataset",
        "DatasetMaxBindCount": 10,
        "UpdateTime": "2022-07-05T10:43:32.429344821+08:00",
        "DatasetMaxEntityCount": 10000000000,
        "TotalFileSize": 0,
        "TemplateId": "Official:ImageManagement"
    }
}

  • Complete sample code (Python SDK v1.27.3)

    # -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client.
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint.
        config.endpoint = f'imm.cn-shenzhen.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # An Alibaba Cloud account AccessKey pair can perform all API operations. For security, we recommend using a RAM user for API access or routine O&M.
        # Do not hard-code your AccessKey ID and AccessKey secret in the code. This practice can lead to security risks if the keys are leaked.
        # This example retrieves an AccessKey pair from environment variables for authentication. For more information about how to configure environment variables, see https://help.aliyun.com/document_detail/2361894.html.
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        get_dataset_request = imm_20200930_models.GetDatasetRequest(
            # Specify the name of the IMM project.
            project_name='test-project',
            # Specify the name of the dataset.
            dataset_name='test-dataset',
            # Do not return statistics, such as the number of files and total file size.
            with_statistics=False
        )
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response of the API operation.
            response = client.get_dataset_with_options(get_dataset_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if an error occurs.
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()
Previous：OverviewNext：Create a metadata index