Use datasets

Prerequisites: Create a dataset

1. Create a bucket or Create a file system.

   This example uses an OSS dataset. We create a bucket named datasets-oss and a directory named /dataset01/v1 in the China (Shanghai) region.

2. Create a dataset.

   In this example, we create an OSS dataset named datasets-oss and mount the OSS path oss://datasets-oss.oss-cn-shanghai.aliyuncs.com/dataset01/v1/ to /mnt/data/dataset01.

1. Configure a dataset for a node

In the Run Configuration of a Shell node or Python node, configure the information for the dataset datasets-oss.

2. Use a dataset in a node

This example uses a Shell node. After you associate an OSS dataset with a Shell node, you can manage OSS data in the Shell node code as if it were local files. The following example uses the default ossfs 2.0 tool to upload a file file01.txt to the mount path /mnt/data/dataset01 of the OSS dataset datasets-oss and write data to it.

Code sample:

echo "Hello World" > /mnt/data/dataset01/file01.txt
ls -tl /mnt/data/dataset01

3. Verify data writes in OSS

After the dataset is configured in the node is completed, the file is automatically written to the OSS storage path that corresponds to the dataset mount path. You can go to the OSS storage path to view the file written from the DataWorks node. In this example, the mount path /mnt/data/dataset01 of the OSS dataset datasets-oss corresponds to the storage path oss://datasets-oss.oss-cn-shanghai.aliyuncs.com/dataset01/v1/. The following shows an example of data written to this OSS path.

Prerequisites: Create a dataset

1. Create a bucket or Create a file system.

2. Create a dataset.

   This example uses a NAS dataset. We create a NAS dataset named datasets-nas in the China (Shanghai) region and mount the NAS path nas://****.cn-shanghai.nas.aliyuncs.com/mnt/dataset02/v1/ to /mnt/data/dataset02.

1. Configure a dataset for a personal development environment

Create a personal development environment instance and select the NAS dataset datasets-nas that you created.

{"nasOptions":"vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"}

2. Read a dataset in a notebook

1. At the top of the data development page, switch to the personal development environment instance, and then create a Notebook development.

2. Configure the following content in the notebook.

   1. Write data to a specified path in the dataset.

      import os
      
      # Define target path and file name
      file_path = "/mnt/data/dataset02/file02.txt"
      
      # Ensure the directory exists (create it if it does not exist)
      os.makedirs(os.path.dirname(file_path),exist_ok=True)
      
      # Write content
      content = "Hello World!"
      try:
          with open(file_path, "w", encoding="utf-8") as file:
              file.write(content)
          print(f"File written successfully: {file_path}")
      except Exception as e:
          print(f"Write failed: {str(e)}")

   2. Read data from a specified path in the dataset.

      file_path = "/mnt/data/dataset02/file02.txt"
      with open(file_path, "r") as file:
          content = file.read()
      content

3. Run the two Python code snippets separately.

   Note

   Before running the code, confirm the Python kernel. This example uses Python 3.11.9.

   

3. Configure scheduling

On the right side of the Notebook node, click Scheduling Settings and add dataset-related options. The parameter settings are the same as those configured when you created the personal development environment instance.

Use ossfs 2.0 to mount and access OSS

ossfs 2.0 is a client designed for high-performance access to OSS (object storage) through mounting. It excels at sequential read and write operations and can fully leverage the high-bandwidth advantage of OSS. It is suitable for scenarios that require high sequential read and write performance, such as AI training and big data processing. These workloads mainly involve sequential and random reads, sequential (append-only) writes, and do not require full POSIX semantics.

In Datasets > Advanced Settings, you can set advanced parameters. Separate multiple configurations with commas (,). For information about the advanced parameters and more configurations, see ossfs 2.0 mount options. The following are some common scenario examples:

• Data source remains unchanged during a task: If all files to be read are not modified during the read process, you can configure a larger cache time to reduce the number of metadata requests. A typical scenario is reading a batch of existing old files and generating a batch of new files after processing.

  {"mountOssType":"ossfs", "attr_timeout": "7200"}

• Fast read and write: Use a smaller metadata cache time to balance cache efficiency and data freshness.

  {"mountOssType":"ossfs", "attr_timeout": "3", "negative_timeout":"0"}

• Read/write consistency for distributed tasks: ossfs updates file data based on metadata caching by default. Use the following configuration to achieve a synchronized view across multiple nodes.

  { "mountOssType":"ossfs","negative_timeout": "0", "close_to_open":"false"}

• OOM caused by opening too many files simultaneously: When task concurrency is high and a large number of files are opened simultaneously, OOM (out of memory) issues may occur. Use the following configuration to mitigate memory pressure.

  {"mountOssType":"ossfs","readdirplus": "false", "inode_cache_eviction_threshold":"300000"}

Use ossfs 1.0 to mount and access OSS

ossfs 1.0 allows you to mount an OSS storage space as a local file system on a Linux system. Compared with ossfs 2.0, ossfs 1.0 supports more comprehensive file operations. If you encounter file operation incompatibility issues after mounting a directory with ossfs 2.0, try using ossfs 1.0.

For the specific parameters required for ossfs 1.0 mounting, see ossfs 1.0 mount options.

Use JindoFuse to mount and access OSS

Use the Configure JindoFuse component to mount an OSS dataset to a specified path in the container. This tool is suitable for the following scenarios:

• You want to read OSS data as if it were local data, or the dataset is small enough to benefit from JindoFuse's local cache acceleration.

• You need to write data to OSS.

In Datasets > Advanced Settings, you can set advanced parameters. Separate multiple configurations with commas (,). The following is only an example of the configuration method. For parameter descriptions and more configurations, see JindoFuse User Guide and Mount and access with JindoFuse.

{ 
  "mountOssType":"jindofuse",
  "fs.oss.download.thread.concurrency": "2x the number of CPU cores",
  "fs.oss.upload.thread.concurrency": "2x the number of CPU cores",
  "attr_timeout": 3,
  "entry_timeout": 0,
  "negative_timeout": 0
}

Use NAS datasets

When the dataset is of the NAS type, you can specify the configurations (nasOptions parameter) for accessing NFS-based File Storage (NAS). The following shows the default configuration. You can also refer to Manually mount an NFS file system to customize parameter values.

{"nasOptions":"vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"}