Dataflow for CPFS for Lingjun

Features

CPFS for Lingjun supports these dataflow features:

• Account-level dataflow

  Supports same-account and cross-account dataflows with OSS buckets.

• Directory-level dataflow

  Map any subdirectory of a CPFS for Lingjun file system to any prefix in an OSS bucket for granular permission control and flexible data transfer.

• Data import and export

  • Create batch or streaming tasks to import and export data between a CPFS for Lingjun file system and OSS. Batch tasks preload an entire dataset before a compute job starts. Streaming tasks handle on-demand writeback and preload of individual files, such as model checkpoints, during training.

    Warning

    CPFS for Lingjun exports file modification timestamps to OSS custom metadata named x-oss-meta-alihbr-sync-mtime. Do not delete or modify this metadata, or the file system timestamps will be incorrect.

  • If a task fails, check the task report to identify the cause.

    Important

    The task report is for reference only. The destination data is the definitive version. Verify data consistency between source and destination.

Dataflow task types

CPFS for Lingjun provides two dataflow task types: batch tasks and streaming tasks.

Batch task

A batch task imports or exports all files in a directory at once, ideal for preloading entire datasets before AI training.

Streaming task

A streaming task transfers individual files on demand, such as model checkpoints during training. Submit a subtask through an API operation to trigger each transfer. The system does not automatically synchronize file changes.

Task descriptions

Four task types are available: import, export, stream import, and stream export.

Limitations

• Dataflow

  • Same-account dataflows require CPFS for Lingjun 2.4.0 or later. Cross-account dataflows require 2.6.0 or later.

  • Maximum 10 dataflows per CPFS for Lingjun file system.

  • Each file system path can link to only one OSS bucket.

  • Dataflows cannot span different regions.

• Restrictions on paths, filenames, and directory names

  • Renaming a non-empty directory in a dataflow-associated path returns a Permission Denied or directory-not-empty error.

  • Use special characters in directory and file names with caution.

    • Supported characters include uppercase and lowercase letters, numbers, exclamation points (!), hyphens (-), underscores (_), periods (.), asterisks (*), and parentheses (()).

    • The following characters are not supported and may cause task failures:

      • A subdirectory or file name cannot be "..".

      • A path cannot contain backslashes (\) or consecutive backslashes (\\).

      • A subdirectory or file name cannot contain forward slashes (/).

  • Maximum path length: 1,023 characters.

• Dataflow task limitations

  • Streaming tasks require CPFS for Lingjun 2.6.0 or later and are API-only.

  • Maximum four concurrent batch tasks per dataflow. Concurrent streaming tasks are unlimited.

  • Import limitations

    • Importing a symbolic link converts it into a regular data file. The symbolic link information is lost.

    • If an OSS bucket has versioning enabled, only the latest version of an object is copied.

    • Filenames or subdirectory names cannot exceed 255 bytes.

  • Export limitations

    • Symbolic links are exported as regular, empty objects. The linked file is not synchronized.

    • Hard links are exported as regular files.

    • Socket, Device, and Pipe files are exported as regular, empty objects.

    • Directory paths cannot exceed 1,023 characters.

Performance metrics

How to achieve higher throughput

1. Expand capacity: Increasing the storage capacity of a CPFS for Lingjun file system directly improves linear throughput.

2. Increase quotas: If you reach the VPC network bottleneck (10 GB/s for reads or 5 GB/s for writes), submit a ticket to request a bandwidth quota increase.

Why is actual speed lower than baseline

1. Check whether the OSS bucket has bandwidth throttling or rate-limiting policies in place.

2. Check whether heavy read/write workloads on the CPFS for Lingjun file system are competing for dataflow bandwidth.

3. Check whether the dataflow task involves a large number of small file operations, many small directories, or wide directories containing many small files, which significantly reduce aggregate throughput.

Billing

Dataflow for CPFS for Lingjun is in public preview and free of charge.

Procedure

1. Create a dataflow.

   • Same-account: Create a same-account dataflow.

   • Cross-account: Create a cross-account dataflow.

2. Create a batch task or a streaming task.

   • Streaming: Manage streaming tasks (OpenAPI).

   • Batch: Manage batch tasks.

3. Verify the data.

   After a task completes, verify data accuracy at the destination.

   Warning

   If you delete source data without verifying destination data, you are responsible for any resulting data loss.