Performance of OSS Connector in AI/ML Dataset Processing

Test description

Test scenarios: Test performance of reading data from datasets by using internal endpoints and accelerated endpoints.

Test data: The performance tests are performed based on approximately 1 TB of 10,000,000 images with an average size of 100 KB.

Test environment: The performance tests are performed based on a network-enhanced general-purpose g7nex Elastic Compute Service (ECS) instance with 128 vCPUs, 512 GB of memory, and 160 Gbit/s internal bandwidth.

Datasets: The dataset of type OssIterableDataset and the dataset of type OssMapDataset are created by using OSS Connector for AI/ML. The dataset of type ossfs with ImageFolder is created by using ossfs.

Performance tests

• Test parameters

  Parameter	Value/Operation	Description
  dataloader batch size	256	Each batch task processes 256 samples.
  dataloader workers	32	Data is loaded in parallel by using 32 processes.
  transform	trans = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) def transform(object): img = Image.open(io.BytesIO(object.read())).convert('RGB') val = trans(img) return val, object.label	Data is preprocessed.

• Test results

  Dataset created by using	Dataset type	Time to load 10 million images via the OSS internal endpoint (s)	Time to load 10 million images via the accelerated endpoint (s)
  OSS Connector for AI/ML	OssIterableDataset	2182	2107
  	OssMapDataset	2493	2288
  ossfs	Ossfs with ImageFolder	178571	39840

Optimal performance tests

• Test parameters

  Parameter	Value/Operation	Description
  dataloader batch size	256	Each batch task processes 256 samples.
  dataloader workers	32	Data is loaded in parallel by using 32 processes.
  transform	def transform(object): data = object.read() return object.key, object.label	Data is not preprocessed.

• Test results

  Dataset created by using	Dataset type	Time to load 10 million images via the OSS internal endpoint (s)	Time to load 10 million images via the accelerated endpoint (s)
  OSS Connector for AI/ML	OssIterableDataset	100	81
  	OssMapDataset	176	127

Performance data analysis

When data is read from datasets, OssIterableDataset and OssMapDataset offer significantly better performance compared with datasets created by using ossfs with ImageFolder. Specifically, they are approximately 80 times faster without the OSS accelerator and approximately 18 times faster with the OSS accelerator. The performance tests show that OSS Connector for AI/ML can significantly improve data processing speed and model training efficiency.

Reading data from OssIterableDataset and OssMapDataset with the OSS accelerator enabled is approximately 1.4 times faster than reading with the OSS accelerator disabled. Even without the OSS accelerator, OSS Connector for AI/ML can handle highly concurrent access at high bandwidth. Combining OSS Connector for AI/ML with the OSS accelerator delivers even more powerful performance.

Conclusion

You can use OSS Connector for AI/ML in your Python code to stream OSS objects. OSS Connector for AI/ML increases data read speeds and is suitable for most model training scenarios. For higher data processing performance, you can use OSS Connector for AI/ML together with the OSS accelerator.