Data lake overview

Data ingestion and import

A data lake allows you to import any amount of data in real time, collect data from multiple sources, and store it in its original format. This approach lets you scale to any data size while saving time on defining data structures, schemas, and transformations. OSS provides the following methods to import data:

• Upload data directly to OSS over the internal network.

• Upload data from your on-premises data center to OSS by using Express Connect.

• Migrate petabytes of data to OSS by using Data Online Migration or Data Transport.

• Upload data directly to OSS over the public network. Because the public network introduces significant security risks, pay close attention to domain name management and access control when using this method. Read the following documents carefully:

  • Access OSS through a custom domain name

  • Block public access

  • Check Referers in OSS resource requests from other websites

Secure and cost-effective storage

A data lake allows you to store massive amounts of unstructured data from sources such as mobile applications, IoT devices, social media, and the Internet of Vehicles (IoV). This data requires automated cost optimization and must be kept secure at all times. OSS provides the following capabilities:

• Five storage classes from hot to cold

• Use lifecycle rules to automatically transition cold data to lower-cost storage classes.

• Use versioning to prevent accidental data deletion.

• Use sensitive data protection to scan your data lake for sensitive data and apply data protection capabilities.

Data management at scale

In a data lake, different business units might store data under different prefixes within a single bucket, or different business units and departments might store data across multiple buckets. This requires both granular management of data within a bucket and seamless data flow between buckets. OSS provides a rich set of capabilities to handle these complex scenarios:

• Use access points to configure data access permissions for different teams.

• Use bucket inventory to get an overview of storage usage by different teams within a bucket.

• Use data replication to automatically synchronize data between buckets in the same region or across different regions.

Multi-workload performance management

• In daily data lake operations, concurrent tasks such as data ingestion, preprocessing, AI training, and debugging can lead to uneven resource allocation and contention between buckets and RAM users. OSS provides the resource pool QoS feature, which allows you to dynamically adjust throttling for buckets and their requesters. This ensures that critical services and compute-intensive tasks get priority access to resources during high-load periods, maintaining business stability.

• For scenarios that demand low latency, such as high-QPS queries in data warehouses and search engines, low-latency responses for online services, and repeated low-latency model pulls in AI inference, OSS provides the OSS accelerator feature. By caching hot files on high-performance NVMe SSDs, this feature reduces data read latency and increases QPS, significantly improving the performance of real-time computing jobs.

Analytics and AI framework integration

A data lake runs a wide variety of analytics and AI computing frameworks. A complete enterprise production workflow often uses multiple computing frameworks. Each framework has different interfaces and methods for data access. To connect to these ecosystems and reduce application refactoring costs, OSS provides a rich set of clients, tools, and features:

• OSS provides a rich set of SDKs for mainstream programming languages used in web applications. If you have programming experience, we recommend using an OSS SDK for high-performance data access. For high-performance programming best practices, see Use concurrent Python libraries to significantly improve bandwidth.

• If you already have experience running Hadoop-related ecosystems on cloud object storage, we first recommend using the OSS Connector for Hadoop to read and write OSS data. This method allows you to efficiently use the unlimited scalability and various enterprise-level features of OSS.

• If you currently rely heavily on open-source HDFS and cannot refactor your applications in the short term, we recommend using OSS-HDFS. This service provides standard interfaces that are fully compatible with HDFS, while offering superior performance and elastic scalability compared to traditional HDFS. OSS-HDFS integrates seamlessly with Alibaba Cloud E-MapReduce (EMR) and open-source components such as Hadoop and Spark. This solution's strong HDFS compatibility helps enterprises smoothly migrate traditional HDFS workloads from on-premises data centers to the cloud without modifying existing HDFS-based big data applications. However, due to differences in feature definitions between open-source HDFS and object storage, you may lose some of the native advanced data management capabilities of OSS. For more information, see Features of OSS-HDFS. Therefore, after migrating to the cloud, we recommend gradually adapting your services to use the OSS Connector. This approach allows you to fully leverage the performance and data management capabilities of OSS.

• If some of your applications need to access data by using traditional file system interfaces and cannot be refactored, OSS provides the ossfs client to meet their data read and write needs:

  • For modern applications like AI training, AI inference, and autonomous driving simulation, which typically have relaxed POSIX semantics requirements, we recommend using ossfs 2.0 for optimal performance. If you are unsure about your application's specific access patterns, we suggest you test with ossfs 2.0 first. If it is not compatible, you can then downgrade to ossfs 1.0.

  • For traditional applications, you can use ossfs 1.0 to read and write data stored in OSS. However, due to significant semantic differences between OSS and NAS, and the high POSIX compatibility and performance demands of some traditional applications, we do not recommend using ossfs 1.0 with OSS as a substitute for NAS. In these cases, for the best compatibility and performance, choose Alibaba Cloud File Storage NAS.

• If you are familiar with loading AI datasets by using the PyTorch dataset framework but are not familiar with OSS SDKs, we recommend that you use the OSS Connector for AI/ML to accelerate model training. This allows you to achieve optimal performance when reading datasets from OSS without needing to learn how to use an SDK.

• For daily file management, administrators and developers can use the following OSS tools:

  • Command-line tool ossutil 2.0

  • Graphical management tool ossbrowser 2.0