Log data originates from many sources—Elastic Compute Service (ECS) instances, containers, mobile terminals, open-source software, website services, and JavaScript. This topic describes how to migrate log data to MaxCompute by using Tunnel, DataHub, Log Service, and DataWorks Data Integration. The right method depends on whether your pipeline is batch-based or streaming, and whether you need real-time processing.
Choose a migration method
| Method | How it works | When to use |
|---|---|---|
| Tunnel | Uploads offline log data directly to a MaxCompute table. No intermediate service required. | Your data is already collected and you need to upload large volumes of offline data to MaxCompute tables. Tunnel is suitable for offline computing. |
| DataHub | Writes data to DataHub in real time. DataHub DataConnector then synchronizes the streaming data to MaxCompute within a few minutes for offline computing. | This method is mainly used for public preview and development. DataHub uploads data in real time and is suitable for stream processing. After data is uploaded to DataHub, the data is stored in a table for real-time processing, and DataHub executes scheduled tasks to synchronize the data to a MaxCompute table within a few minutes for offline computing. |
| LogHub | Simple Log Service (SLS) provides a data shipping feature to ship real-time logs collected by LogHub to MaxCompute. For more information, see Migrate log data to MaxCompute using Simple Log Service. | This method is suitable for scenarios with large data volumes and has no data volume limit. It supports various storage formats, such as row-oriented, column-oriented, and TEXTFILE, and allows for user-defined partition configurations. |
| DataWorks Data Integration | Runs batch synchronization nodes on a regular schedule to pull log data into MaxCompute. | You need to configure batch synchronization tasks in DataWorks Data Integration. The tasks are executed on a regular basis. |
Recommended starting point: Use Tunnel for straightforward large-volume batch loads. If your data arrives continuously and requires real-time processing, switch to DataHub. If you need scheduled synchronization with automated, recurring tasks, use DataWorks Data Integration.