MaxCompute provides a variety of tools to upload and download data. Most of these tools are open source on GitHub and maintained by the community. Select a tool based on your use case.
Alibaba Cloud services
MaxCompute client (via Tunnel)
The client uses the Tunnel SDK and provides a built-in tunnel command to upload and download data.
For installation and basic usage, see MaxCompute client.
NoteThis project is open source. You can view the code at aliyun-odps-console.
DataWorks Data Integration (via Tunnel)
Data Integration is a stable, efficient, and elastic platform for data synchronization. It provides offline full and real-time incremental data synchronization, integration, and exchange services for various heterogeneous data storage systems on Alibaba Cloud.
Supported data sources include MaxCompute, RDS (MySQL, SQL Server, and PostgreSQL), Oracle, FTP, ADS (AnalyticDB), OSS, Memcache, and DRDS.
Data Transmission Service (DTS) (via Tunnel)
Data Transmission Service (DTS) is an Alibaba Cloud service that provides data migration, real-time data subscription, and real-time data synchronization between various data sources, such as RDBMS (relational database), NoSQL, and OLAP.
DTS supports real-time data synchronization from RDS and MySQL instances to MaxCompute tables, but does not support synchronizing data from other types of data sources to MaxCompute. For more information, see Create a real-time synchronization job to sync data from an RDS instance to a MaxCompute table.
MaxCompute Migration Service (MMS)
MaxCompute Migration Service (MMS) is a cloud migration service for large-scale, platform-wide data migration. MMS integrates with the MaxCompute Spark engine to automatically migrate table schemas and data. It supports large-scale data migration to MaxCompute from various sources, including BigQuery, Hive, Databricks, Redshift, and other MaxCompute instances. For more information, see MaxCompute Migration Service (MMS)
Open source tools
Sqoop (via Tunnel)
Built on the community edition of Sqoop 1.4.6, this tool is enhanced to support MaxCompute. You can use it to import data to MaxCompute from sources such as relational databases (like MySQL), HDFS, or Hive, or to export data from MaxCompute to relational databases. For more information, see MaxCompute Sqoop.
NoteThis project is open source. You can view the code at aliyun-maxcompute-data-collectors.
Kettle (via Tunnel)
Kettle is an open source ETL tool written in Java. It runs on Windows, Unix, and Linux and provides a graphical user interface to design ETL workflows with a drag-and-drop interface. For more information, see Use the Kettle plug-in for MaxCompute to migrate data.
NoteThis project is open source. You can view the code at aliyun-maxcompute-data-collectors.
Flume (via DataHub)
Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large volumes of log data from various sources to a centralized data store. It supports multiple Source and Sink plug-ins.
The DataHub Sink plug-in for Apache Flume uploads log data to DataHub in real time and archives the data to MaxCompute tables. For more information, see flume_plugin.
NoteThis project is open source. You can view the code at aliyun-maxcompute-data-collectors.
Fluentd (via DataHub)
Fluentd is an open source software for collecting logs from various sources, including application logs, system logs, and access logs. It supports plug-ins that allow you to filter log data and store it in different destinations, such as MySQL, Oracle, MongoDB, Hadoop, and Treasure Data.
The DataHub plug-in for Fluentd uploads log data to DataHub in real time and archives the data to MaxCompute tables. For more information, see Fluentd plug-in.
Logstash (via DataHub)
Logstash is an open source log collection and processing framework. The logstash-output-datahub plug-in imports data into DataHub. With simple configuration, you can collect and transmit data, then use MaxCompute and StreamCompute to build an end-to-end stream analytics solution.
The DataHub plug-in for Logstash uploads log data to DataHub in real time and archives the data to MaxCompute tables.
OGG (via DataHub)
The DataHub plug-in for OGG supports real-time incremental synchronization of data from an Oracle database to DataHub, which is then archived to MaxCompute tables. For more information, see Use the OGG DataHub plug-in to synchronize data from Oracle.
NoteThis project is open source. You can view the code at aliyun-maxcompute-data-collectors.