The UNLOAD command exports data from a MaxCompute project to external storage, such as Object Storage Service (OSS) or Hologres, for use by other compute engines.
Permissions
MaxCompute permissions: The account used for the operation must have read (Select) permission on the table data to be exported. For more information about how to grant permissions, see MaxCompute permissions.
External storage permissions: Before you export data from MaxCompute to external storage, you must grant MaxCompute the permissions to access the external storage, which can be OSS or Hologres.
OSS authorization: One-click authorization is supported for higher security. For more information, see Authorization solutions for accessing external data sources. The examples use one-click authorization and the role is named
AliyunODPSDefaultRole.Hologres authorization: To complete the authorization, create a RAM role, grant it the permissions to access MaxCompute, and add the role to the Hologres instance. For more information, see Create a Hologres foreign table (STS mode).
Export data to external storage
The UNLOAD command uses append mode. When you export data to the same destination path multiple times, the system does not overwrite existing files. Instead, it creates new files in the path. To overwrite the data, you must manually clear the destination path before you run the UNLOAD command.
Export data to OSS
Export data in a text format (CSV or TSV)
This method uses a built-in StorageHandler to export data. A .csv or .tsv file name extension is added to the output files by default.
Syntax
UNLOAD FROM {<select_statement>|<table_name> [PARTITION (<pt_spec>)]}
INTO | OVERWRITE
LOCATION <oss_location>
STORED BY <StorageHandler>
[WITH SERDEPROPERTIES ('<property_name>'='<property_value>',...)];Core configurations
Configuration item | Description |
oss_location | The destination path in OSS. The format is |
StorageHandler | Built-in processor:
|
SERDEPROPERTIES | Used to configure export properties. The most common properties are:
|
Examples
Example 1: Export data in CSV format and compress the data into a GZIP file
Example 2: Export partitioned data in TSV format and compress the data
Example 1: Export data in CSV format and compress the data into a GZIP file
Export data from the sale_detail table in a MaxCompute project to OSS. The following code shows the data in the sale_detail table:
-- Partition fields: sale_date, region
+------------+-------------+-------------+------------+------------+
| shop_name | customer_id | total_price | sale_date | region |
+------------+-------------+-------------+------------+------------+
| s1 | c1 | 100.1 | 2013 | china |
| s2 | c2 | 100.2 | 2013 | china |
| s3 | c3 | 100.3 | 2013 | china |
| null | c5 | NULL | 2014 | shanghai |
| s6 | c6 | 100.4 | 2014 | shanghai |
| s7 | c7 | 100.5 | 2014 | shanghai |
+------------+-------------+-------------+------------+------------+Procedure:
On the OSS side: Log on to the OSS console, create the
mc-unload/data_location/folder in an OSS bucket that is in theoss-cn-hangzhouregion, and then construct an OSS path. For more information about how to create an OSS bucket, see Create a bucket in the console. Construct the OSS path based on the bucket, region, and endpoint information. For example:oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_locationOn the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
Method 1: Use a SELECT query
-- Set the amount of data to be read from the MaxCompute table by a single worker to 256 MB. This indirectly controls the number of exported files. SET odps.stage.mapper.split.size=256; UNLOAD FROM (SELECT * FROM sale_detail) INTO LOCATION 'oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_location' STORED BY 'com.aliyun.odps.CsvStorageHandler' WITH SERDEPROPERTIES ( 'odps.properties.rolearn'='acs:ram::<uid>:role/AliyunODPSDefaultRole', 'odps.text.option.gzip.output.enabled'='true' );Method 2: Directly specify the table name (no query fees are generated)
-- Set the amount of data to be read from the MaxCompute table by a single worker to 256 MB. This indirectly controls the number of exported files. SET odps.stage.mapper.split.size=256; UNLOAD FROM sale_detail INTO LOCATION 'oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_location' STORED BY 'com.aliyun.odps.CsvStorageHandler' WITH SERDEPROPERTIES ( 'odps.properties.rolearn'='acs:ram::<uid>:role/AliyunODPSDefaultRole', 'odps.text.option.gzip.output.enabled'='true' );Verify the result: Log on to the OSS console and view the exported data in the destination OSS path. The exported file is in CSV format and is compressed using GZIP.
Example 2: Export partitioned data in TSV format and compress the data
Export the data in the partitions for which sale_date='2013' and region='china' are specified from the sale_detail table in TSV format and compress the data using GZIP.
-- Partition fields: sale_date, region
+------------+-------------+-------------+------------+------------+
| shop_name | customer_id | total_price | sale_date | region |
+------------+-------------+-------------+------------+------------+
| s1 | c1 | 100.1 | 2013 | china |
| s2 | c2 | 100.2 | 2013 | china |
| s3 | c3 | 100.3 | 2013 | china |
| null | c5 | NULL | 2014 | shanghai |
| s6 | c6 | 100.4 | 2014 | shanghai |
| s7 | c7 | 100.5 | 2014 | shanghai |
+------------+-------------+-------------+------------+------------+Procedure:
On the OSS side: Log on to the OSS console, create the
mc-unload/data_location/folder in an OSS bucket that is in theoss-cn-hangzhouregion, and then construct an OSS path. For more information about how to create an OSS bucket, see Create a bucket in the console. Construct the OSS path based on the bucket, region, and endpoint information. For example:oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_locationOn the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
SET odps.stage.mapper.split.size=256; UNLOAD FROM sale_detail PARTITION (sale_date='2013',region='china') INTO LOCATION 'oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_location' STORED BY 'com.aliyun.odps.TsvStorageHandler' WITH SERDEPROPERTIES ( 'odps.properties.rolearn'='acs:ram::<uid>:role/AliyunODPSDefaultRole', 'odps.text.option.gzip.output.enabled'='true' );Verify the result: Log on to the OSS Management Console and view the import result in the target OSS path. The exported file is a GZIP-compressed file in the TSV format.
Export data in an open source format (such as Parquet or ORC)
This method supports exporting data to various open-source columnar storage or structured data formats.
Syntax
UNLOAD FROM {<select_statement>|<table_name> [PARTITION (<pt_spec>)]}
INTO | OVERWRITE
LOCATION <oss_location>
[ROW FORMAT SERDE '<serde_class>'
[WITH SERDEPROPERTIES ('<property_name>'='<property_value>',...)]
]
STORED AS <file_format>
[PROPERTIES ('<tbproperty_name>'='<tbproperty_value>')];Core configurations
Configuration item | Description |
oss_location | The destination path in OSS. The format is |
serde_class | Specifies the serialization and deserialization library. For example, use |
SERDEPROPERTIES |
|
file_format | Required. Specifies the file format, such as |
PROPERTIES | - |
Supported formats and compression methods
File format | Supported compression formats | Description |
PARQUET | SNAPPY, LZO | A columnar storage format that is suitable for analytical queries. |
ORC | SNAPPY, LZO | A columnar storage format that is suitable for the Hadoop ecosystem. |
TEXTFILE | GZIP | A text format that supports custom separators. |
RCFILE | - | A row-column hybrid storage format. |
SEQUENCEFILE | - | A Hadoop sequence file format. |
AVRO | - | A data serialization format. |
JSON | - | A JSON format. |
Examples
Example 1: Export data in PARQUET format and compress the data using SNAPPY
Example 3: Export data in TEXTFILE format and specify a separator
Example 1: Export data in PARQUET format and compress the data using SNAPPY
Export data from the sale_detail table in a MaxCompute project to OSS. The following code shows the data in the sale_detail table:
-- Partition fields: sale_date, region
+------------+-------------+-------------+------------+------------+
| shop_name | customer_id | total_price | sale_date | region |
+------------+-------------+-------------+------------+------------+
| s1 | c1 | 100.1 | 2013 | china |
| s2 | c2 | 100.2 | 2013 | china |
| s3 | c3 | 100.3 | 2013 | china |
| null | c5 | NULL | 2014 | shanghai |
| s6 | c6 | 100.4 | 2014 | shanghai |
| s7 | c7 | 100.5 | 2014 | shanghai |
+------------+-------------+-------------+------------+------------+Procedure:
On the OSS side: Log on to the OSS console, create the
mc-unload/data_location/folder in an OSS bucket that is in theoss-cn-hangzhouregion, and then construct an OSS path. For more information about how to create an OSS bucket, see Create a bucket in the console. Construct the OSS path based on the bucket, region, and endpoint information. For example:oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_locationOn the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
-- Set the amount of data to be read from the MaxCompute table by a single worker to 256 MB. This indirectly controls the number of exported files. SET odps.stage.mapper.split.size=256; UNLOAD FROM (SELECT * FROM sale_detail) INTO LOCATION 'oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_location' ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' WITH SERDEPROPERTIES ( 'odps.properties.rolearn'='acs:ram::<uid>:role/AliyunODPSDefaultRole' ) STORED AS PARQUET PROPERTIES('mcfed.parquet.compression'='SNAPPY');Verification Result: Log on to the OSS Management Console and view the import result in the target OSS path. The exported files are in PARQUET format and compressed using SNAPPY.
Example 2: Export partitioned data in PARQUET format
Export the data in the partitions for which sale_date='2013' and region='china' are specified from the MaxCompute sale_detail table in PARQUET format and compress the data using SNAPPY. The following code shows the data in the sale_detail table:
-- Partition fields: sale_date, region
+------------+-------------+-------------+------------+------------+
| shop_name | customer_id | total_price | sale_date | region |
+------------+-------------+-------------+------------+------------+
| s1 | c1 | 100.1 | 2013 | china |
| s2 | c2 | 100.2 | 2013 | china |
| s3 | c3 | 100.3 | 2013 | china |
| null | c5 | NULL | 2014 | shanghai |
| s6 | c6 | 100.4 | 2014 | shanghai |
| s7 | c7 | 100.5 | 2014 | shanghai |
+------------+-------------+-------------+------------+------------+Procedure:
On the OSS side: Log on to the OSS console, create the
mc-unload/data_location/folder in an OSS bucket that is in theoss-cn-hangzhouregion, and then construct an OSS path. For more information about how to create an OSS bucket, see Create a bucket in the console. Construct the OSS path based on the bucket, region, and endpoint information. For example:oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_locationOn the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
-- Set the amount of data to be read from the MaxCompute table by a single worker to 256 MB. This indirectly controls the number of exported files. SET odps.stage.mapper.split.size=256; UNLOAD FROM sale_detail PARTITION (sale_date='2013',region='china') INTO LOCATION 'oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_location' ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' WITH SERDEPROPERTIES ('odps.properties.rolearn'='acs:ram::<uid>:role/AliyunODPSDefaultRole') STORED AS PARQUET PROPERTIES('mcfed.parquet.compression'='SNAPPY');Verify the result: Log on to the OSS console and view the exported data in the destination OSS path. The exported file is in PARQUET format and is compressed using SNAPPY.
Example 3: Export data in TEXTFILE format and specify a separator
Export data from the sale_detail table to a TXT file and use a comma (,) as the separator.
-- Partition fields: sale_date, region
+------------+-------------+-------------+------------+------------+
| shop_name | customer_id | total_price | sale_date | region |
+------------+-------------+-------------+------------+------------+
| s1 | c1 | 100.1 | 2013 | china |
| s2 | c2 | 100.2 | 2013 | china |
| s3 | c3 | 100.3 | 2013 | china |
| null | c5 | NULL | 2014 | shanghai |
| s6 | c6 | 100.4 | 2014 | shanghai |
| s7 | c7 | 100.5 | 2014 | shanghai |
+------------+-------------+-------------+------------+------------+Procedure:
On the OSS side: Log on to the OSS console, create the
mc-unload/data_location/folder in an OSS bucket that is in theoss-cn-hangzhouregion, and then construct an OSS path. For more information about how to create an OSS bucket, see Create a bucket in the console. Construct the OSS path based on the bucket, region, and endpoint information. For example:oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_locationOn the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
-- Set the amount of data to be read from the MaxCompute table by a single worker to 256 MB. This indirectly controls the number of exported files. SET odps.sql.allow.fullscan=true; UNLOAD FROM (SELECT * FROM sale_detail) INTO LOCATION 'oss://oss-cn-beijing-internal.aliyuncs.com/mc-unload/data_location/' ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ('field.delim'=',') STORED AS TEXTFILE PROPERTIES ('odps.external.data.enable.extension'='true');Verify the result: Log on to the OSS console and view the exported data in the destination OSS path. The exported file is in TXT format and uses commas as separators.
Customize file name prefixes, suffixes, and extensions
When you use the UNLOAD command to export data to OSS, you can use properties in PROPERTIES or SERDEPROPERTIES to customize the prefix, suffix, and extension of output files.
Syntax
UNLOAD FROM {<select_statement>|<table_name> [PARTITION (<pt_spec>)]}
INTO | OVERWRITE
LOCATION <oss_location>
[ROW FORMAT SERDE '<serde_class>'
[WITH SERDEPROPERTIES ('<property_name>'='<property_value>',...)]
]
STORED AS <file_format>
[PROPERTIES ('<tbproperty_name>'='<tbproperty_value>',...)];Core configurations
property_name/tbproperty_name | Description | Example value |
odps.external.data.output.prefix | The prefix of the file name. It can contain letters, digits, and underscores (_). The prefix can be 1 to 10 characters in length. | 'mc_' |
odps.external.data.output.suffix | The suffix of the file name. It can contain letters, digits, and underscores (_). | '_hangzhou' |
odps.external.data.enable.extension | Specifies whether to show default extensions such as | 'true' |
odps.external.data.output.explicit.extension | A custom file name extension. This property has a higher priority than the default extension. | 'jsonl' |
Suffix reference
The following table describes the extensions that are automatically generated for different foreign tables when you set the odps.external.data.enable.extension=true parameter.
File format | SerDe | Suffix |
SEQUENCEFILE | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | .sequencefile |
TEXTFILE | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | .txt |
RCFILE | org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe | .rcfile |
ORC | org.apache.hadoop.hive.ql.io.orc.OrcSerde | .orc |
PARQUET | org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | .parquet |
AVRO | org.apache.hadoop.hive.serde2.avro.AvroSerDe | .avro |
JSON | org.apache.hive.hcatalog.data.JsonSerDe | .json |
CSV | org.apache.hadoop.hive.serde2.OpenCSVSerde | .csv |
When you export data that is compressed in SNAPPY or LZO format, the .snappy or .lzo extension is not displayed in the names of the exported files.
Examples
Example 1: Export a TEXTFILE format file and add a prefix and suffix
Example 2: Export a JSON format file and customize the file name extension
Example 1: Export a TEXTFILE format file and add a prefix and suffix
Export data from the sale_detail table in a MaxCompute project to OSS as a TXT file. The file is named in the mc_<system-generated>_beijing.txt format. The following code shows the data in the sale_detail table:
-- Partition fields: sale_date, region
+------------+-------------+-------------+------------+------------+
| shop_name | customer_id | total_price | sale_date | region |
+------------+-------------+-------------+------------+------------+
| s1 | c1 | 100.1 | 2013 | china |
| s2 | c2 | 100.2 | 2013 | china |
| s3 | c3 | 100.3 | 2013 | china |
| null | c5 | NULL | 2014 | shanghai |
| s6 | c6 | 100.4 | 2014 | shanghai |
| s7 | c7 | 100.5 | 2014 | shanghai |
+------------+-------------+-------------+------------+------------+Procedure:
On the OSS side: Log on to the OSS console, create the
mc-unload/data_location/folder in an OSS bucket that is in theoss-cn-hangzhouregion, and then construct an OSS path. For more information about how to create an OSS bucket, see Create a bucket in the console. Construct the OSS path based on the bucket, region, and endpoint information. For example:oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_locationOn the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
UNLOAD FROM (SELECT * FROM sale_detail) INTO LOCATION 'oss://oss-cn-beijing-internal.aliyuncs.com/***/textfile' row format serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' STORED AS textfile PROPERTIES ( 'odps.external.data.output.prefix'='mc_', 'odps.external.data.output.suffix'='_beijing', 'odps.external.data.enable.extension'='true');Verify the result: Log on to the OSS console and view the exported data in the destination OSS path. The name of the exported file is
mc_<system-generated>_beijing.txt.
Example 2: Export a JSON format file and customize the file name extension
Export data from the sale_detail table in a MaxCompute project to OSS as a JSON file. The file is named in the mc_<system-generated file name>_beijing.json format. The following code shows the data in the sale_detail table:
-- Partition fields: sale_date, region
+------------+-------------+-------------+------------+------------+
| shop_name | customer_id | total_price | sale_date | region |
+------------+-------------+-------------+------------+------------+
| s1 | c1 | 100.1 | 2013 | china |
| s2 | c2 | 100.2 | 2013 | china |
| s3 | c3 | 100.3 | 2013 | china |
| null | c5 | NULL | 2014 | shanghai |
| s6 | c6 | 100.4 | 2014 | shanghai |
| s7 | c7 | 100.5 | 2014 | shanghai |
+------------+-------------+-------------+------------+------------+Procedure:
On the OSS side: Log on to the OSS console, create the
mc-unload/data_location/folder in an OSS bucket that is in theoss-cn-hangzhouregion, and then construct an OSS path. For more information about how to create an OSS bucket, see Create a bucket in the console. Construct the OSS path based on the bucket, region, and endpoint information. For example:oss://oss-cn-hangzhou-internal.aliyuncs.com/mc-unload/data_locationOn the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
UNLOAD FROM (SELECT * FROM sale_detail) INTO LOCATION 'oss://oss-cn-beijing-internal.aliyuncs.com/***/json' ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' WITH SERDEPROPERTIES ( 'odps.external.data.output.prefix'='mc_', 'odps.external.data.output.suffix'='_beijing', 'odps.external.data.output.explicit.extension'='json') STORED AS JSON;Verify the result: Log on to the OSS console and view the exported data in the destination OSS path. The name of the exported file is
mc_<system-generated file name>_beijing.json.
Export data to Hologres
Limits
Dual-signature authorization: Dual-signature authorization is not supported for exporting data to Hologres.
Partitioned tables: You cannot export data to Hologres partitioned tables.
Type mapping: The data types of fields in the destination Hologres table must correspond to the data types of fields in the MaxCompute table. For more information, see Data type mapping between MaxCompute and Hologres.
Syntax
UNLOAD FROM {<select_statement>|<table_name> [PARTITION (<pt_spec>)]}
INTO
LOCATION <hologres_location>
STORED BY <StorageHandler>
[WITH SERDEPROPERTIES ('<property_name>'='<property_value>',...)];Core configurations
Configuration item | Description |
hologres_location | The Java Database Connectivity (JDBC) connection string of the destination Hologres table. The format is |
StorageHandler | Specifies a built-in handler. The value is fixed at |
SERDEPROPERTIES | This parameter must contain the following three properties:
|
Example
Export data from the MaxCompute table data_test to the Hologres table mc_2_holo. The following code shows the data in the data_test table:
+------------+------+
| id | name |
+------------+------+
| 3 | rgege |
| 4 | Gegegegr |
+------------+------+Procedure:
On the Hologres side: Create the destination table
mc_2_holoin Hologres.CREATE TABLE mc_2_holo (id INT, name TEXT);On the MaxCompute side: Log on to a MaxCompute client and run the UNLOAD command.
UNLOAD FROM (SELECT * FROM data_test) INTO LOCATION 'jdbc:postgresql://hgprecn-cn-5y**********-cn-hangzhou-internal.hologres.aliyuncs.com:80/test?ApplicationName=MaxCompute¤tSchema=public&useSSL=false&table=mc_2_holo/' STORED BY 'com.aliyun.odps.jdbc.JdbcStorageHandler' WITH SERDEPROPERTIES ( 'odps.properties.rolearn'='acs:ram::13**************:role/aliyunodpsholorole', 'mcfed.mapreduce.jdbc.driver.class'='org.postgresql.Driver', 'odps.federation.jdbc.target.db.type'='holo' );Verify the result: View the exported data in Hologres.
SELECT * FROM mc_2_holo;The following result is returned:
id name 4 Gegegegr 3 rgege
Billing
Command billing
The UNLOAD command itself is not billed.
Billing for query clauses: The query clause in an UNLOAD command scans data and uses computing resources to generate the exported data. Therefore, the query clause is billed as a standard SQL job.
Cost estimation
Storing structured data in OSS and storing data in MaxCompute each have their own advantages. To reduce costs when you export data from MaxCompute, perform a comprehensive cost estimation that includes direct storage costs, data volume changes, and subsequent query costs.
Direct storage costs
MaxCompute provides the Standard, Infrequent Access (IA), and Archive storage classes. All storage classes support zone-redundant storage. For more information about storage billing, see Storage fees.
OSS also provides Standard, IA, and Archive Storage classes, all of which support zone-redundant storage. The price of each storage class is the same as that in MaxCompute. OSS also provides Cold Archive and Deep Cold Archive storage classes that are less expensive but do not support zone-redundant storage. For more information, see OSS storage fees.
Data volume changes
Data imported into MaxCompute has a compression ratio of approximately 5:1. Billing is based on the amount of data after compression.
For data stored in OSS, you must specify and maintain the data format and compression method. For more information about the data formats and compression methods that are supported for exporting data from MaxCompute, see OSS foreign tables. If data is not compressed, the data volume expands after the data is exported to OSS.
Subsequent query costs
Data stored in MaxCompute can be queried at any time. However, if you use storage classes such as Infrequent Access or Archive, you must pay additional data access fees for reading data. If data is stored in non-standard OSS storage classes such as Infrequent Access, Archive, or Cold Archive, data retrieval fees are also incurred. If the data is deleted or retrieved before the minimum storage period required for the storage class is met, you may need to pay early deletion fees. For more information, see the pricing calculator.
References
To import data in CSV or other open-source formats from external storage into MaxCompute, see LOAD.