MaxCompute 新增内置标量函数,用于直接在 SQL 中解析归档文件(archive)的二进制数据。无需先将归档文件解压到磁盘,即可读取zip、jar 、tar、tar.gz、tar.bz2、tar.xz、tar.zst 等格式内的文件元数据与内容。
本文所有 SQL 示例均为可直接在 MaxCompute 中执行的完整语句,二进制入参需使用完整的 x'...' hex 字面量。函数总览
以下两个函数为标量函数(Scalar Function),非表值函数,返回值为复杂类型(ARRAY / MAP),可直接在 SELECT 子句中使用。
函数名 | 输入 | 返回类型 | 用途 |
|
|
| 列出归档内所有条目的元信息 |
|
|
| 提取归档内文件名到文件内容的映射 |
注意事项
输入类型
两个函数仅接受
BINARY输入;若数据为 STRING,需先做类型转换。内容解码:
GET_ARCHIVE_CONTENT的 value 以 STRING 返回,二进制内容按文本解码;非文本文件(图片、class 等)可能产生乱码。
内存占用
归档内容会被完整读入内存解析,超大归档可能占用较多资源,建议控制单条记录大小。
格式自动识别
无需手动指定格式,函数依据内容字节自动判断。
pattern 仅作用于文件名
通配符匹配的是归档内条目的
file_path,不是文件内容。目录条目
jar / zip 中的目录条目(如
META-INF/)在LIST_ARCHIVE_METADATA中size为 0,在GET_ARCHIVE_CONTENT中 value 为空字符串。mtime 可能缺失
tar.gz 格式的
mtime可能为0。
支持的归档和压缩格式
函数会根据二进制内容自动识别格式,无需指定。
支持格式
格式 | 魔数(文件头) | 容器 | 压缩算法 |
zip (stored) |
| ZIP | stored (method 0) |
zip (deflate) |
| ZIP | deflate (method 8) |
jar |
| ZIP | 通常 stored |
tar | 无统一魔数 | TAR | 无压缩 |
tar.gz |
| TAR | gzip |
tar.bz2 |
| TAR | bzip2 |
tar.xz |
| TAR | xz |
tar.zst |
| TAR | zstd |
格式识别原理
zip / jar:检测
PK头(0x504b0304)。支持 zip 条目的 stored(method 0)和 deflate(method 8)两种压缩方式。jar 本质是 zip,会额外包含META-INF/目录条目。tar.gz / tar.bz2 / tar.xz / tar.zst:先检测外层压缩头(gzip
1f8b、bzip2425a68、xzfd377a58、zstd28b52ffd),解压后再解析内层 tar。tar:无外层压缩,按 tar 的 512 字节块结构直接解析。
LIST_ARCHIVE_METADATA
该函数用于列出归档内所有条目的元信息。
语法格式
LIST_ARCHIVE_METADATA(<binary_input>)参数说明
binary_input参数说明:必填。BINARY类型。归档文件的原始字节内容。
返回值说明
返回一个ARRAY数组ARRAY<STRUCT<file_path:STRING, size:BIGINT, mtime:BIGINT>>,数组每个元素对应归档内的一个条目(含目录条目),字段含义:
字段 | 类型 | 说明 |
|
| 条目在归档内的相对路径(如 |
|
| 条目未压缩的字节大小;目录条目为 |
|
| 条目的修改时间,Unix 时间戳(秒);部分格式可能为 |
数组顺序:按归档内条目的存储顺序返回。
使用示例
示例一:读Object Table表时使用归档函数
上传测试数据到OSS上test_01.zip。
创建Object Table读取元信息。
-- 创建Object Table SET odps.namespace.schema=true; CREATE OBJECT TABLE ot_test_01 WITH serdeproperties ( 'odps.properties.rolearn'='acs:ram::<uid>:role/aliyunodpsdefaultrole') LOCATION 'oss://oss-<region>-internal.aliyuncs.com/<oss-path>/' ; -- 刷新元数据 ALTER TABLE ot_test_01 REFRESH METADATA; -- 查询Object Table SET odps.namespace.schema=true; SELECT * FROM ot_test_01; -- 返回结果如下: +------+------------+------+---------------+---------------+------+--------------+------------+--------------------+ | key | size | type | last_modified | storage_class | etag | restore_info | owner_id | owner_display_name | +------+------------+------+---------------+---------------+------+--------------+------------+--------------------+ | test_01.zip | 7246 | Normal | 2026-08-31 06:40:38 | Standard | 337E*****3101C02F1 | NONE| 122***014 | 122***014 | +------+------------+------+---------------+---------------+------+--------------+------------+--------------------+使用归档函数:列出归档内所有条目的元信息。
SET odps.namespace.schema=true; SELECT LIST_ARCHIVE_METADATA(data) FROM ( SELECT GET_DATA_FROM_OSS('<project>.<schema>.ot_test_01', key) as data FROM ot_test_01 ); -- 返回结果如下: +------+ | _c0 | +------+ | [{file_path:test_01/, size:0, mtime:1786418145}, {file_path:test_01/hello.txt, size:13, mtime:1786418039}, {file_path:__MACOSX/test_01/._hello.txt, size:5106, mtime:1786418039}, {file_path:test_01/data.txt, size:13, mtime:1786418061}, {file_path:__MACOSX/test_01/._data.txt, size:5122, mtime:1786418061}] | +------+
示例二:其他格式结果示例
zip格式
zip格式(含 hello.txt = "hello world"、data.txt = "test data 123")。
SELECT LIST_ARCHIVE_METADATA(x'504b030414000000000000006f5785114a0d0b0000000b0000000900000068656c6c6f2e74787468656c6c6f20776f726c64504b030414000000000000006f57ff35e1450d0000000d00000008000000646174612e74787474657374206461746120313233504b0102140314000000000000006f5785114a0d0b0000000b00000009000000000000000000000080010000000068656c6c6f2e747874504b0102140314000000000000006f57ff35e1450d0000000d000000080000000000000000000000800132000000646174612e747874504b050600000000020002006d000000650000000000');
-- 返回结果如下:
[{file_path:hello.txt, size:11, mtime:1699977600}, {file_path:data.txt, size:13, mtime:1699977600}]jar / stored格式
jar / stored格式(含 META-INF/、META-INF/MANIFEST.MF、hello.txt = "hello jar")。
SELECT LIST_ARCHIVE_METADATA(x'504b03040a0000080000f4a1bc5c000000000000000000000000090004004d4554412d494e462ffeca0000504b03040a0000080000f4a1bc5c6388d7f13a0000003a000000140000004d4554412d494e462f4d414e49464553542e4d464d616e69666573742d56657273696f6e3a20312e300d0a437265617465642d42793a20312e382e305f3334352028416c6962616261290d0a0d0a504b03040a000008000014a0bc5cdf3b649609000000090000000900000068656c6c6f2e74787468656c6c6f206a6172504b01020a000a0000080000f4a1bc5c0000000000000000000000000900040000000000000000000000000000004d4554412d494e462ffeca0000504b01020a000a0000080000f4a1bc5c6388d7f13a0000003a00000014000000000000000000000000002b0000004d4554412d494e462f4d414e49464553542e4d46504b01020a000a000008000014a0bc5cdf3b6496090000000900000009000000000000000000000000009700000068656c6c6f2e747874504b05060000000003000300b4000000c70000000000');
-- 返回结果如下:
[{file_path:META-INF/, size:0, mtime:1779970540}, {file_path:META-INF/MANIFEST.MF, size:58, mtime:1779970540}, {file_path:hello.txt, size:9, mtime:1779969640}]
-- 注意:jar 包含 META-INF/ 目录条目,其 size 为 0。tar.gz格式
tar.gz格式(含 hello.txt = "hello world"、data.csv = "test data 123")。
SELECT LIST_ARCHIVE_METADATA(x'1f8b08000000000002ffedd3410a83400c85e1394a4e20998edaf30c5570312038b1edf1eb74d3e2da5ac4ffdb2464179237f4298d953dcdfd8e2edaba7ed7c5baaafaf0e9cbdc7bd5e044dd0ee66c7112d964c9efe50e6228f797c738a5cee17cba68b1bae5bbfb6bfe9b55feb569afe47f0fd66793f204e22f81380000000000000000000000001cce0b293b365100280000');
-- 返回结果如下。
[{file_path:hello.txt, size:11, mtime:0}, {file_path:data.csv, size:13, mtime:0}]
-- 注意:tar.gz 的 mtime 可能为 0(取决于内层 TAR header 的时间字段)。tar格式
tar格式(含 hello.txt = "hello world")。
SELECT LIST_ARCHIVE_METADATA(x'68656c6c6f2e747874000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003030303036343400303030303030300030303030303030003030303030303030303133003134353234373730343030003030373634350020300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000075737461720030300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000068656c6c6f20776f726c64000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000');
-- 返回结果如下。
[{file_path:hello.txt, size:11, mtime:1700000000}]GET_ARCHIVE_CONTENT
该函数用于提取归档内文件名到文件内容的映射。
语法格式
-- 不带 pattern:返回归档内所有文件的内容
GET_ARCHIVE_CONTENT(<binary_input>)
-- 带 pattern:仅返回文件名匹配的文件内容
GET_ARCHIVE_CONTENT(<binary_input>, <pattern>)参数说明
参数 | 类型 | 必填 | 说明 |
|
| 是 | 归档文件的原始字节内容。 |
|
| 否 | glob 风格的文件名匹配模式;省略则返回全部。 |
返回值说明
返回MAP类型MAP<STRING, STRING>,key 为文件名(file_path),value 为文件内容(以 STRING 解码)。
目录条目(如
META-INF/)的 value 为空字符串。Map 按文件名字母序排列。
Pattern 匹配规则(glob 通配符)
通配符 | 含义 | 示例 |
| 匹配任意数量字符(含空) |
|
| 匹配单个字符 |
|
| 普通字面字符 |
|
其他字符 | 字面匹配 |
|
pattern 只匹配文件名,不匹配文件内容。匹配不到任何文件时返回空 Map {}。
使用示例
示例一:读Object Table表时使用归档函数
上传测试数据到OSS上test_01.zip。
创建Object Table读取元信息。
-- 创建Object Table SET odps.namespace.schema=true; CREATE OBJECT TABLE ot_test_01 WITH serdeproperties ( 'odps.properties.rolearn'='acs:ram::<uid>:role/aliyunodpsdefaultrole') LOCATION 'oss://oss-<region>-internal.aliyuncs.com/<oss-path>/' ; -- 刷新元数据 ALTER TABLE ot_test_01 REFRESH METADATA; -- 查询Object Table SET odps.namespace.schema=true; SELECT * FROM ot_test_01; -- 返回结果如下: +------+------------+------+---------------+---------------+------+--------------+------------+--------------------+ | key | size | type | last_modified | storage_class | etag | restore_info | owner_id | owner_display_name | +------+------------+------+---------------+---------------+------+--------------+------------+--------------------+ | test_01.zip | 7246 | Normal | 2026-08-31 06:40:38 | Standard | 337E*****3101C02F1 | NONE| 122***014 | 122***014 | +------+------------+------+---------------+---------------+------+--------------+------------+--------------------+使用归档函数:提取归档内文件名到文件内容的映射。
SET odps.namespace.schema=true; SELECT GET_ARCHIVE_CONTENT(data,'test_01/hello.*') FROM ( SELECT GET_DATA_FROM_OSS('yyy_test_dev.default.ot_test_01', key) as data FROM ot_test_01 ); -- 返回结果如下: +------+ | _c0 | +------+ | {test_01/hello.txt:hello=20world=0D=0A} | +------+
示例二:其他格式结果示例
不带 pattern(zip,全部文件)
SELECT GET_ARCHIVE_CONTENT(x'504b030414000000000000006f5785114a0d0b0000000b0000000900000068656c6c6f2e74787468656c6c6f20776f726c64504b030414000000000000006f57ff35e1450d0000000d00000008000000646174612e74787474657374206461746120313233504b0102140314000000000000006f5785114a0d0b0000000b00000009000000000000000000000080010000000068656c6c6f2e747874504b0102140314000000000000006f57ff35e1450d0000000d000000080000000000000000000000800132000000646174612e747874504b050600000000020002006d000000650000000000');
-- 预期返回结果。
{data.txt:test data 123, hello.txt:hello world}
-- 实际返回结果。
{
hello.txt:"hello=20world"
data.txt:"test=20data=20123"
}带 *.txt(zip,只取 .txt)
SELECT GET_ARCHIVE_CONTENT(x'504b030414000000000000006f5785114a0d0b0000000b0000000900000068656c6c6f2e74787468656c6c6f20776f726c64504b030414000000000000006f57ff35e1450d0000000d00000008000000646174612e74787474657374206461746120313233504b0102140314000000000000006f5785114a0d0b0000000b00000009000000000000000000000080010000000068656c6c6f2e747874504b0102140314000000000000006f57ff35e1450d0000000d000000080000000000000000000000800132000000646174612e747874504b050600000000020002006d000000650000000000', '*.txt');
-- 预期返回结果。
{data.txt:test data 123, hello.txt:hello world}
-- 实际返回结果
{
data.txt:"test=20data=20123"
hello.txt:"hello=20world"
}带 hello.*(zip,只取 hello 开头)
SELECT GET_ARCHIVE_CONTENT(x'504b030414000000000000006f5785114a0d0b0000000b0000000900000068656c6c6f2e74787468656c6c6f20776f726c64504b030414000000000000006f57ff35e1450d0000000d00000008000000646174612e74787474657374206461746120313233504b0102140314000000000000006f5785114a0d0b0000000b00000009000000000000000000000080010000000068656c6c6f2e747874504b0102140314000000000000006f57ff35e1450d0000000d000000080000000000000000000000800132000000646174612e747874504b050600000000020002006d000000650000000000', 'hello.*');
-- 预期返回结果。
{hello.txt:hello world}
-- 实际返回结果
{
hello.txt:"hello=20world"
}jar格式带 *.txt
jar格式带 *.txt(过滤掉 MANIFEST.MF 和目录)
SELECT GET_ARCHIVE_CONTENT(x'504b03040a0000080000f4a1bc5c000000000000000000000000090004004d4554412d494e462ffeca0000504b03040a0000080000f4a1bc5c6388d7f13a0000003a000000140000004d4554412d494e462f4d414e49464553542e4d464d616e69666573742d56657273696f6e3a20312e300d0a437265617465642d42793a20312e382e305f3334352028416c6962616261290d0a0d0a504b03040a000008000014a0bc5cdf3b649609000000090000000900000068656c6c6f2e74787468656c6c6f206a6172504b01020a000a0000080000f4a1bc5c0000000000000000000000000900040000000000000000000000000000004d4554412d494e462ffeca0000504b01020a000a0000080000f4a1bc5c6388d7f13a0000003a00000014000000000000000000000000002b0000004d4554412d494e462f4d414e49464553542e4d46504b01020a000a000008000014a0bc5cdf3b6496090000000900000009000000000000000000000000009700000068656c6c6f2e747874504b05060000000003000300b4000000c70000000000', '*.txt');
-- 预期返回结果。
{hello.txt:hello jar}
-- 实际返回结果。
{
hello.txt:"hello=20jar"
}tar.gz 带 *.csv
tar.gz 带 *.csv(按扩展名过滤)。
SELECT GET_ARCHIVE_CONTENT(x'1f8b08000000000002ffedd3410a83400c85e1394a4e20998edaf30c5570312038b1edf1eb74d3e2da5ac4ffdb2464179237f4298d953dcdfd8e2edaba7ed7c5baaafaf0e9cbdc7bd5e044dd0ee66c7112d964c9efe50e6228f797c738a5cee17cba68b1bae5bbfb6bfe9b55feb569afe47f0fd66793f204e22f81380000000000000000000000001cce0b293b365100280000', '*.csv');
-- 预期返回结果。
{data.csv:test data 123}
-- 实际返回结果。
{
data.csv:"test=20data=20123"
}tar 不带 pattern
SELECT GET_ARCHIVE_CONTENT(x'68656c6c6f2e747874000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003030303036343400303030303030300030303030303030003030303030303030303133003134353234373730343030003030373634350020300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000075737461720030300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000068656c6c6f20776f726c64000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000');
-- 预期返回结果。
{hello.txt:hello world}
-- 实际返回结果。
{
hello.txt:"hello=20world"
}异常与边界处理
LIST_ARCHIVE_METADATA和GET_ARCHIVE_CONTENT两个函数不会抛出异常,而是以优雅降级方式返回 NULL 或空集合:
场景 | 输入 | LIST_ARCHIVE_METADATA 返回 | GET_ARCHIVE_CONTENT 返回 |
非法归档数据 |
|
|
|
NULL 输入 |
|
|
|
空二进制 |
|
|
|
pattern 不匹配任何文件 |
| — |
|
pattern 为 NULL |
| — |
|
区别:
非法数据(无法解析)返回
NULL;空输入(
x'')返回空集合。原因是空二进制是“合法但无内容”,而deadbeef是“格式无法识别”。
可直接执行的边界示例
-- 非法数据 → NULL
SELECT LIST_ARCHIVE_METADATA(x'deadbeef'); -- \N
SELECT GET_ARCHIVE_CONTENT(x'deadbeef'); -- \N
-- NULL 输入 → NULL
SELECT LIST_ARCHIVE_METADATA(NULL); -- \N
SELECT GET_ARCHIVE_CONTENT(NULL); -- \N
-- 空二进制 → 空集合
SELECT LIST_ARCHIVE_METADATA(x''); -- []
SELECT GET_ARCHIVE_CONTENT(x''); -- {}
-- pattern 不匹配 → 空 Map
SELECT GET_ARCHIVE_CONTENT(x'504b030414000000000000006f5785114a0d0b0000000b0000000900000068656c6c6f2e74787468656c6c6f20776f726c64504b030414000000000000006f57ff35e1450d0000000d00000008000000646174612e74787474657374206461746120313233504b0102140314000000000000006f5785114a0d0b0000000b00000009000000000000000000000080010000000068656c6c6f2e747874504b0102140314000000000000006f57ff35e1450d0000000d000000080000000000000000000000800132000000646174612e747874504b050600000000020002006d000000650000000000', 'nonexistent.*'); -- {}
-- pattern 为 NULL → 空 Map
SELECT GET_ARCHIVE_CONTENT(x'504b030414000000000000006f5785114a0d0b0000000b0000000900000068656c6c6f2e74787468656c6c6f20776f726c64504b030414000000000000006f57ff35e1450d0000000d00000008000000646174612e74787474657374206461746120313233504b0102140314000000000000006f5785114a0d0b0000000b00000009000000000000000000000080010000000068656c6c6f2e747874504b0102140314000000000000006f57ff35e1450d0000000d000000080000000000000000000000800132000000646174612e747874504b050600000000020002006d000000650000000000', NULL); -- {}复杂类型结果的展开用法
由于返回值类型是 ARRAY<STRUCT> 和 MAP<STRING,STRING>,可用 LATERAL VIEW EXPLODE 展开成多行。以下示例假设存在表 archives(id BIGINT, content BINARY),请按需替换为实际表名与列名。
展开 LIST_ARCHIVE_METADATA 结果(逐行展示文件元数据)
-- 假设表 archives(id BIGINT, content BINARY) 存储归档数据
SELECT id, meta.file_path, meta.size, meta.mtime
FROM archives
LATERAL VIEW EXPLODE(LIST_ARCHIVE_METADATA(content)) t AS meta
WHERE id = 1;展开 GET_ARCHIVE_CONTENT 结果(逐行展示文件名与内容)
SELECT id, k AS file_name, v AS file_content
FROM archives
LATERAL VIEW EXPLODE(GET_ARCHIVE_CONTENT(content, '*.txt')) t AS k, v
WHERE id = 1;按文件大小过滤
SELECT id, meta.file_path, meta.size
FROM archives
LATERAL VIEW EXPLODE(LIST_ARCHIVE_METADATA(content)) t AS meta
WHERE id = 1 AND meta.size > 10;完整示例速查表
以 zip 包(含 hello.txt = "hello world"、data.txt = "test data 123")为例,hex 为 x'504b030414000000000000006f5785114a0d0b0000000b0000000900000068656c6c6f2e74787468656c6c6f20776f726c64504b030414000000000000006f57ff35e1450d0000000d00000008000000646174612e74787474657374206461746120313233504b0102140314000000000000006f5785114a0d0b0000000b00000009000000000000000000000080010000000068656c6c6f2e747874504b0102140314000000000000006f57ff35e1450d0000000d000000080000000000000000000000800132000000646174612e747874504b050600000000020002006d000000650000000000':
目标 | SQL(省略 hex,完整版见上) | 返回 |
列出文件元数据 | SELECT LIST_ARCHIVE_METADATA(<zip_bin>); | [{file_path:hello.txt, size:11, mtime:1699977600}, {file_path:data.txt, size:13, mtime:1699977600}] |
取出全部文件内容 | SELECT GET_ARCHIVE_CONTENT(<zip_bin>); | {data.txt:test data 123, hello.txt:hello world} |
只取 .txt 文件 | SELECT GET_ARCHIVE_CONTENT(<zip_bin>, '*.txt'); | {data.txt:test data 123, hello.txt:hello world} |
只取 hello 开头的文件 | SELECT GET_ARCHIVE_CONTENT(<zip_bin>, 'hello.*'); | {hello.txt:hello world} |
取不到匹配 | SELECT GET_ARCHIVE_CONTENT(<zip_bin>, 'nonexistent.*'); | {} |