Simple Log Service can store logs shipped to Object Storage Service (OSS) in multiple formats. This topic covers the Parquet storage format.
Parameters
When you create an OSS data shipping job (new version), set Storage Format to Parquet. In the Parquet Fields section, set the Key Name and Type for each field. For example, add a field with a Key Name of __topic__ and a Type of string, and another field with a Key Name of concent and a Type of string. Click the + button to add a new field row.
The following table describes the Parquet-specific parameters.
|
Parameter |
Description |
|
Key Name |
The name of the log field to ship to OSS. You can find available log fields on the Raw Logs tab of a Logstore. We recommend that you add log fields one by one. The fields are stored in the Parquet file in the order you add them, and the field names become the column names in the Parquet file. The log fields that you can ship to OSS include the reserved fields, such as __time__, __topic__, and __source__. For more information, see Reserved fields. Column values in a Parquet file are null in the following scenarios:
Note
|
|
Type |
The Parquet format supports the following data types: string, boolean, int32, int64, float, and double. String fields are stored as the byte_array type in Parquet, and the |
Sample URLs of OSS objects
After logs are shipped to OSS, they are stored in OSS buckets. The following table shows sample object URLs.
-
If you specify an object suffix when you create a data shipping job, the OSS objects use the suffix.
-
If you do not specify an object suffix when you create a data shipping job, the OSS objects use the suffix that is generated based on the compression type.
|
Compression type |
Object suffix |
Sample URL |
Description |
|
Not compressed |
If you specify an object suffix, the specified suffix takes effect. Example: .suffix. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix |
You can download the OSS object to your computer and consume its data. For more information, see Data consumption. |
|
If you do not specify an object suffix, the suffix .parquet is used. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.parquet |
||
|
Snappy |
If you specify an object suffix, the specified suffix takes effect. Example: .suffix. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix |
|
|
If you do not specify an object suffix, the suffix .snappy.parquet is used. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.snappy.parquet |
||
|
Gzip |
If you specify an object suffix, the specified suffix takes effect. Example: .suffix. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix |
|
|
If you do not specify an object suffix, the suffix .gz.parquet is used. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.gz.parquet |
||
|
Zstandard |
If you specify an object suffix, the specified suffix takes effect. Example: .suffix. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix |
|
|
If you do not specify an object suffix, the suffix .zst.parquet is used. |
oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.zst.parquet |
Data consumption
-
You can consume data that is shipped to OSS by using E-MapReduce, Spark, or Hive. For more information, see LanguageManual DDL.
-
You can also consume data by using inspection tools.
You can use the parquet-tools utility for Python to inspect Parquet files, view file details, and read data. Install the utility by running the following command or by using a different method:
pip3 install parquet-tools-
View the data of columns in a Parquet file
-
Command
View the data of the remote_addr and body_bytes_sent columns.
parquet-tools show -n 2 -c remote_addr,body_bytes_sent 44_1693464263000000000_2288ff590970d092.parquet -
Response
+----------------+-------------------+ | remote_addr | body_bytes_sent | |----------------+-------------------| | 61.243.1.63 | b'1904' | | 112.235.74.182 | b'4996' | +----------------+-------------------+
-
-
View the content in a Parquet file (Convert the file into the CSV format.)
-
Command
parquet-tools csv -n 2 44_1693464263000000000_2288ff590970d092.parquet -
Response
remote_addr,body_bytes_sent,time_local,request_method,request_uri,http_user_agent,remote_user,request_time,request_length,http_referer,host,http_x_forwarded_for,upstream_response_time,status b'61.**.**.63',b'1904',b'31/Aug/2023:06:44:01',b'GET',b'/request/path-0/file-7',"b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.801.0 Safari/535.1'",b'uh2z',b'49',b'4082',b'www.kwm.mock.com',b'www.ap.mock.com',b'222.**.**.161',b'2.63',b'200' b'112.**.**.182',b'4996',b'31/Aug/2023:06:44:01',b'GET',b'/request/path-1/file-5',b'Mozilla/5.0 (Windows NT 6.1; de;rv:12.0) Gecko/20120403211507 Firefox/12.0',b'tix',b'71',b'1862',b'www.gx.mock.com',b'www.da.mock.com',b'36.**.**.237',b'2.43',b'200'
-
-
View the details of a Parquet file
-
Command
parquet-tools inspect 44_1693464263000000000_2288ff590970d092.parquet -
Response
############ file meta data ############ created_by: SLS version 1 num_columns: 14 num_rows: 4661 num_row_groups: 1 format_version: 1.0 serialized_size: 2345 ############ Columns ############ remote_addr body_bytes_sent time_local request_method request_uri http_user_agent remote_user request_time request_length http_referer host http_x_forwarded_for upstream_response_time status ############ Column(remote_addr) ############ name: remote_addr path: remote_addr max_definition_level: 1 max_repetition_level: 0 physical_type: BYTE_ARRAY logical_type: None converted_type (legacy): NONE compression: UNCOMPRESSED (space_saved: 0%) ......
-
-