ALIYUN::SLS::OssExport

ALIYUN::SLS::OssExport类型用于日志服务数据传输管理,该服务提供了将日志库中的数据传输至OSS产品存储的功能。

语法

{
  "Type": "ALIYUN::SLS::OssExport",
  "Properties": {
    "Configuration": Map,
    "DisplayName": String,
    "ExportName": String,
    "ProjectName": String,
    "Description": String
  }
}

属性

属性名称

类型

必须

允许更新

描述

约束

Configuration

Map

导出任务的配置。

详情参考Configuration属性。

DisplayName

String

OSS导出的显示名称。

无。

ExportName

String

投递配置名称。

它只能包含小写字母、数字、破折号 - 和下划线 _。它必须以小写字母或数字开头和结尾,名称的长度必须为2到64个字符。

ProjectName

String

日志项目名称。

Description

String

导出任务的描述。

其长度可以是0到256个字符。

Configuration语法

"Configuration": {
  "ToTime": Integer,
  "Sink": Map,
  "Logstore": String,
  "FromTime": Integer,
  "RoleArn": String
}

Configuration属性

属性名称

类型

必须

允许更新

描述

约束

Logstore

String

项目的日志存储名称。

RoleArn

String

用于控制写入数据到OSS和读取日志存储中数据的权限。

示例:acs:ram::13234:role/aliyunlogdefaultrole。

Sink

Map

目标名称。

FromTime

Integer

时间范围的开始时间。

ToTime

Integer

时间范围的结束时间。

Sink语法

"Sink": {
  "BufferInterval": Integer,
  "ContentType": String,
  "ContentDetail": Map,
  "PathFormat": String,
  "Prefix": String,
  "PathFormatType": String,
  "RoleArn": String,
  "BufferSize": Integer,
  "TimeZone": String,
  "Suffix": String,
  "Endpoint": String,
  "DelaySeconds": Integer,
  "Bucket": String,
  "CompressionType": String
}

Sink属性

属性名称

类型

必须

允许更新

描述

约束

BufferInterval

Integer

投递时间。

数据投递的最大时间间隔。有效值:300至900。单位:秒。

BufferSize

Integer

投递大小。

该参数的值决定了投递并存储在一个对象中的原始日志数据的最大大小。单位:MB。如果您想要投递的日志数据的大小达到指定值,将自动创建一个投递任务。

Bucket

String

OSS 存储桶的名称。

ContentType

String

存储格式。

取值:

  • json

  • parquet

  • csv

  • orc

您可以选择将数据以 Parquet 格式存储以便进行数据分析。与存储在 CSV 或 JSON 格式的数据相比,以 Parquet 格式存储的数据可以以更高的效率进行扫描。当您以 Parquet 格式投递数据时,相比于 CSV 或 JSON 格式,会产生更多的费用。根据您的业务需求选择数据投递的存储格式。

RoleArn

String

用于控制写入数据到OSS和读取日志存储中数据的权限。

示例:acs:ram::13234:role/aliyunlogdefaultrole。

ContentDetail

Map

内容详情。

如果 ContentType=json,默认值为 {"EnableTag": false}。

CompressionType

String

OSS数据的压缩方法。

有效取值:

  • none

  • snappy

  • gzip

  • zstd

none 表示原始数据不进行压缩。 snappy、gzip 或 zstd 表示使用指定的算法压缩数据,这可以减少OSS存储桶的存储空间使用。不同的压缩方法基于不同的存储格式。

DelaySeconds

Integer

投递延迟。

数据投递之后的时间周期。有效值:900至63244800。单位:秒。

注意:请勿指定超过 Logstore 数据保留期限的延迟时间。我们建议您为延迟预留几天的缓冲期。否则,可能会发生数据丢失。

Endpoint

String

OSS 存储桶的Endpoint。

PathFormat

String

分区格式用于生成子目录。

子目录是基于投递时间动态生成的。默认的分区格式是 %Y/%m/%d/%H/%M。例如:2017/01/23/12/00。注意,分区格式不能以正斜杠(/)开头。有关如何与 E-MapReduce 的计算引擎(如 Hive 和 Impala)集成以查询和分析数据的更多信息。

Prefix

String

OSS 对象名称的前缀。

PathFormatType

String

分区格式类型。

取值:

  • time

Suffix

String

OSS 对象名称的后缀。

TimeZone

String

时区。

分区路径会根据时区的不同而有所变化。该值应在 -1200 到 +1400 之间。例如,+0800。

返回值

Fn::GetAtt

  • ProjectName:日志项目名称。

  • ExportName:投递配置名称。

示例

  • YAML格式

    ROSTemplateFormatVersion: '2015-09-01'
    Parameters:
      Configuration:
        AssociationPropertyMetadata:
          Parameters:
            ToTime:
              Type: Number
              Description:
                en: The end time of the time range.
              Required: false
              Default: 0
            Sink:
              AssociationPropertyMetadata:
                Parameters:
                  BufferInterval:
                    Type: Number
                    Description:
                      en: 'The shipping time. The maximum interval at which data is shipped. Valid values: 300 to 900. Unit: seconds.'
                    Required: true
                    MinValue: 300
                    MaxValue: 900
                  ContentType:
                    Type: String
                    Description:
                      en: The storage format. You can store data in the Parquet format for data analysis. Compared with data stored in the CSV or JSON format, data stored in the Parquet format can be scanned at a higher efficiency.You are charged more fees when you ship data in the Parquet format than in the CSV or JSON format. Select a storage format for data shipping based on your business requirements.
                    AllowedValues:
                      - json
                      - parquet
                      - csv
                      - orc
                    Required: true
                  ContentDetail:
                    Type: Json
                    Description:
                      en: 'The content detail. If ContentType=json, the default value is {"EnableTag": false}.'
                    Required: false
                  PathFormat:
                    Type: String
                    Description:
                      en: 'The partition format is used to generate subdirectories. A subdirectory is dynamically generated based on the shipping time. The default partition format is %Y/%m/%d/%H/%M. Example: 2017/01/23/12/00. Note that the partition format cannot start with a forward slash (/). For more information about how to integrate with the compute engines of E-MapReduce such as Hive and Impala to query and analyze data'
                    Required: false
                    Default: '%Y/%m/%d/%H/%M'
                  Prefix:
                    Type: String
                    Description:
                      en: The prefix of the OSS object name.
                    Required: false
                  PathFormatType:
                    Type: String
                    Description:
                      en: The partition format type. Only support time.
                    AllowedValues:
                      - time
                    Required: false
                    Default: time
                  RoleArn:
                    Type: String
                    Description:
                      en: 'Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole.'
                    Required: true
                  BufferSize:
                    Type: Number
                    Description:
                      en: 'The shipping size. The value of this parameter determines the maximum size of raw log data that is shipped and stored in an object. Unit: MB. If the size of log data that you want to ship reaches the specified value, a shipping job is automatically created.'
                    Required: true
                    MinValue: 5
                    MaxValue: 256
                  TimeZone:
                    Type: String
                    Description:
                      en: The time zone. Partition paths vary based on time zones. The value should be -1200 to +1400. For example, +0800.
                    Required: false
                  Suffix:
                    Type: String
                    Description:
                      en: The suffix of the OSS object name.
                    Required: false
                  Endpoint:
                    Type: String
                    Description:
                      en: The endpoint of the OSS bucket.
                    Required: false
                  DelaySeconds:
                    Type: Number
                    Description:
                      en: |-
                        The shipping latency. The period of time after which data is shipped. Valid values: 900 to 63244800. Unit: seconds.
                        Note: Do not specify a latency that exceeds the data retention period of the Logstore. We recommend that you reserve a buffer period of a few days for the latency. Otherwise, data loss may occur.
                    Required: false
                    MinValue: 900
                    MaxValue: 63244800
                  Bucket:
                    Type: String
                    Description:
                      en: The name of the OSS bucket.
                    Required: true
                  CompressionType:
                    Type: String
                    Description:
                      en: 'The compression method of OSS data. Valid values: none, snappy, gzip, and zstd. The value none indicates that raw data is not compressed. The value snappy, gzip, or zstd indicates that a specified algorithm is used to compress data, which can reduce the storage space usage of OSS buckets. The compression methods vary based on storage formats.'
                    AllowedValues:
                      - none
                      - snappy
                      - gzip
                      - zstd
                    Required: false
              Type: Json
              Description:
                en: The sink name.
              Required: true
            Logstore:
              Type: String
              Description:
                en: The logstore name of the project.
              Required: true
            FromTime:
              Type: Number
              Description:
                en: The start time of the time range.
              Required: false
              Default: 0
            RoleArn:
              Type: String
              Description:
                en: 'Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole.'
              Required: true
        Type: Json
        Description:
          en: The configuration of the export job.
        Required: true
      ProjectName:
        Type: String
        Description:
          en: The project name of SLS.
        Required: true
        AllowedPattern: ^[a-zA-Z0-9_-]+$
        MinLength: 3
        MaxLength: 63
      DisplayName:
        Type: String
        Description:
          en: The display name of the export job. It must be 4 to 100 characters in length.
        Required: true
        MinLength: 4
        MaxLength: 100
      ExportName:
        Type: String
        Description:
          en: The name of the export job. This value should be unique. It must be 2 to 64 characters in length and can contain lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or a digit.
        Required: true
        AllowedPattern: ^[a-zA-Z0-9_-]+$
        MinLength: 2
        MaxLength: 64
    Resources:
      OssExport:
        Type: ALIYUN::SLS::OssExport
        Properties:
          Configuration:
            Ref: Configuration
          ProjectName:
            Ref: ProjectName
          DisplayName:
            Ref: DisplayName
          ExportName:
            Ref: ExportName
    Outputs:
      ProjectName:
        Description: The project name of SLS.
        Value:
          Fn::GetAtt:
            - OssExport
            - ProjectName
      ExportName:
        Description: The name of the export job.
        Value:
          Fn::GetAtt:
            - OssExport
            - ExportName
    
  • JSON格式

    {
      "ROSTemplateFormatVersion": "2015-09-01",
      "Parameters": {
        "Configuration": {
          "AssociationPropertyMetadata": {
            "Parameters": {
              "ToTime": {
                "Type": "Number",
                "Description": {
                  "en": "The end time of the time range."
                },
                "Required": false,
                "Default": 0
              },
              "Sink": {
                "AssociationPropertyMetadata": {
                  "Parameters": {
                    "BufferInterval": {
                      "Type": "Number",
                      "Description": {
                        "en": "The shipping time. The maximum interval at which data is shipped. Valid values: 300 to 900. Unit: seconds."
                      },
                      "Required": true,
                      "MinValue": 300,
                      "MaxValue": 900
                    },
                    "ContentType": {
                      "Type": "String",
                      "Description": {
                        "en": "The storage format. You can store data in the Parquet format for data analysis. Compared with data stored in the CSV or JSON format, data stored in the Parquet format can be scanned at a higher efficiency.You are charged more fees when you ship data in the Parquet format than in the CSV or JSON format. Select a storage format for data shipping based on your business requirements."
                      },
                      "AllowedValues": [
                        "json",
                        "parquet",
                        "csv",
                        "orc"
                      ],
                      "Required": true
                    },
                    "ContentDetail": {
                      "Type": "Json",
                      "Description": {
                        "en": "The content detail. If ContentType=json, the default value is {\"EnableTag\": false}."
                      },
                      "Required": false
                    },
                    "PathFormat": {
                      "Type": "String",
                      "Description": {
                        "en": "The partition format is used to generate subdirectories. A subdirectory is dynamically generated based on the shipping time. The default partition format is %Y/%m/%d/%H/%M. Example: 2017/01/23/12/00. Note that the partition format cannot start with a forward slash (/). For more information about how to integrate with the compute engines of E-MapReduce such as Hive and Impala to query and analyze data"
                      },
                      "Required": false,
                      "Default": "%Y/%m/%d/%H/%M"
                    },
                    "Prefix": {
                      "Type": "String",
                      "Description": {
                        "en": "The prefix of the OSS object name."
                      },
                      "Required": false
                    },
                    "PathFormatType": {
                      "Type": "String",
                      "Description": {
                        "en": "The partition format type. Only support time."
                      },
                      "AllowedValues": [
                        "time"
                      ],
                      "Required": false,
                      "Default": "time"
                    },
                    "RoleArn": {
                      "Type": "String",
                      "Description": {
                        "en": "Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole."
                      },
                      "Required": true
                    },
                    "BufferSize": {
                      "Type": "Number",
                      "Description": {
                        "en": "The shipping size. The value of this parameter determines the maximum size of raw log data that is shipped and stored in an object. Unit: MB. If the size of log data that you want to ship reaches the specified value, a shipping job is automatically created."
                      },
                      "Required": true,
                      "MinValue": 5,
                      "MaxValue": 256
                    },
                    "TimeZone": {
                      "Type": "String",
                      "Description": {
                        "en": "The time zone. Partition paths vary based on time zones. The value should be -1200 to +1400. For example, +0800."
                      },
                      "Required": false
                    },
                    "Suffix": {
                      "Type": "String",
                      "Description": {
                        "en": "The suffix of the OSS object name."
                      },
                      "Required": false
                    },
                    "Endpoint": {
                      "Type": "String",
                      "Description": {
                        "en": "The endpoint of the OSS bucket."
                      },
                      "Required": false
                    },
                    "DelaySeconds": {
                      "Type": "Number",
                      "Description": {
                        "en": "The shipping latency. The period of time after which data is shipped. Valid values: 900 to 63244800. Unit: seconds.\nNote: Do not specify a latency that exceeds the data retention period of the Logstore. We recommend that you reserve a buffer period of a few days for the latency. Otherwise, data loss may occur."
                      },
                      "Required": false,
                      "MinValue": 900,
                      "MaxValue": 63244800
                    },
                    "Bucket": {
                      "Type": "String",
                      "Description": {
                        "en": "The name of the OSS bucket."
                      },
                      "Required": true
                    },
                    "CompressionType": {
                      "Type": "String",
                      "Description": {
                        "en": "The compression method of OSS data. Valid values: none, snappy, gzip, and zstd. The value none indicates that raw data is not compressed. The value snappy, gzip, or zstd indicates that a specified algorithm is used to compress data, which can reduce the storage space usage of OSS buckets. The compression methods vary based on storage formats."
                      },
                      "AllowedValues": [
                        "none",
                        "snappy",
                        "gzip",
                        "zstd"
                      ],
                      "Required": false
                    }
                  }
                },
                "Type": "Json",
                "Description": {
                  "en": "The sink name."
                },
                "Required": true
              },
              "Logstore": {
                "Type": "String",
                "Description": {
                  "en": "The logstore name of the project."
                },
                "Required": true
              },
              "FromTime": {
                "Type": "Number",
                "Description": {
                  "en": "The start time of the time range."
                },
                "Required": false,
                "Default": 0
              },
              "RoleArn": {
                "Type": "String",
                "Description": {
                  "en": "Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole."
                },
                "Required": true
              }
            }
          },
          "Type": "Json",
          "Description": {
            "en": "The configuration of the export job."
          },
          "Required": true
        },
        "ProjectName": {
          "Type": "String",
          "Description": {
            "en": "The project name of SLS."
          },
          "Required": true,
          "AllowedPattern": "^[a-zA-Z0-9_-]+$",
          "MinLength": 3,
          "MaxLength": 63
        },
        "DisplayName": {
          "Type": "String",
          "Description": {
            "en": "The display name of the export job. It must be 4 to 100 characters in length."
          },
          "Required": true,
          "MinLength": 4,
          "MaxLength": 100
        },
        "ExportName": {
          "Type": "String",
          "Description": {
            "en": "The name of the export job. This value should be unique. It must be 2 to 64 characters in length and can contain lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or a digit."
          },
          "Required": true,
          "AllowedPattern": "^[a-zA-Z0-9_-]+$",
          "MinLength": 2,
          "MaxLength": 64
        }
      },
      "Resources": {
        "OssExport": {
          "Type": "ALIYUN::SLS::OssExport",
          "Properties": {
            "Configuration": {
              "Ref": "Configuration"
            },
            "ProjectName": {
              "Ref": "ProjectName"
            },
            "DisplayName": {
              "Ref": "DisplayName"
            },
            "ExportName": {
              "Ref": "ExportName"
            }
          }
        }
      },
      "Outputs": {
        "ProjectName": {
          "Description": "The project name of SLS.",
          "Value": {
            "Fn::GetAtt": [
              "OssExport",
              "ProjectName"
            ]
          }
        },
        "ExportName": {
          "Description": "The name of the export job.",
          "Value": {
            "Fn::GetAtt": [
              "OssExport",
              "ExportName"
            ]
          }
        }
      }
    }