日志服务支持将加工后的数据跨账号发送到多个目标Logstore。

数据分发主要包含两种方法:跨账号多目标Logstore分发和动态目标Logstore的分发,其特点如下表所示。
方法 优点 缺点
配置多个目标Logstore 支持不同账号的目标Logstore。 多个目标时,配置与使用较为繁琐。目标一般在代码中静态指定,目前最多20个目标。
配置少量目标,在代码中重置Project,Logstore参数。 目标Project,Logstore可以在LOG DSL规则中动态获取与设置,可以发送超过20个以上目标。 秘钥使用目标中的配置,不能动态修改,因此最多跨20个账号。
说明 当前数据加工仅支持同Region下分发。

场景1:跨账号多目标Logstore的分发

  • 原始日志
    """
    以下日志都存储在同一个logstore中,该logstore的默认逻辑名为target0
    """
    "日志1"
    http_host:  m1.abcd.com
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "日志2"
    http_host:  m2.abcd.com
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "日志3"
    http_host:  m3.abcd.com
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "日志4"
    http_host:  m4.abcd.com
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
  • 分发目标
    • http_status2XX的日志事件保留在源Logstore target0中,并设置主题为success_event
    • http_status3XX的日志事件分发到Logstore target1中,并设置主题为redirection_event
    • http_status4XX的日志事件分发到Logstore target2中,并设置主题为unauthorized_event
    • http_status5XX的日志事件分发到Logstore target3中,并设置主题为internal_server_error_event
  • LOG DSL编排
    e_switch(e_match("status", r"2\d+"), e_set("__topic__", "success_event"),
             e_match("status", r"3\d+"), e_compose(e_set("__topic__", "redirection_event"), e_output("target1")),
             e_match("status", r"4\d+"), e_compose(e_set("__topic__", "unauthorized_event"), e_output("target2")),
             e_match("status", r"5\d+"), e_compose(e_set("__topic__", "internal_server_error_event`"), e_output("target3"))
        )
  • 加工结果
    """
    源logstore: target0
    """
    __topic__:  success_event
    http_host:  m1.abcd.com
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    """
    目标logstore: target1
    """
    __topic__:  redirection_event
    http_host:  m2.abcd.com
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    """
    目标logstore: target2
    """
    __topic__: unauthorized_event
    http_host:  m3.abcd.com
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    """
    目标logstore: target3
    """
    __topic__: internal_server_error_event
    http_host:  m4.abcd.com
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
    • 不同Logstore的逻辑名target0,target1等,可在LOG DSL规则配置项中的存储目标设置,在这里可以通过AccessKey配置多账号下的Project和Logsotre。
      数据分发
    • 调用e_output之后,对应事件会在源Logstore被删除,如果输出之后仍然想在源Logstore中保留对应事件,可调用e_coutput

场景2:动态目标Logstore的分发

  • 原始日志
    "project1 logstore1"
    __tag__:type: dynamic_dispatch
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "project1 logstore2"
    __tag__:type: dynamic_dispatch
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "project2 logstore1"
    __tag__:type:  dynamic_dispatch
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "project2 logstore2"
    __tag__:type: dynamic_dispatch
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
  • 分发目标
    • 根据日志事件Project和Logstore字段值的不同,进行日志事件的动态分发。
    • 为日志事件添加标签__tag:__type,值为dynamic_dispatch
  • LOG DSL编排
    e_output(project=v("project"), logstore=v("logstore"), tags={"type": "dynamic_dispatch"})
    • 动态目标分发默认使用的AccessKey信息是加工配置项中配置的第一个存储目标对应的AccessKey账号信息。
    • 第一个存储目标的Project和Logstore信息对以上加工规则不会有影响,因为该加工规则中的e_output会动态提取Project和Logstore的值进行事件分发。
  • 加工结果
    "project1 logstore1"
    __tag__:type: dynamic_dispatch
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "project1 logstore2"
    __tag__:type: dynamic_dispatch
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "project2 logstore1"
    __tag__:type:  dynamic_dispatch
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "project2 logstore2"
    __tag__:type: dynamic_dispatch
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https

场景3:跨账号动态目标Logstore的分发

说明 该场景是场景1和场景2的结合使用场景。
  • 原始日志
    """
    project1属于账号1,该Project中有两个Logstore:logstore1, lostore2
    project2属于账号2,该Project中有两个Logstore:logstore1, lostore2
    """
    "日志1"
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "日志2"
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "日志3"
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "日志4"
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
  • 分发目标
    • 根据日志事件Project和Logstore字段值的不同,进行日志事件的动态分发。
    • 分发的目标属于不同的账号,project1属于账号1,project2属于账号2。
  • LOG DSL编排
    e_switch(e_match(v("project"), "project1"), e_output(name="target0", project=v("project"), logstore=v("logstore")),
             e_match(v("project"), "project2"), e_output(name="target1", project=v("project"), logstore=v("logstore")))
    • 在任务配置项中为存储目标target0和target1分别配置账号1和账号2的AccessKey信息。
    • 存储目标target0和target1的Project和Logstore信息对以上加工规则不会有影响,因为该加工规则中的e_output会动态提取Project和Logstore的值进行事件分发。
  • 加工结果
    """
    账号1
    project1 logstore1
    """
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    """
    账号1
    project1 logstore2
    """
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    """
    账号2
    project2 logstore1
    """
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    """
    账号2
    project2 logstore2
    """
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https