Upload logs by using the Syslog protocol

Limits

• The Syslog protocol must be the standard RFC 5424. If you use other Syslog protocols, we recommend that you see Collect Syslog from Linux systems.

• The maximum size of a log entry is 64 KB.

• To ensure data transmission security, you must transmit data over TCP using Transport Layer Security (TLS) 1.2.

Configuration

To upload logs using the Syslog protocol, you must configure a log upload address in the ProjectName.Endpoint:SyslogPort format. Example: test-project-1.cn-hangzhou-intranet.log.aliyuncs.com:10009. Select an endpoint based on the region where your project is located. For more information, see Endpoints. The Syslog port is 10009. You must also configure parameters in the STRUCTURED-DATA field, such as the Simple Log Service project and LogStore, and your Alibaba Cloud account's AccessKey pair.

Example: Collect logs by using syslog-ng

1. Open the syslog-ng configuration file.

   The configuration file is typically located at /etc/syslog-ng/syslog-ng.conf.

2. Add the following configuration block to the end of the syslog-ng configuration file, and then configure the parameters within it based on your business requirements.

   ### Syslog-ng Logging Config for LogService ### 
   template LogServiceFormat { 
       template("<${PRI}>1 ${ISODATE} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} [logservice project=\"test-project-1\" logstore=\"test-logstore-1\" access-key-id=\"<yourAccessKeyId>\" access-key-secret=\"<yourAccessKeySecret>\"] $MSG\n"); template_escape(no); 
   }; 
   destination d_logservice{ 
        tcp("test-project-1.cn-hangzhou.log.aliyuncs.com" port(10009) 
        tls(peer-verify(required-untrusted)) 
        template(LogServiceFormat)); 
   }; 
   log { 
        source(s_sys); # default use s_sys 
        destination(d_logservice); 
   }; 
   ### END Syslog-ng Logging Config for LogService ###

3. Restart syslog-ng.

   Run the sudo /etc/init.d/syslog-ng restart, sudo service syslog-ng restart, or sudo systemctl restart syslog-ng command to restart syslog-ng.

4. Run the logger command to generate a test log.

   Example: Run the logger hello world! command to generate a log.

Sample log

After you upload logs, you must configure indexes before you can view the logs in the Simple Log Service console. For more information, see Create an index.

03-28 11:01:01      __source__: xxx
                    __topic__: syslog-forwarder
                    _facility_: 3
                    _hostname_: xxx
                    _priority_: 30
                    _program_: systemd
                    _severity_: 6
                    _unixtimestamp_: 1553742061117098000
                    content: Started Session 59532 of user root.
03-28 11:00:15      __source__: mymachine.example.com
                    __topic__: syslog-forwarder
                    _facility_: 4
                    _hostname_: mymachine.example.com
                    _message_id_: ID47
                    _priority_: 34
                    _program_: su
                    _severity_: 2
                    _unixtimestamp_: 1553742015003000000
                    content: this is a test message

FAQ

• Test log uploads

  You can use Netcat to test log uploads to verify network connectivity and ensure that the AccessKey has upload permissions.

  1. Log on to the server on which you want to test log uploads.

  2. Run the following command to install Netcat:

     sudo yum install nmap-ncat

  3. Run the following command to connect to Simple Log Service:

     ncat --ssl <yourProject>.<yourEndpoint> 10009

     Sample command:

     ncat --ssl test-project-1.cn-hangzhou.log.aliyuncs.com 10009

  4. Netcat does not automatically detect network disconnections. Within 30 seconds of running the ncat command, enter the log message and press Enter.

     <34>1 2019-03-28T03:00:15.003Z mymachine.example.com su - ID47 [logservice project="<yourProject>" logstore="<yourLogstore>" access-key-id="<yourAccessKeyID>" access-key-secret="<yourAccessKeySecret>"] this is a test message

     Sample command:

     <34>1 2019-03-28T03:00:15.003Z mymachine.example.com su - ID47 [logservice project="trace-doc-test" logstore="doc-test-001-logs" access-key-id="<yourAccessKeyID>" access-key-secret="<yourAccessKeySecret>"] this is a test message

  5. Preview logs in the Simple Log Service console to check whether the logs are uploaded.

     For more information, see Preview logs.

     A log entry with a timestamp of 2019-03-28 11:00:15 and a source of mymachine.example.com appears in the preview results. The Syslog fields in the content column, including _hostname_, _severity_, _facility_, _message_id_, _unixtimestamp_, _program_, and _priority_, are parsed. This indicates that the log was uploaded and parsed successfully.

• Diagnose collection errors

  If a manual log upload fails, you can view specific error messages by diagnosing collection errors. For more information, see How to view Logtail collection error information.