Data parsing plugins

Nginx servers generate access logs based on the log_format and access_log directives. The directives default to the following:

log_format main  '$remote_addr - $remote_user [$time_local] "$request" '
                 '$request_time $request_length '
                 '$status $body_bytes_sent "$http_referer" '
                 '"$http_user_agent"';
access_log /var/logs/nginx/access.log main

An Apache server generates logs based on the log format, path, and name specified in its log configuration file. For example, the directive CustomLog "/var/log/apache2/access_log" combined writes logs in the combined format to /var/log/apache2/access_log.

• Apache log formats

  • combined format

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

  • common format

    LogFormat "%h %l %u %t \"%r\" %>s %b" 

  • custom format

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %f %k %p %q %R %T %I %O" customized

  The following table describes the fields. Refer to mod_log_config for the full reference.

  Format	Parameter	Description
  %a	client_addr	The client IP address.
  %A	local_addr	The local IP address.
  %b	response_size_bytes	The response size in bytes. A hyphen (-) is logged for null values.
  %B	response_bytes	The response size in bytes. A 0 is logged for null values.
  %D	request_time_msec	The time taken to serve the request, in microseconds.
  %f	filename	The filename.
  %h	remote_addr	The remote hostname.
  %H	request_protocol_supple	The request protocol.
  %I	bytes_received	The bytes received by the server. Requires the mod_logio module.
  %k	keep_alive	The number of requests handled on this connection.
  %l	remote_ident	The remote logname from identd, if supplied.
  %m	request_method_supple	The request method.
  %O	bytes_sent	The bytes sent by the server. Requires the mod_logio module.
  %p	remote_port	The server port number.
  %P	child_process	The child process ID.
  %q	request_query	The query string. An empty string is logged if one does not exist.
  %r	request	The request line, which includes the method, URI, and HTTP protocol.
  %R	response_handler	The type of handler on the server.
  %s	status	The initial HTTP status of the response.
  %>s	status	The final HTTP status of the response.
  %t	time_local	The server time.
  %T	request_time_sec	The time taken to serve the request, in seconds.
  %u	remote_user	The remote user, if the request is authenticated.
  %U	request_uri_supple	The requested URI path, without the query string.
  %v	server_name	The server name.
  %V	server_name_canonical	The server name as determined by the UseCanonicalName directive.
  %{User-Agent}i	http_user_agent	The User-Agent HTTP request header.
  %{Referer}i	http_referer	The URL of the page from which the request originated.

A Windows server generates IIS logs based on the log format that you select (IIS, NCSA, or W3C).

• Log format

  #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

  • Field prefix descriptions

    Prefix	Description
    s-	Server action.
    c-	Client action.
    cs-	Client-to-server action.
    sc-	Server-to-client action.

  • Field descriptions

    Field	Description
    date	The date of the request.
    time	The time of the request.
    s-sitename	The Internet service name and instance number of the site that the client accessed.
    s-computername	The name of the server that generated the log entry.
    s-ip	The IP address of the server that generated the log entry.
    cs-method	The request method, such as GET or POST.
    cs-uri-stem	The URI resource that the client requested.
    cs-uri-query	The URI query: the information that follows the question mark (?) in the request.
    s-port	The server port.
    cs-username	The authenticated domain or username. For an authenticated user, the format is Domain\Username. For an anonymous user, a hyphen (-) is displayed.
    c-ip	The IP address of the client that accessed the server.
    cs-version	The protocol version, such as HTTP/1.0 or HTTP/1.1.
    cs(User-Agent)	The browser used by the client.
    Cookie	The content of the cookie that was sent or received. If no cookie is present, a hyphen (-) is displayed.
    referer	The site from which the user was referred.
    cs-host	The host header information.
    sc-status	The HTTP status code.
    sc-substatus	The HTTP substatus code.
    sc-win32-status	The Windows-specific operational status code.
    sc-bytes	The number of bytes sent by the server.
    cs-bytes	The number of bytes received by the server.
    time-taken	The time taken to process the request, in milliseconds.