Configuration management

Common LoongCollector commands

Start, stop, and restart LoongCollector

If you have specific performance requirements for data collection, modify the startup parameter configuration file (ilogtail_config.json) before you start LoongCollector.

sudo /etc/init.d/loongcollectord start # Start
sudo /etc/init.d/loongcollectord stop # Stop
sudo /etc/init.d/loongcollectord restart # Restart

Check the LoongCollector status

sudo /etc/init.d/loongcollectord status # "loongcollector is running" indicates the agent started successfully.

Check the LoongCollector version

cat /usr/local/ilogtail/app_info.json # The version is in the loongcollector_version field.

Startup parameter configuration file (ilogtail_config.json)

• Configures startup parameters for LoongCollector. Improper settings can degrade collection performance or cause errors.

• File path: /usr/local/ilogtail/ilogtail_config.json.

• If the collection performance is insufficient, you can modify the startup parameters. For specific use cases, see Solve common collection issues by modifying configurations.

• File example:

  {
      "primary_region" : "cn-beijing", // Default region. You can ignore this parameter.
      "config_servers" : // The address used to obtain collection configurations.
      [
          "http://logtail.cn-beijing.log.aliyuncs.com"
      ],
      "data_servers" : // The address for data transmission with Simple Log Service (SLS).
      [
          {
              "region" : "cn-beijing",
              "endpoint_list": [
                  "cn-beijing.log.aliyuncs.com"
              ]
          }
      ],
      "cpu_usage_limit" : 0.4,
      "mem_usage_limit" : 384,
      "max_bytes_per_sec" : 20971520,
      "bytes_per_sec" : 1048576,
      "buffer_file_num" : 25,
      "buffer_file_size" : 20971520,
      "buffer_map_num" : 5
  }

• The following table lists common startup parameters. Add or modify them as needed, and retain the default values for any parameters not listed.

  ilogtail_config.json startup parameters

  Parameter	Type	Description	Example
  cpu_usage_limit	double	The CPU usage threshold, based on a single CPU core. Valid values: 0.1 to the number of CPU cores on the machine. Default value: 0.4 Warning The cpu_usage_limit parameter is a soft limit. The actual CPU usage may exceed this value. If the limit is exceeded for five consecutive minutes, a circuit breaker is triggered and LoongCollector automatically restarts. For example, if you set this parameter to 0.4, SLS attempts to limit the collector's CPU usage to 40% of a single CPU core. If the usage exceeds this limit, LoongCollector restarts. Typically, a single core can process about 100 MB/s in minimalist mode or 20 MB/s in full regular expression mode.	"cpu_usage_limit" : 0.4
  mem_usage_limit	int	The memory usage threshold in megabytes (MB). Valid values: 128 MB to 8192 MB Default value: 384 MB (for hosts), 2048 MB (for ACK components) Warning The mem_usage_limit parameter is a soft limit. The actual memory usage may exceed this value. If the limit is exceeded for five consecutive minutes, a circuit breaker is triggered and LoongCollector automatically restarts. The mem_usage_limit parameter affects the collection rate, the number of monitored directories and files, and sending congestion. For more information, see Limits.	"mem_usage_limit" : 384
  max_bytes_per_sec	int	The maximum sending rate for raw data, in bytes per second. Valid values: 1,024 Byte/s to 52,428,800 Byte/s Default value: 20,971,520 Byte/s Important A value greater than 20,971,520 Byte/s (20 MB/s) indicates no speed limit. For example, if you set this parameter to 2097152, the data sending rate is 2 MB/s.	"max_bytes_per_sec" : 2097152
  process_thread_count	int	The number of data processing threads. Valid values: 1 to 64 Default value: 1 A single thread can typically handle 24 MB/s of data writes in minimalist mode or 12 MB/s in full regular expression mode. You do not need to adjust this parameter value under normal circumstances.	"process_thread_count" : 1
  send_request_concurrency	int	The level of asynchronous concurrency. Valid values: 15 to 80 Default value: 15 If the write TPS is high, you can set a higher asynchronous concurrency value. You can estimate that one concurrent request supports approximately 0.5 MB/s to 1 MB/s of network throughput, depending on network latency. Note Setting the asynchronous concurrency value too high can consume too many network ports, which requires you to adjust TCP-related parameters. For more information, see Adjust TCP-related parameters.	"send_request_concurrency" : 15
  buffer_file_num	int	The maximum number of cache files. Valid values: 1 to 100 Default value: 25 If network issues or write quota overruns occur, the collector caches the parsed logs to local files in the installation directory. It retries sending them after the issue is resolved.	"buffer_file_num" : 25
  buffer_file_size	int	The maximum size of a single cache file in bytes. Valid values: 1,048,576 bytes to 104,857,600 bytes Default value: 20,971,520 bytes The maximum disk space for cache files is buffer_file_size * buffer_file_num.	"buffer_file_size" : 20971520
  buffer_file_path	String	The directory where cache files are stored. The default value is empty, which means cache files are stored in the installation directory /usr/local/ilogtail. If you set this parameter, you must manually move files named logtail_buffer_file_* from the original directory to the new directory. This ensures that the collector can read the cache files, send the data, and then delete the files.	"buffer_file_path" : ""
  bind_interface	String	The name of the network interface card to bind. If left empty, LoongCollector automatically binds to an available card. If you set this parameter to a specific network interface card, such as eth1, LoongCollector uses that card to upload logs.	"bind_interface" : ""
  check_point_filename	String	The path where the checkpoint file is saved. Default value: /tmp/logtail_check_point.	"check_point_filename" : "/tmp/logtail_check_point"
  check_point_dump_interval	int	The interval at which to update the checkpoint file, in seconds. The default value is 900 (15 minutes).	"check_point_dump_interval" : 900
  user_config_file_path	String	The path where the configuration file is saved. By default, this is the directory of the process binary, and the filename is user_log_config.json.	"user_config_file_path" : "user_log_config.json"
  discard_old_data	Boolean	Specifies whether to discard historical logs. Default value: true. If this parameter is set to true, logs older than 12 hours are discarded.	"discard_old_data" : true
  ilogtail_discard_interval	int	The age threshold for discarding historical logs, in seconds. Default value: 43200 (12 hours).	"ilogtail_discard_interval": 43200
  working_ip	String	Specifies the server's reported IP address. If left empty, the IP address is automatically obtained from the server.	"working_ip" : ""
  working_hostname	String	Specifies the server's reported hostname. If left empty, the hostname is automatically obtained from the server.	"working_hostname" : ""
  max_read_buffer_size	long	The maximum size of a single log entry, in bytes. Default value: 524,288 (512 KB). Maximum value: 8,388,608 (8 MB). If a single log entry exceeds 524,288 bytes, increase the value of this parameter.	"max_read_buffer_size" : 524288
  oas_connect_timeout	long	The connection timeout for requests to get collection configurations or AccessKey information, in seconds. Default value: 5. Increase this value for slow network connections.	"oas_connect_timeout" : 5
  oas_request_timeout	long	The total request timeout for requests to get collection configurations or AccessKey information, in seconds. Default value: 10. Increase this value if requests time out due to a slow network.	"oas_request_timeout" : 10
  data_server_port	long	If you set data_server_port to 443, data is sent to SLS over HTTPS.	"data_server_port": 443
  enable_log_time_auto_adjust	Boolean	If you set enable_log_time_auto_adjust to true, the log time automatically adjusts to the server local time. For data security, SLS validates the timestamp in each request, including requests from LoongCollector. It rejects requests with a timestamp that differs from the SLS server time by more than 15 minutes. The collector includes the server local time in its requests. If the server local time is changed (for example, to a future time for testing), requests can be rejected and data writes can fail. Use this parameter to allow the log time to adapt to the server local time. Important When this feature is enabled, the offset between the SLS time and the server local time is added to the log time. Because the offset is updated only when a request is rejected by the SLS server, the log time that you query in SLS might not match the actual write time. Some of LoongCollector's logic relies on a monotonically increasing system time. We recommend restarting the collector after you adjust the machine's time.	"enable_log_time_auto_adjust": true
  accept_multi_config	Boolean	Specifies whether to allow multiple collection configurations to collect logs from the same file. Default value: false. By default, a file can be collected by only one collection configuration. You can use this parameter to remove this restriction. The processing for each collection configuration is independent. Allowing multiple configurations to collect from the same file multiplies the CPU and memory consumption.	"accept_multi_config": true
  enable_checkpoint_sync_write	Boolean	Specifies whether to enable the sync write feature. Default value: false. The sync write feature is mainly used with the ExactlyOnce write feature. When ExactlyOnce is enabled, the collector records fine-grained, file-level checkpoint information to the local disk. However, for performance reasons, the sync call is not used by default when writing checkpoints. If the machine restarts before buffer data is written to the disk, the checkpoint may be lost. In this case, you can set enable_checkpoint_sync_write to true to enable sync writes. For more information, see Logtail configuration (Old version).	"enable_checkpoint_sync_write": false
  enable_env_ref_in_config	Boolean	Specifies whether to enable environment variable replacement in collection configurations. Default value: false. When this feature is enabled, you can use ${xxx} in the collection configuration in the console as a placeholder for the environment variable xxx. For example, if you set the collection path to /${xxx}/logs and the environment variable is xxx=user, the effective collection path is /user/logs. If your configuration needs to use the literal characters ${ or }, you can escape them by using $${ and $}.	"enable_env_ref_in_config": false
  logreader_max_rotate_queue_size	Int	The maximum length of the rotation queue. Default value: 20. When log collection is blocked or delayed, files pending collection hold their file handles and wait in this queue. If collection is delayed and you need to control the maximum disk usage, consider reducing this value. Warning If the number of delayed files exceeds this value, the collection of new files is skipped.	"logreader_max_rotate_queue_size" : 10
  data_endpoint_policy	string	The switching policy for SLS endpoints. Valid values: Note Check the data_server parameter in the ilogtail_config.json file to see if a default endpoint is configured. designated_first (Default) If a default endpoint is specified for a region and is available, the system uses the default endpoint. If a default endpoint is specified for a region but is unavailable, the system automatically selects an available endpoint. If no default endpoint is specified for a region, the system automatically selects an available endpoint. designated_locked If a default endpoint is specified for a region, the system uses only that endpoint, regardless of its availability. If no default endpoint is specified for a region, the system automatically selects an available endpoint.	"data_endpoint_policy" : "designated_first"
  inotify_black_list	Array<String>	An exact-match blacklist for the inotify listener. Directories in this list are excluded from inotify monitoring.	"inotify_black_list": ["/tmp"]
  host_path_blacklist	String	A global blacklist for host paths, matched by substring. On Linux, separate multiple substrings with a colon (:). For example, "host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-" prevents the collection of data from NAS mounts.	"host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-"
  LOGTAIL_LOG_LEVEL	String	The log level, which must be configured as an environment variable. The default value is empty, which corresponds to info. Valid values are trace, debug, info, warning, error, and fatal.	LOGTAIL_LOG_LEVEL=info
  FORCE_RELEASE_STOP_CONTAINER_FILE	Boolean	Configuration: This parameter can be configured only as an environment variable. Description: When this parameter is set to true, the collector immediately releases the container's file handle when the application container stops. This operation prevents the container from failing to exit due to an unreleased file handle. Notes: This setting does not guarantee the completeness of data collection from the container. We recommend that you add a delay of a few seconds before your application exits to ensure all logs are collected.	"FORCE_RELEASE_STOP_CONTAINER_FILE" : "true"

Performance tuning

SLS limits LoongCollector's resource consumption to prevent it from affecting other services. You can modify the startup parameters to improve collection performance.

Recommended parameter values

The following values are recommended for collecting common JSON files. Full regular expression mode and separator mode perform similarly to JSON mode, while minimalist mode is about five times faster. Data complexity, rule complexity, and the number of monitored directories and files all affect CPU and memory consumption. Adjust the parameters based on your requirements.

Add or modify the parameter values in /usr/local/ilogtail/ilogtail_config.json and restart LoongCollector for the changes to take effect. The following table provides recommended values for different collection rates.

Note

When you configure startup parameters based on the Collection rate greater than 40 MB/s column in the table, the collection performance approaches its limit.

Parameter	Description	Rate > 10 MB/s	Rate > 20 MB/s	Rate > 40 MB/s
cpu_usage_limit	The CPU usage threshold, based on a single CPU core. Valid values: 0.1 to the number of CPU cores on the machine. Default value: 0.4 A value of 0.4 means SLS attempts to limit the collector's CPU usage to 40% of a single core. If the usage exceeds this limit, LoongCollector automatically restarts. This is a soft limit. The actual CPU usage may exceed this value. If the limit is exceeded for five consecutive minutes, a circuit breaker is triggered and LoongCollector automatically restarts.	1	2	4
mem_usage_limit	The memory usage threshold. Valid values: 128 MB to 8,192 MB Default value: 384 MB The collection rate, the number of monitored directories and files, and sending congestion are affected by this parameter. For more information, see Limits. This is a soft limit. The actual memory usage may exceed this value. If the limit is exceeded for five consecutive minutes, a circuit breaker is triggered and LoongCollector automatically restarts.	1024	2048	4096
process_thread_count	The number of threads for data processing. Valid values: 1 to 64 Default value: 1 A single thread can handle 24 MB/s of data writes in minimalist mode or 12 MB/s in full regular expression mode. You do not need to adjust this parameter value under normal circumstances.	2	4	8
max_bytes_per_sec	The maximum rate at which raw data can be sent, in bytes per second. Valid values: 1,024 Byte/s to 52,428,800 Byte/s Default value: 20,971,520 Byte/s Important A value greater than 20,971,520 Byte/s (20 MB/s) indicates no speed limit.	209715200	209715200	209715200
send_request_concurrency	The amount of asynchronous concurrency. Valid values: 15 to 80 Default value: 15 If the write TPS is high, you can set a higher asynchronous concurrency value. You can estimate that each concurrent request can handle 0.5 MB/s to 1 MB/s of network throughput, depending on network latency.	20	40	80

Other common configuration files

LoongCollector uses the following configuration and information files during runtime.

Collection configuration (user_log_config.json)

• Description: Records the collection configuration that LoongCollector obtains from Simple Log Service (SLS). The file is updated whenever the collection configuration changes. Do not modify this file, except to manually configure sensitive information such as an AccessKey or database passwords.

• Path: /usr/local/ilogtail/user_log_config.json.

• Use case: You can check this file to confirm whether a collection configuration has been delivered to the server. If this file exists and its content matches the collection configuration in SLS, the configuration has been successfully delivered.

AppInfo file (app_info.json)

• Description: Records LoongCollector's startup time, retrieved IP address, and hostname. This file is read-only; any modifications do not take effect.

• Path: /usr/local/ilogtail/app_info.json.

• Use case: View the server IP address identified by SLS to verify that it matches the IP address in the IP-based machine group. This is typically used to troubleshoot heartbeat failures in IP-based machine groups.

  LoongCollector retrieves the IP address in the following order of precedence: the working_ip parameter in ilogtail_config.json, the hostname-to-IP binding in the server's /etc/hosts file, and the IP address of the first network interface card.

LoongCollector runtime log (loongcollector.LOG)

• Description: Records LoongCollector's runtime logs. The log levels, from lowest to highest, are INFO, WARN, and ERROR.

• Path: /usr/local/ilogtail/loongcollector.LOG.

• Use case: If a collection error occurs, first use Diagnose and monitor LoongCollector to check for errors. Then, troubleshoot the issue by using the log collection error types and the LoongCollector runtime log.

User ID file

• Description: Contains the Alibaba Cloud account ID of the project to which the collected logs belong. The filename is the account ID, with no suffix. Its presence indicates that this account has permission to collect logs from this server.

• Path: /etc/ilogtail/users/{Alibaba Cloud account ID}.

• Use case: Configure a user ID only when collecting logs from ECS instances under other accounts, self-managed servers, or servers from other cloud providers. If multiple accounts need to collect logs from the same server, create multiple user ID files on that server.

Custom identifier file

• Description: Configures a custom identifier used as the value for a custom-identifier machine group, which helps SLS discover LoongCollector on the server and establish a heartbeat.

• Path: /etc/ilogtail/user_defined_id.

• Use case: Required when using a custom-identifier machine group. For more information, see Associate a machine group with a collection configuration.

Checkpoint file

• Description: Records the current collection position to ensure data integrity.

• Path: The default path is /tmp/logtail_check_point.

• Use case: Manage the checkpoint file by modifying the startup parameter file. For more information, see Limits.