Troubleshooting Logtail machine group issues

Machine group installation examples

Troubleshooting checklist

1. Step 1: Verify that Logtail is running: Check if the Logtail process is active on the server.

2. Step 2: Verify that the machine group IP address matches the IP address that Logtail obtains: A mismatched IP address between the machine group and the Logtail app_info.json file can cause heartbeat failures.

3. Step 3: Check whether the Logtail startup parameters are correct: Check whether the Project region configured in the ilogtail_config.json file is correct.

4. Step 4: Check network connectivity: Ensure the server can connect to the project endpoints.

5. Step 5: Verify the system time on the Logtail server: If the system time is significantly different from the actual time, correct it.

6. Step 6: Verify the user identifier for cross-account collection: This step is required if the server is not an ECS instance or if the instance and project belong to different Alibaba Cloud accounts.

7. Step 7: Verify the custom identifier for a custom identifier-based machine group: If you use a custom identifier-based machine group, ensure that the custom identifier is configured on the server.

8. Step 8: Restart Logtail: After making changes, restart Logtail to apply them.

Next steps

View Logtail collection errors: If the heartbeat status is OK but logs are still not collected, check Logtail error messages to continue troubleshooting.

Step 1: Verify Logtail status

1. Obtain the value of the ip field from the app_info.json file.

   The following table lists the default paths to this file on different operating systems.

   Operating system	Logtail	app_info.json path
   Linux	Logtail (64-bit)	/usr/local/ilogtail/app_info.json
   Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\app_info.json
   	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\app_info.json
   Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\app_info.json

   Logtail records the collected IP address in the ip field of the app_info.json file.

   {
     "UUID" : "",
     "hostname" : "iZ8vbdlzf******azuhZ",
     "instance_id" : "E9633380-***********-00163E1AA597_172.16.2.200_166****11",
     "ip" : "172.**.**.200",
     "logtail_version" : "1.3.1",
     "os" : "Linux; 4.19.91-26.1.al7.x86_64; #1 SMP Tue Jul 26 17:52:28 CST 2022; x86_64",
     "update_time" : "2022-12-27 05:38:33"
   }

2. Verify that the machine group uses the IP address that Logtail obtains.

   Simple Log Service provides two types of machine groups: IP address-based machine groups and custom identifier-based machine groups. For more information, see Machine groups.

   • IP address-based machine group: Check if the IP Address list contains the IP address that you obtained in the previous step.

     If the IP address is not included, first confirm the server's correct IP address. If the address in the IP Address list is incorrect, update it in the machine group settings. If the IP address obtained by Logtail in the previous step is wrong, modify the working_ip parameter as described in Set Logtail startup parameters and restart Logtail. After making the correction, monitor the machine's heartbeat. If the status becomes normal, the issue is resolved.

   • Custom identifier-based machine group: In the Machine Group Status list, check for the IP address you obtained in the previous step. If the Heartbeat status is OK, the issue is resolved.

The ilogtail_config.json file contains the startup parameters for Logtail.

1. Log on to the server where Logtail is installed.

2. Find the ilogtail_config.json file.

   The following table lists the default paths to this file on different operating systems.

   Operating system	Logtail	ilogtail_config.json path
   Linux	Logtail (64-bit)	/usr/local/ilogtail/ilogtail_config.json
   Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json
   	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json
   Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json

   1. Open the ilogtail_config.json file and check if the configuration parameters are correct.

      {
        "config_server_address" : "http://logtail.<config_region>.log.aliyuncs.com",
        "data_server_list" :
        [
          {
            "cluster" : "<project_region>",
            "endpoint" : "<endpoint>"
          }
        ],
        ...
      }

      • If the startup parameters in the ilogtail_config.json file match the following descriptions, they are correct.

      • If the startup parameters are incorrect, modify the ilogtail_config.json file based on the following table and then restart Logtail. For more information, see Restart Logtail.

        For information about project regions, see Supported regions.

        Scenario	Network type	<config_region>	<endpoint>
        The server is an ECS instance, and it is in the same region as the project.	Alibaba Cloud internal network	<project_region>-intranet	<project_region>-intranet.log.aliyuncs.com
        Other scenarios	Internet	<project_region>	<project_region>.log.aliyuncs.com
        	transfer acceleration		log-global.aliyuncs.com

For successful data uploads, the Logtail server must be able to connect to the following addresses.

1. The address specified by the config_server_address field in the ilogtail_config.json file and its HTTPS version.

2. http://<project-name>.<endpoint>.

   • You can find the project name and region as shown in the following figure.

     

   • <endpoint> is the address specified by the data_server_list.endpoint parameter in the ilogtail_config.json file.

3. http://ali-<project-region>-sls-admin.<endpoint>, where <endpoint> is the address specified by the data_server_list.endpoint field in the ilogtail_config.json file.

Follow these steps to check and resolve network issues:

Linux

1. Log on to the server where Logtail is installed.

2. Run the date command to view the system time.

   Wed Dec 28 06:59:26 UTC 2022

   If the system time deviates significantly from the actual time, take one of the following actions.

   • Adjust the system time to the correct time.

   • If you cannot modify the system time, modify the Logtail startup parameters by adding the configuration item "enable_log_time_auto_adjust": true to the ilogtail_config.json file. After you modify the file, restart Logtail. For more information, see Restart Logtail. For the path of the ilogtail_config.json file, see Step 3: Check whether the Logtail startup parameters are correct.

Windows

1. Log on to the server where Logtail is installed.

2. Check the time in the taskbar at the bottom-right of the desktop.

   • Adjust the system time to the correct time.

   • If you cannot change the system time, modify the Logtail startup parameters. Add the "enable_log_time_auto_adjust": true configuration to the ilogtail_config.json file. After the modification, you must restart Logtail. For more information, see Restart Logtail. For the path to the ilogtail_config.json file, see Step 3: Verify startup parameters.

Step 6: Verify the user identifier

Check for the user identifier file in its specified directory to verify its configuration. This file grants the project's account permission to access the server.

• If a user identifier file does not exist in the specified path or is misconfigured, follow these steps to resolve the issue:

  • For Linux systems: Run the cd /etc/ilogtail/users/ && touch <uid> command to create a user identifier file. In this command, <uid> is the Alibaba Cloud account ID to which the project belongs.

  • On Windows, go to the C:\LogtailData\users\ directory and create an empty file named <uid>, where <uid> is the Alibaba Cloud account ID of the project.

• If a file named after the project's Alibaba Cloud account ID exists in the specified path, the user identifier is correctly configured.

Step 7: Verify the custom identifier

If you use a custom identifier-based machine group, you can check the user_defined_id file in the specified directory to determine if a custom identifier is configured on the server.

• If the result is empty, you need to check whether the user_defined_id file exists or whether a user-defined identifier is configured in the file.

  Note

  The paths to the user_defined_id file are:

  • Linux: /etc/ilogtail/user_defined_id

  • Windows: C:\LogtailData\user_defined_id

  • If the user_defined_id file does not exist, create it and enter the custom identifier for the machine group. For more information, see Configure a custom identifier.

  • If the user_defined_id file exists but does not contain the correct custom identifier, add the custom identifier for the machine group to the file. For more information, see Configure a custom identifier.

  • If the user_defined_id file already contains the custom identifier that you set for the machine group, the configuration is correct.

Step 8: Restart Logtail

After completing these changes, you must restart Logtail.