Resolve garbled Chinese characters in collected logs

Symptoms

Collected logs that contain Chinese characters display as garbled text.

Collection notes

Configuration changes apply only to new logs and do not correct existing data. To fix encoding issues in historical logs, apply the solutions in this topic and then re-process the files as described in Import historical log files.

Step 1: Check the log file encoding

1. Open the original log file in an editor like Notepad++. You can find the file's current encoding in the lower-right corner.

   2023-10-05 14:30:15 [INFO] [OrderService] Starting to create a new order, UserID: 100023, ProductID: SKU-7782
   2023-10-05 14:30:17 [DEBUG] [OrderService] Order validation passed - sufficient stock (remaining stock: 85), address verification: Hangzhou, Zhejiang Province
   2023-10-05 14:30:19 [WARN] [OrderService] Duplicate submission request detected, UserSessionID: SES-20231005143012, this request has been ignored
   2023-10-05 14:30:22 [INFO] [OrderService] Payment gateway callback successful, OrderNo: PO-20231005143015, Amount: ¥358.00
   2023-10-05 14:30:25 [INFO] [OrderService] Order status update complete, current status: Awaiting shipment, Logistics provider: SF Express
   2023-10-05 14:30:31 [METRICS] [OrderService] API performance - /api/order/create: Average response time 128 ms
   2023-10-05 14:30:34 [DEBUG] [OrderService] Redis cache hit for order details, OrderNo: PO-20231005143015
   2023-10-05 14:30:37 [AUDIT] [OrderService] Sensitive operation log - Administrator [admin] modified the order amount, Original amount: ¥358.00
   2023-10-05 14:30:40 [INFO] [OrderService] E-invoice generated successfully, InvoiceNo: INV-20231005143015

2. If the file is not UTF-8 encoded, convert it to UTF-8. After the conversion, verify that the log content displays correctly.

   Note

   Back up the log file before converting the encoding to prevent data corruption.

Step 2: Modify the Logtail encoding

1. Log on to the Simple Log Service console. In the Projects section, click the one you want.

   

2. On the Log Storage > Logstores tab, click the > icon next to the target Logstore, and then choose Data Import > Logtail Configurations.

3. In the Logtail Configuration list, click the target Logtail configuration.

4. On the Logtail configuration page, click Modify.

5. Click Advanced, set File Encoding to UTF8, and then click Save.

Step 3: Check the index settings

1. On the Log Storage > Logstores tab, click the logstore you want.

2. On the Query and Analysis page of the Logstore, click Query & Analysis > Properties.

3. On the Search & Analysis page, enable Contains Chinese for both Full-text Index and Field Index.

Step 4: Query logs

On the Search/Analysis page, enter a query or an analysis statement in the search box, and then click Search/Analysis.

The query results now correctly display the Chinese log content.