This topic covers common issues and solutions for monitoring Object Storage Service (OSS) data with Alibaba Cloud CloudMonitor.
Object Storage Service (OSS) pushes data to CloudMonitor for analysis and processing. The storage capacity and traffic metrics displayed in the OSS console come from CloudMonitor.
Data from OSS may take 1 to 2 hours to appear in CloudMonitor. The interval between data pushes cannot exceed 5 minutes. Otherwise, CloudMonitor will reject the stale data. Data backfilling is not supported. Therefore, do not use CloudMonitor data to reconcile your bill. To verify your bill, contact technical support.
Case: "Insufficient Data" alert status
Problem analysis: You can check the data in the Monitoring Service Overview section of User Profile. The "Insufficient Data" message is displayed if no data is generated.
Case: High upload or download latency
Analysis: CloudMonitor collects latency data from its monitoring nodes using probe requests. This data may not reflect the actual experience of your end users.
Solution: If CloudMonitor reports high access latency, follow these steps to troubleshoot the issue:
-
Check if your clients are actually experiencing latency.
-
If users report latency when they access the affected bucket, perform a packet capture to analyze the access data.
-
You can also use log analysis to check for access latency for the corresponding time period.
Case: High latency for OSS requests in your system
If your own monitoring system detects high latency for requests to OSS, follow these steps to troubleshoot the issue:
-
Check your corporate network. Ping other websites to test for general network latency.
-
Create an ECS instance in the same region as your OSS bucket and test the access latency from the instance.
-
Submit the request ID for the delayed OSS request to technical support for investigation.
-
Perform a packet capture to analyze the upload data. Use the following command:
tcpdump -i <outbound network interface> -s0 'host <public IP address> and host <OSS endpoint>' -w result.pcap
Case: The request validity rate drops
Symptom: You receive an alert from CloudMonitor, such as "Object Storage Service (OSS) (<)bucket=p2xxx, userId=135114002(>), request validity rate (30.51<90%), duration 0 minutes".
Solution: The request validity rate is the percentage of requests with a 2xx or 3xx status code out of the total number of requests. To diagnose the drop, check the distribution of status codes in the OSS console. An increase in error status codes, such as 4xx or 5xx, will lower the request validity rate. You can also enable OSS access logging for detailed log analysis.
Case: CloudMonitor 404 error alert
Symptom: You receive an alert from CloudMonitor, such as "Object Storage Service (OSS) instance: bucket=***-ali, userId=197*****745, number of 'resource not found' error requests returned to normal at 11:45, value was 30, duration 5 minutes".
Analysis: This alert occurs when a client requests a resource that does not exist in the bucket. While this is a 404 error, it is an expected response from the service and does not indicate an issue with OSS itself.
Case: NoSuchWebsiteConfiguration error
Analysis: This 404 error occurs when a client accesses a bucket as a website, but the static website hosting feature is not configured for that bucket. This is an expected error if the feature is not enabled. A 200 status code indicates that the feature is configured and the request was successful.
Case: No data in API statistics chart
Analysis: API monitoring data is processed and displayed with a one-day delay. For example, the complete data for October 12 is available on October 13.
Case: CloudMonitor data is inconsistent with bills
OSS pushes data to CloudMonitor with a delay of 1 to 2 hours. The push interval cannot exceed 5 minutes. If a data push is delayed beyond this window, CloudMonitor rejects it, and the data cannot be backfilled. Due to these potential delays and data gaps, CloudMonitor data is not suitable for billing reconciliation. Instead, use one of the following methods:
-
Enable OSS access logging. Compare the statistics from your logs with your bill.
-
Enable the log analysis feature for OSS. After you import the logs, you can directly view the analysis results.
Case: Request validity rate is 0 in CloudMonitor but normal in OSS
Analysis: The request validity rate is the percentage of successful requests, which are requests with 2xx and 3xx status codes, out of the total requests. If you see a sudden drop to 0 in CloudMonitor, first check your OSS console or access logs to confirm if there are any actual request failures.
Cause: This issue can occur if OSS pushes a large batch of log data from its cluster to CloudMonitor, and the push exceeds CloudMonitor's data ingestion time window. Because CloudMonitor does not support backfilling rejected data, CloudMonitor may report 0 for that period, which causes the calculated request validity rate to drop to 0.