Causes and troubleshooting of errors when getting data from OSS-Simple Log Service(SLS)-阿里云帮助中心

If a transformation rule involves loading resources from Object Storage Service (OSS), resource loading or refresh errors may occur. Learn about common causes and how to troubleshoot them.

After the DPI engine reads data from the source Logstore, it transforms the log events. If the transformation rule loads external resources from OSS, ApsaraDB RDS, or other Logstores, resource loading or refresh errors may occur.

Error impact

The logging.levelname field in the error log specifies the error level: WARNING or ERROR.

For ERROR-level errors, the affected log event is discarded. The transformation output does not include this event, and processing continues with the next log event without a retry.
For WARNING-level errors, such as a regex mismatch, the current DSL step is skipped for that event, and processing moves to the next step.

Troubleshooting

Check the logging.levelname field in the error log to determine the error level.
Check the message field in the error log to identify which log event caused the error. For more information, see View error logs.
Check the reason field in the error log to determine the cause of the error for that log event.

Based on the cause, add logic to your transformation script to handle these exceptions. You can use process control functions, such as e_if and e_switch, to catch and manage these errors. The following is an error log example: The message field contains the details of the log event that failed, and the reason field shows the cause of the error, integer division or modulo by zero:

07-26 16:21:48
    __source__:
    __tag__:__job_name__: etl-preview-1564110081-604324
    __tag__:__schedule_type__: DryRun
    __topic__: __etl-log-status__
    etl_context: {
      "project": "ali-licheng-test-uk2",
      "logstore": "source_logstore",
      "shard_id": "2"}
    event_id: transorm_data:process:fail
    extra_info_params: {"src_event": {"__time__": "1563266137", "__topic__": "", "__source__": xxx, "__tag__:__client_ip__": xxx,
    "__tag__:__receive_time__": "1563774221", "body_bytes_sent": "740", "client_ip": "1.2.3.4", "host": "m.abcd.com", "status": "403"}}
    logging: {"process": "34", "levelname": "ERROR", "funcName": "_transform_events_to_logstore", "module": "logclient_operator", "threadName": "MainThread",
    "thread": "139890150307648", "processName": "ForkProcess-1"}
    message: [ali-licheng-test-uk2/source_logstore/2]transform_data: fail to process event: {"__time__": "1563266137", "__topic__": "", "__source__": xxx,
    "__tag__:__client_ip__": xxx, "__tag__:__receive_time__": "1563774221", "body_bytes_sent": "740", "client_ip": "1.2.3.4", "host": "m.abcd.com", "status": "403"},
    detail=integer division or modulo by zero
    reason: integer division or modulo by zero

Common errors when getting data from OSS

Note

In the following examples, AK_ID is the abbreviation for AccessKeyId, and AK_KEY is the abbreviation for AccessKeySecret.

Incorrect file path or file format

A 404 error log is generated because the `data` directory does not exist in OSS or the file format is incorrect.

Example transformation rule

# Assume that the correct directory is test.
e_set("oss_file",res_oss_file(endpoint, ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), bucket, 'data/test.txt', format='text', change_detect_interval=20))
# This format does not exist in the test directory.
e_set("oss_file",res_oss_file(endpoint, ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), bucket, 'test/test.dat', format='text', change_detect_interval=20))

Error log

message:  Exception: {'status': 404, 'x-oss-request-id': '5D49****878', 'details': {'Code': 'NoSuchKey', 'Message': 'The specified key does not exist.', 'RequestId': '5D4***8878', 'HostId': 'lo***g.oss-cn-hangzhou.aliyuncs.com', 'Key': 'oss/test.txt'}}

Troubleshooting

Check the log event that caused the error. If the status in the error message is 404 and the Messages field is related to the file, first check whether the file path is correct.

Solution

Change the res_oss_file parameter in the orchestration of the file function to the correct file path.

# Assume that the correct directory is test.
e_set("oss_file",res_oss_file(endpoint, ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), bucket, 'test/test.txt', format='text', change_detect_interval=20))

File decoding error

An error occurs when a file pulled from OSS is decoded or when a non-existent decoding method is used.

Example transformation rule

e_set("oss_file",res_oss_file(endpoint, ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), bucket, 'test/test.txt', format='text', change_detect_interval=20, encoding='unicode'))

Error log

{
  "reason": "LookupError: unknown encoding: unicode"
}

Troubleshooting

Check the log event that caused the error. If the error message is unknown encoding: unicode, check whether the encoding parameter for file decoding is correct.

Solution

Change the value of the encoding parameter in the orchestration of the res_oss_file function to the correct decoding format.

# Assume that the correct decoding format is utf8.
e_set("oss_file",res_oss_file(endpoint, ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), bucket, 'test/test.txt', format='text', change_detect_interval=20,encoding='utf8'))

Endpoint error

The Endpoint is incorrect when you use the res_oss_file function.

The endpoint does not exist

Example transformation rule

e_set("oss_file",res_oss_file("https://oss-cn-asd.aliyuncs.com", ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), 'your bucket', 'test/test.txt', format='text', change_detect_interval=20))

Error log

message:  get_oss_bytes get errors:{'status': -2, 'x-oss-request-id': '', 'details': "RequestError: HTTPSConnectionPool(host='log-etl-staging.oss-cn-asd.aliyuncs.com', port=443)

Troubleshooting

Check the error log event. If the error message is -2 and the Message indicates a RequestError, check whether the endpoint exists.
Solution

Change the value of the endpoint parameter in the orchestration of the res_oss_file function to a correct value.

The OSS Endpoint does not match the bucket name

The authorized AccessKey has permissions to perform operations in the Hangzhou region, but the Endpoint is set to the Shanghai region.

Example transformation rule

e_set("oss_file",res_oss_file("https://oss-cn-shanghai.aliyuncs.com", ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), 'your bucket', 'test/test.txt', format='text', change_detect_interval=20))

Error log

message:  get_oss_bytes get errors:{'status': 403, 'x-oss-request-id': '5D7219353A90A2852B234D14', 'details': {'Code': 'AccessDenied', 'Message': 'The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.', 'RequestId': '5D7**14', 'HostId': 'log-**.oss-cn-shanghai.aliyuncs.com', 'Bucket': 'log-**', 'Endpoint': 'oss-cn-hangzhou.aliyuncs.com'}}

Troubleshooting

Check the log event that caused the error. If the error message is 403 and the Message field refers to the bucket and endpoint, verify that the endpoint and the AccessKey are in the same region.
Solution

In the res_oss_file function orchestration, set the encoding parameter to the correct decoding format.

Permission error

Incorrect AccessKey

The AccessKey is incorrect when you use the res_oss_file function.

Example transformation rule

e_set("oss_file",res_oss_file(endpoint, ak_id=res_local("AK_ID"),ak_key=res_local("AK_KEY"), 'your bucket', 'test/test.txt', format='text', change_detect_interval=20))

Error log

message: Exception: {'status': 403, 'x-oss-request-id': '5D***BEB6', 'details': {'Code': 'InvalidAccessKeyId', 'Message': 'The OSS Access Key Id you provided does not exist in our records.', 'RequestId': '5D4***BEB6', 'HostId': 'lo***g.oss-cn-hangzhou.aliyuncs.com', 'OSSAccessKeyId': 'LT***Ai'}}

Troubleshooting

Check the log event that caused the error. If the error message is 403 and the Message field is related to the AccessKey, check whether the AccessKey is correct.
Solution

Correct the AccessKey information in the orchestration of the res_oss_file function.

Bucket permission error

A bucket error occurs when you use the res_oss_file function.

Example transformation rule

e_set("oss_file",res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',ak_id=os.getenv("SLS_ACCESS_KEY_ID"),
                     ak_key=os.getenv("SLS_ACCESS_KEY_SECRET"),
                     bucket='log', file='test.txt',
                     change_detect_interval=30, encoding='utf8'))

Error log

message: Exception: {'status': 403, 'x-oss-request-id': '5D674CE8BE0EBC45166026C5', 'details': {'Code': 'AccessDenied', 'Message': 'You have no right to access this object because of bucket acl.', 'RequestId': '5D4***BEB6', 'HostId': 'log.oss-cn-hangzhou.aliyuncs.com'}}

Troubleshooting

Check the log event that caused the error. If the error message is 403 and the Message field is related to the bucket, check whether you have permissions to operate the bucket.
Solution

Correct the bucket information in the orchestration of the res_oss_file function.

The bucket does not exist

The specified bucket does not exist in OSS.

Example transformation rule

e_set("oss_file",res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',ak_id=os.getenv("SLS_ACCESS_KEY_ID"),
                     ak_key=os.getenv("SLS_ACCESS_KEY_SECRET"),
                     bucket='twiss', file='test.txt',
                     change_detect_interval=30, encoding='utf8'))

Error log

message: Exception: {'status': 404, 'x-oss-request-id': '5D75F6E9BB4097C678A381EF', 'details': {'Code': 'NoSuchBucket', 'Message': 'The specified bucket does not exist.', 'RequestId': '5D75F6E9BB4097C678A381EF', 'HostId': 'twiss.oss-cn-hangzhou.aliyuncs.com', 'BucketName': 'twiss'}}

Troubleshooting

Examine the error log event. If the error code is 403 and the Message field is related to the bucket, verify that the bucket exists.
Solution

Correct the bucket information in the orchestration of the res_oss_file function.

Error logs for scheduled file updates

The following error logs appear only in backend log records during scheduled file updates.

The following log indicates that the first attempt to retrieve the resource failed, possibly due to a network issue, and a retry is being prepared.
```
{
  "reason":"Failed to pull data from oss for the first time and it is preparing to re-pull ..."
}
```
The following log indicates that the backend performs updates at regular intervals, but the new resource function failed to retrieve data and is preparing to retry.
```
{
  "reason":"get_oss_byte get errors,begin retry ..."
}
```
The following log indicates that the retry failed and another retry is being prepared. A maximum of three retries are performed.
```
{
  "reason":"get_oss_byte get errors,refresh_interval ..."
}
```