Troubleshoot upload callback errors

About upload callbacks

After a file is successfully uploaded, OSS can send a callback to your callback server. You can enable this feature by including the required callback parameters in your upload request. The API operations that support upload callbacks are PutObject, PostObject, and CompleteMultipartUpload. For more information, see Upload Callback and Callback API.

Scenarios

• Notifications

  A typical application of upload callbacks is to allow an authorized third party to specify callback parameters when uploading a file. After the upload is complete, OSS sends a callback request to the callback server, which then records the upload information.

• Processing, review, and statistics

  After the callback server receives a callback request, it processes, reviews, or collects statistics on the uploaded file.

Data flow

The following figure shows the OSS upload callback flow.

The OSS upload callback flow is described as follows:

SDK/PostObject

During an upload, you can set callback parameters to specify the callback server URL, the data to send, and the data format. The callback server uses system variables, such as bucket and object, for context. You can use custom variables for any other context information.

An upload callback includes the following parameters:

There are two ways to include upload callback parameters in an upload request:

• Include the parameters in the x-oss-callback header. This is the common and recommended method.

• Include the parameters in the callback query string.

The value of x-oss-callback or callback is generated based on the following rules:

Callback := Base64(CallbackJson)
CallbackJson := '{' CallbackUrlItem, CallbackBodyItem [, CallbackHostItem, CallbackBodyTypeItem] '}' 
CallbackUrlItem := '"'callbackUrl'"' ':' '"'CallbackUrlValue'"'
CallbackBodyItem := '"'callbackBody'"' ':' '"'CallbackBodyValue'"'
CallbackHostItem := '"'callbackHost'"' ':' '"'CallbackHostValue'"'
CallbackBodyTypeItem := '"'callbackBodyType'"' : '"'CallbackBodyType'"'
CallbackBodyType := application/x-www-form-urlencoded | application/json

The following code shows examples of the CallbackJson value:

{
    "callbackUrl" : "http://abc.com/test.php",
    "callbackHost" : "oss-cn-hangzhou.aliyuncs.com",
    "callbackBody" : "{\"bucket\":${bucket}, \"object\":${object},\"size\":${size},\"mimeType\":${mimeType},\"my_var\":${x:my_var}}",
    "callbackBodyType" : "application/json"
}

or

{
    "callbackUrl" : "http://abc.com/test.php",
    "callbackBody" : "bucket=${bucket}&object=${object}&etag=${etag}&size=${size}&mimeType=${mimeType}&my_var=${x:my_var}"
}

System and custom variables

The callbackBody in the CallbackJson example includes variables such as ${bucket}, ${object}, and ${size}. These are system variables defined by OSS. During a callback, OSS replaces the system variables with their actual values. The following table describes the system variables defined by OSS.

The callbackBody in the CallbackJson example includes variables such as ${x:my_var}. These are custom variables. During a callback, OSS replaces the custom variables with their values. You can define and include custom variable values in an upload request in one of the following two ways:

• Include the variables in the x-oss-callback-var header. This is the common and recommended method.

• Include the variables in the callback-var query string.

The value of x-oss-callback-var or callback-var is generated based on the following rules:

CallbackVar := Base64(CallbackVarJson)
CallbackVarJson := '{' CallbackVarItem [, CallbackVarItem]* '}'
CallbackVarItem := '"''x:'VarName'"' : '"'VarValue'"'

The following code shows an example of the CallbackVarJson value:

{
    "x:my_var1" : "value1",
    "x:my_var2" : "value2"
}

SDK usage examples

Some SDKs, such as the Java and JavaScript SDKs, encapsulate the preceding steps. For other SDKs, such as the Python, PHP, and C SDKs, you must use the preceding rules to generate the upload callback parameters and custom variables. The following table provides SDK usage examples.

PostObject usage examples

PostObject supports upload callbacks. Callback parameters are included in the callback form field, and custom variables are included in separate form fields. For more information, see PostObject.

The following example shows how to use PostObject:

Callback server

A callback server is an HTTP server that processes POST messages for callback requests sent from OSS. The URL of the callback server is the callbackUrl specified in the upload callback parameters. You can implement the processing logic on the callback server to record, review, process, and collect statistics on the uploaded data.

• Callback signature

  To confirm that a POST request is from an OSS upload callback, the callback server must verify the signature of the POST message. The callback server can also process the message directly without verifying the signature. However, to improve the security of the callback server, we recommend that you verify the message signature. For more information about the callback signature rules, see Callback signature.

  Note

  The OSS callback server examples provide an implementation for signature verification. We recommend that you use this code directly.

• Message processing

  The main logic of the callback server is to process callback requests from OSS. Note the following:

  • The callback server must process POST requests from OSS.

  • The timeout period for OSS callbacks is 5 seconds. The callback server must complete processing and return a response within 5 seconds.

  • The message body that the callback server returns to OSS must be in JSON format.

  • The callback server uses your own logic. OSS provides examples but does not provide specific business logic implementations.

• Implementation examples

  The following is an example implementation of a callback server:

  Language	Example	How to run
  Java	AppCallbackServer.zip	Decompress the package and run java -jar oss-callback-server-demo.jar 9000.
  PHP	callback-php-demo.zip	Deploy and run in an Apache environment.
  Python	callback_app_server.py.zip	Decompress the package and run python callback_app_server.py.
  Ruby	oss-callback-server	Run ruby aliyun_oss_callback_server.rb.

Debugging steps

Debugging an upload callback involves two parts: the upload client and the callback server. We recommend that you first debug the client-side upload and then debug the callback server. After you debug both parts separately, you can run the complete upload callback.

• Debug the client

  To debug the client, you can use the callback server that is provided by OSS at http://oss-demo.aliyuncs.com:23450. This URL is the callbackUrl parameter. This callback server only verifies the signature of the callback request and does not process the request. For callback requests with a valid signature, it returns {"Status":"OK"}. For requests with an invalid signature, it returns 400 Bad Request. For non-POST requests, it returns 501 Unsupported method. For the code of the example callback server, see callback_app_server.py.zip.

• Debug the callback server

  The callback server is an HTTP server that supports POST requests. You can modify the examples that are provided by OSS or implement your own server. The following table provides callback server examples from OSS:

  Language	Example	How to run
  Java	AppCallbackServer.zip	Decompress the package and run java -jar oss-callback-server-demo.jar 9000.
  PHP	callback-php-demo.zip	Deploy and run in an Apache environment.
  Python	callback_app_server.py.zip	Decompress the package and run python callback_app_server.py.
  C#	callback-server-dotnet.zip	Compile the program and run aliyun-oss-net-callback-server.exe 127.0.0.1 80.
  Go	callback-server-go.zip	Compile the program and run aliyun_oss_callback_server.
  Ruby	oss-callback-server	Run ruby aliyun_oss_callback_server.rb.

  You can debug the callback server using cURL commands. The following commands may be useful:

  # To send a POST request with the message body 'object=test_obj' to the callback server, run the following command: 
  curl -d "object=test_obj"  http://oss-demo.aliyuncs.com:23450 -v
  # To send a POST request with the content of the 'post.txt' file as the message body to the callback server, run the following command: 
  curl -d @post.txt  http://oss-demo.aliyuncs.com:23450 -v
  # To send a POST request with the content of the 'post.txt' file as the message body and include a specified 'Content-Type' header, run the following command:
  curl -d @post.txt -H "Content-Type: application/json" http://oss-demo.aliyuncs.com:23450 -v

  Note

  • When you debug the callback server, you can ignore signature verification at first because it is difficult to simulate the signature function using cURL.

  • The OSS examples provide the signature verification feature. We recommend that you use it directly.

  • We recommend that the callback server has a logging feature to record all received messages. This feature facilitates debugging and tracking.

  • After the callback server correctly processes a callback request, it must return a 200 status code, not another 20x status code.

  • The message body that the callback server returns to OSS must be in JSON format, and the Content-Type must be set to application/json.

Common errors and causes

• InvalidArgument

  <Error>
    <Code>InvalidArgument</Code>
    <Message>The callback configuration is not json format.</Message>
    <RequestId>587C79A3DD373E2676F73ECE</RequestId>
    <HostId>bucket.oss-cn-hangzhou.aliyuncs.com</HostId>
    <ArgumentName>callback</ArgumentName>
    <ArgumentValue>{"callbackUrl":"8.8.8.8:9090","callbackBody":"{"bucket":${bucket},"object":${object}}","callbackBodyType":"application/json"}</ArgumentValue>
  </Error>

  Note

  The callback parameters are incorrectly set, or the parameter format is invalid. A common error is that the callback parameters in ArgumentValue are not in a valid JSON format. In JSON, \ and " are escape characters. For example, "callbackBody":"{"bucket":${bucket},"object":${object}}" must be changed to "callbackBody":"{\"bucket\":${bucket},\"object\":${object}}". For specific SDKs, see the corresponding upload callback examples. For more information, see SDK usage examples.

  Escaped character	Unescaped characters
  \\	\\\\
  \"	\\\"
  \b	\\b
  \f	\\f
  \n	\\n
  \r	\\r
  \t	\\t

• CallbackFailed

  The following examples show common `CallbackFailed` errors:

  • Example 1:

    <Error>
      <Code>CallbackFailed</Code>
      <Message>Response body is not valid json format.</Message>
      <RequestId>587C81A125F797621829923D</RequestId>
      <HostId>bucket.oss-cn-hangzhou.aliyuncs.com</HostId>
    </Error>

    Note

    The message body that the callback server returns to OSS is not in JSON format. You can run curl -d "<Content>" <CallbackServerURL> -v or capture packets to check the content. We recommend that you use Wireshark to capture packets in Windows and use the tcpdump command to capture packets in Linux. Examples of invalid response bodies include OK and \357\273\277{"Status":"OK"}, which contains a bill of materials (BOM) header with the three bytes ef bb bf.

  • Example 2:

    <Error>
      <Code>CallbackFailed</Code>
      <Message>Error status : -1.OSS can not connect to your callbackUrl, please check it.</Message>
      <RequestId>587C8735355BE8694A8E9100</RequestId>
      <HostId>bucket.oss-cn-hangzhou.aliyuncs.com</HostId>
    </Error>

    Note

    If the processing time of the callback server exceeds 5 seconds, OSS considers the request a timeout. We recommend that you implement asynchronous processing for the callback server to ensure that a result is returned to OSS within 5 seconds.

  • Example 3:

    <Error>
      <Code>CallbackFailed</Code>
      <Message>Error status : -1 8.8.8.8:9090 reply timeout, cost:5000ms, timeout:5000ms (err -4, errno115)</Message>
      <RequestId>587C8D382AE0B92FA3EEF62C</RequestId>
      <HostId>bucket.oss-cn-hangzhou.aliyuncs.com</HostId>
    </Error>

    Note

    The processing time of the callback server exceeds 5 seconds, which OSS considers a timeout.

  • Example 4:

    <Error>
      <Code>CallbackFailed</Code>
      <Message>Error status : 400.</Message>
      <RequestId>587C89A02AE0B92FA3C7981D</RequestId>
      <HostId>bucket.oss-cn-hangzhou.aliyuncs.com</HostId>
    </Error>

    Note

    The status code of the message that the callback server returns to OSS is 400. You can check the processing logic of the callback server.

  • Example 5:

    <Error>
      <Code>CallbackFailed</Code>
      <Message>Error status : 502.</Message>
      <RequestId>587C8D382AE0B92FA3EEF62C</RequestId>
      <HostId>bucket.oss-cn-hangzhou.aliyuncs.com</HostId>
    </Error>

    Note

    The callback server is not started, the CallbackUrl is missing from the upload callback parameters, or the network connection between OSS and the callback server is interrupted. We recommend that you deploy the callback server on an Elastic Compute Service (ECS) instance in the same internal network as OSS. This deployment saves traffic costs and ensures network quality.

• The returned body is not in JSON format

  For example:

  

  This error occurs in the following two situations:

  • The body that the application server returns to OSS is not in JSON format. See the following figure:

    If `Resp_body` is not in a valid JSON format, OSS reports the error. This cause is usually obvious. However, less obvious causes can exist. For example, an exception is thrown during processing on the application server. As a result, the server returns stack information to OSS instead of the expected response.

  • The body that the application server returns to OSS contains a BOM header.

    This error is common in application servers that are written in PHP. Because PHP returns a BOM header, the body that OSS receives contains three extra bytes and does not conform to the JSON format. This causes OSS to report the error. If you capture packets on the application server, you can see the following information.

    

    In the preceding figure, the three bytes ef bb bf are the BOM header.

    Note

    When the application server returns a response to OSS, you must remove the BOM header.

• Incorrect status

  This type of error is common, such as 502 and 400. For example:

  

  Note

  • A status code of 400, 404, or 403 indicates that the application server returned that HTTP status to OSS. Under normal circumstances, the application server must return a 200 status code to OSS.

  • A 502 status code is returned because the web service is not enabled on the application server and is not listening for callback requests from OSS.

• Timeout

  For example:

  

  Note

  For security reasons, OSS waits for a callback response for only 5 seconds. If no response is returned after 5 seconds, OSS disconnects from the application server and returns a timeout error to the client. You can ignore the IP address in the error message.