Check data integrity by using CRC-64

Usage notes

• PutObject, AppendObject, PostObject, and MultipartUploadPart return a CRC-64 value. After an upload completes, compare the server-returned CRC-64 with the locally computed value.

• MultipartComplete returns the CRC-64 value of the entire object only if every part has a CRC-64 value. For example, if a part was uploaded before CRC-64 support was available, no CRC-64 value is returned for the part or the complete object.

• GetObject, HeadObject, and GetObjectMeta return the CRC-64 value of the object if one exists. After a GetObject call, compare the server-returned CRC-64 with the locally computed value.

  Note

  A GET request with a Range header still returns the CRC-64 value of the entire object.

• Objects created by copy operations such as CopyObject and UploadPartCopy may not have a CRC-64 value.

Examples

The following Python example shows how to verify data integrity by using CRC-64.

import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
import crcmod
import random
import string
from oss2.models import PartInfo

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
# Specify the name of the bucket. 
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'yourBucketName')
# Create a CRC-64 verification function. 
do_crc64 = crcmod.mkCrcFun(0x142F0E1EBA9EA3693, initCrc=0, xorOut=0xffffffffffffffff, rev=True)

# Verify the CRC-64 values and display the result. 
def check_crc64(local_crc64, oss_crc64, msg="check crc64"):
    if local_crc64 != oss_crc64:
        print("{0} check crc64 failed. local:{1}, oss:{2}.".format(msg, local_crc64, oss_crc64))
        return False
    else:
        print("{0} check crc64 ok.".format(msg))
        return True

# Generate a random string of the specified length. 
def random_string(length):
    return ''.join(random.choice(string.ascii_lowercase) for i in range(length))

# Generate a random string with a length of 1024. 
content = random_string(1024)

# Specify the path of the object. 
key = 'normal-key'

# Verify data integrity for a PutObject operation. 
result = bucket.put_object(key, content)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(content)))
check_crc64(local_crc64, oss_crc64, "put object")

# Verify data integrity for a GetObject operation. 
result = bucket.get_object(key)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(result.resp.read()))
check_crc64(local_crc64, oss_crc64, "get object")

# Verify data integrity for parts and the final object. 
part_info_list = []
key = "multipart-key"
result = bucket.init_multipart_upload(key)
upload_id = result.upload_id
part_1 = random_string(1024 * 1024)
result = bucket.upload_part(key, upload_id, 1, part_1)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_1)))
# Check the integrity of part 1. 
check_crc64(local_crc64, oss_crc64, "upload_part object 1")
part_info_list.append(PartInfo(1, result.etag, len(part_1)))
part_2 = random_string(1024 * 1024)
result = bucket.upload_part(key, upload_id, 2, part_2)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_2)))
# Check the integrity of part 2. 
check_crc64(local_crc64, oss_crc64, "upload_part object 2")
part_info_list.append(PartInfo(2, result.etag, len(part_2)))
result = bucket.complete_multipart_upload(key, upload_id, part_info_list)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_2), do_crc64(oss2.to_bytes(part_1))))
# Check whether the final object uploaded to OSS is identical to the local file. 
check_crc64(local_crc64, oss_crc64, "complete object")

Supported OSS SDKs

The following table lists OSS SDK support for CRC-64.