On-premises and third-party cloud database backups-Data Management-阿里云帮助中心

Data disaster recovery supports Alibaba Cloud databases, self-managed databases on ECS instances, on-premises databases, and databases on third-party cloud platforms.

Usage notes

If the source database contains tables with poorly designed schemas, large tables, or large fields, a backup instance with low specifications can exhaust its resources and trigger backup exceptions. To prevent these issues, select a backup instance with higher specifications.

Procedure

Product automatic backup

Purchase a backup schedule (logical backup)

Log in to DMS 5.0.
In the top navigation bar, choose Security and disaster recovery (DBS) > Data Disaster Recovery (DBS) > Disaster Recovery Data Source.

Note
If you use the DMS console in simple mode, move the pointer over the icon in the upper-left corner of the DMS console and choose All Features > Security and disaster recovery (DBS) > Data Disaster Recovery (DBS) > Disaster Recovery Data Source.
Select a region at the top of the page. On the On-premises and third-party cloud databases tab, click Product Automatic Backup, and then click the ID of the target data source to open its details page.
On the Backup Policy page, click Configure Backup Policy.
On the Select Backup Schedule page, click Purchase Backup Schedule to open the Database Backup Service (DBS) buy page.

Configure the following parameters and click Buy Now in the lower-right corner.

Parameter	Description
Product type	Select Backup Schedule (Subscription).
Backup schedule region	Select the region where you want to store the backup data. Note Make sure that the backup schedule and the ECS instance are in the same region.
Data source type	Currently, only MySQL is supported.
Specification	The backup schedule specifications that you want to use. Higher specifications offer higher backup and restoration performance. DBS supports the following backup schedule specifications: micro, small, medium, large, and xlarge. The xlarge specification type provides extra large specifications without an upper limit on the amount of backup data. Note If your database instance, such as a production database, requires a high-performance backup schedule for fast backup and restore tasks, we recommend selecting the xlarge or large specification. If high backup and restore performance is not a priority, you can calculate your requirements to select the most cost-effective backup schedule specification. For more information, see How to select a backup schedule specification. If the databases and tables that you want to back up involve issues such as unreasonable schemas, large tables, and large fields, the resources of a backup instance of lower specifications may be insufficient to complete the backup. As a result, a backup error occurs. We recommend that you purchase a backup instance of higher specifications to prevent backup errors.
Backup method	Select logical backup.
Storage space	You do not need to specify a storage capacity when purchasing the backup schedule. You are charged based on the amount of data stored in the DBS built-in storage. For more information about billing, see Storage fees.
Resource group	Select the default or a custom resource group to facilitate the management of backup schedules.
Quantity	Select the desired quantity. A separate backup schedule is required for each database instance. For example, to back up database instance A and database instance B, you must purchase two backup schedules.
Subscription duration	Select the subscription duration for the backup schedule.

On the Confirm Order page, review the order details, read and agree to the Terms of Service, and then click Pay.

After you complete the payment, return to the Select Backup Schedule page from Step 5 and click Paid. The newly purchased backup schedule appears in the list.

Configure a backup policy

On the Select Backup Schedule page, select the backup schedule that you want to configure and click Next.
On the Select Databases and Tables page, select the databases or tables that you want to back up, click to move them to the Selected Objects box, and then click Submit.
After the settings are submitted, go to the Backup Policy page. On the Logical Backup tab, click Start to begin the backup.

After you click Start, the system immediately initiates a full backup task and an incremental backup task.

Note
If you need to perform other operations first, such as modifying the backup policy, you can choose not to start the backup immediately. The system will automatically start the backup at the time specified in the backup policy.

User-provided automatic backup

Important

Currently, only MySQL 5.5 data sources are supported.
Currently, only the China (Hangzhou) region is supported.

Configure backup source and upload files

Log in to DMS 5.0.
In the top navigation bar, choose Security and disaster recovery (DBS) > Data Disaster Recovery (DBS) > Disaster Recovery Data Source.

Note
If you use the DMS console in simple mode, move the pointer over the icon in the upper-left corner of the DMS console and choose All Features > Security and disaster recovery (DBS) > Data Disaster Recovery (DBS) > Disaster Recovery Data Source.
Select a region at the top of the page. On the On-premises and third-party cloud databases tab, choose how to add a data source based on its type.

Click Add Data Source. In the dialog box that appears, configure the following parameters, select the target backup schedule, and then click Next.

Parameter	Description
Data source name	Enter a descriptive name.
Engine type	Database engine type. Only MySQL is supported.
Engine version	Select the version of the database engine.
Engine parameters	{"lower_case_table_names":1}

If no backup schedule is available, click Purchase Backup Schedule to go to the DBS buy page and purchase one.

	Description
Product type	Select Backup Schedule (Subscription). The pay-as-you-go billing method is not supported.
Backup schedule region	Select the region where you want to store the backup data.
Data source type	Select MySQL.
Specification	Select xmicro. For information about the free data quota included with this specification, see Backup schedule specifications.
Backup method	Select physical backup.
Storage space	You do not need to specify a storage capacity when purchasing the backup schedule. You are charged based on the amount of data stored in the DBS built-in storage. For more information about billing, see Storage fees.
Resource group	Select the default or a custom resource group to facilitate the management of backup schedules.
Quantity	Select the desired quantity. A separate backup schedule is required for each database instance. For example, to back up database instance A and database instance B, you must purchase two backup schedules.
Subscription duration	Select the subscription duration for the backup schedule.

Upload the backup set to the specified bucket. For details, see Upload data for user-provided automatic backups.
After the upload is complete, click OK.

View backup information

On the User-provided automatic backup tab, click the ID of the target data source.

On the data source details page, the left-side navigation pane provides options including Basic Information, Backup Policy, Backup Data, Recovery Drill, and Task Management. By default, the Basic Information tab is displayed, where you can view fields such as Data Source ID, Data Source Name, Resource Group ID, Data Source Type, Backup Method, Access Time, Engine Type, Data Source Authorization Status, Source Database Endpoint, and Port.

Configure a backup policy

On the User-provided automatic backup tab, find the target data source and click View Backup Policy in the Actions column.

In the Backup Policy dialog box, configure Full backup retention period (7 to 7,300 days, default: 30), Log backup retention period (7 to 7,300 days, default: 7), and Snapshot retention period (1 to 7,300 days, default: 7), and then click OK.
After you complete the configuration, click OK.

View and download backup data

On the User-provided automatic backup tab, click the ID of the target data source.
In the left-side navigation pane, click Backup Data.

Note
After you run the data upload script and create the data source, the backup management system synchronizes any newly generated backup sets to the Backup Data page.
Find the target backup set and click Download in the Actions column to download it to your local computer.

Create a restore task

Note

To create a restore task, a generated backup set must exist on the Backup Data page, and its status must be Completed.

On the User-provided automatic backup tab, click the ID of the target data source.

In the left-side navigation pane, click Backup Data, and then click Create Restore Task to configure the restore parameters.

Parameter	Description
Restore task name	Enter a descriptive name for the restore task.
Restore to	Select the destination for the restored instance. By default, data is restored to a new RDS instance.
Database location	Select the location of the database. The default value is RDS.
Instance region	Select the region where the instance resides. Currently, only China (Hangzhou) is supported.
VPC	Select the VPC.
vSwitch	Select the vSwitch.
Instance series	Select the instance series for the restored instance.
Instance specification	Select the instance specification for the restored instance.
Storage space	Select the required storage space.
Restore mode	Select a restore mode. Currently, only point-in-time restore is supported.
Restore time	Select a point in time. For the available time range, see the information next to the Restore mode parameter.

After configuring the task, click Submit. A restore task is created for the selected point in time, and its details appear on the restore task page.

View restore tasks

On the User-provided automatic backup tab, click the ID of the target data source.
In the left-side navigation pane, choose Task Management > Restore Tasks.
Click the instance ID in the Restore Result column to go to the Basic Information page of the restored RDS instance.

View recovery drills

On the User-provided automatic backup tab, click the ID of the target data source.

In the left-side navigation pane, click Recovery Drill.

Recovery drill overview

View the recovery drill overview, which includes Restore task success rate, Average restore duration, Data backup drill coverage, and Log backup drill coverage.

Parameter	Description
Restore task success rate	The success rate of restore tasks initiated within the filtered time range.
Average restore duration	The average time taken for successful restore tasks within the filtered time range.
Data backup drill coverage	The percentage of data backups from open-source MySQL or RDS for MySQL instances that have undergone a recovery drill within the filtered time range.
Log backup drill coverage	The percentage of log backups from open-source MySQL or RDS for MySQL instances that have undergone a recovery drill within the filtered time range.

Recovery drill timeline

The timeline displays recovery drill details. You can click a point on the timeline to view information for that specific time.

Recovery drill details for backup data

Click Data Backup to view recovery drill details for data backups. Click the instance ID in the Drill Result column to go to the Basic Information page of the restored RDS instance.

Click Log Backup to view the list of recovery drill details for log backups.

Upload user backup data

Before you begin

Configure a data source and obtain its ID. For more information, see Add a data source.
Create a RAM user, grant the RAM user permissions to manage the required Alibaba Cloud service, and obtain an AccessKey ID and an AccessKey secret. For more information, see Create a RAM user and Manage RAM user permissions.

Dependencies

Command-line tools: Bash and Python 3.
Python libraries: oss2, alibabacloud_openapi_util, alibabacloud_tea_openapi, and alibabacloud_tea_util.

## Install the Alibaba Cloud OpenAPI SDK.
pip3 install --upgrade pip
pip3 install -i https://mirrors.aliyun.com/pypi/simple/ oss2 alibabacloud_openapi_util alibabacloud_tea_openapi alibabacloud_tea_util

Complete upload scripts

The backup data upload scripts consist of two parts:

Bash script

This is the entry point for script execution. Replace the placeholder parameters with your actual values as needed.

boot_backup.sh: This script simulates creating a local full backup by using Xtrabackup.

#!/bin/bash
## The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations is a high-risk practice. We recommend that you create and use a RAM user for API access or routine O&M.
## For security reasons, do not hardcode your AccessKey ID and AccessKey secret in your code. This can lead to credential leaks. Instead, store them in a configuration file or use environment variables.
AK=<YOUR_ACCESS_KEY_ID>
SK=<YOUR_ACCESS_KEY_SECRET>
DBS_API=dbs-api.<your-region-id>.aliyuncs.com
DATASOURCE_ID=<your-data-source-id>
## Get the current path of the script.
BASE_PATH=$(cd `dirname $0`; pwd)
TS=`date +%Y%m%d_%H%M%S`
XTRA_DATA_DIR=~/tool/xtrabackup_data/$TS
mkdir -p $XTRA_DATA_DIR
## Run the Xtrabackup command and redirect error logs to the xtrabackup.log file.
~/innobackupex --defaults-file=/etc/my.cnf --user='<your-db-account>' --password='<your-db-password>' --host='<your-db-host>' --port=<your-db-port> --socket='/var/lib/mysql/mysql.sock' --parallel=4 $XTRA_DATA_DIR/ 2>$XTRA_DATA_DIR/xtrabackup.log
## Get the current Xtrabackup backup directory.
backup_dir=`ls $XTRA_DATA_DIR | grep -v xtrabackup.log | head -n1`
echo -e "\033[33mexecute innobackupex success, backup_dir: $backup_dir" && echo -n -e "\033[0m" && chmod 755 $XTRA_DATA_DIR/$backup_dir
## Package the directory into a .tar.gz file.
cd $XTRA_DATA_DIR/$backup_dir && tar -czvf ../$backup_dir.tar.gz . && file ../$backup_dir.tar.gz
echo -e "\033[33mpackage to $backup_dir.tar.gz" && echo -n -e "\033[0m" && sleep 2
## Call the Python script to upload the backup file and register its metadata.
python3 $BASE_PATH/upload_and_register_backup_set.py --access_key_id $AK --access_key_secret $SK --endpoint $DBS_API --datasource_id $DATASOURCE_ID --region_code=cn-hangzhou --data_type=FullBackup --file_path=$XTRA_DATA_DIR/$backup_dir.tar.gz --xtrabackup_log_path=$XTRA_DATA_DIR/xtrabackup.log

Python scripts

upload_and_register_backup_set.py: This script uploads full backup and log backup data, parses the corresponding metadata, and registers the backup set metadata with Database Backup Service (DBS).

# -*- coding: utf-8 -*-
# This file is auto-generated, don't edit it. Thanks.
import os
import argparse
import re
import time
import json
from datetime import datetime
import oss2
from alibabacloud_openapi_util.client import Client as OpenApiUtilClient
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_tea_openapi.client import Client as OpenApiClient
from alibabacloud_tea_util import models as util_models
import xtrabackup_info_parser
import xtrabackup_log_parser
def init_command_args():
    parser = argparse.ArgumentParser(description="A sample command-line parser.")
    parser.add_argument("--access_key_id", help="Your Alibaba Cloud AccessKey ID.")
    parser.add_argument("--access_key_secret", help="Your Alibaba Cloud AccessKey secret.")
    parser.add_argument("--endpoint", help="The API endpoint for your service.")
    parser.add_argument("--region_code", help="The region code of your data source.")
    parser.add_argument("--datasource_id", help="The ID of your data source.")
    parser.add_argument("--data_type", help="The type of the backup set: FullBackup or LogBackup.")
    parser.add_argument("--file_path", help="The file path of the backup set.")
    parser.add_argument("--xtrabackup_info_path", help="The path to the xtrabackup_info file.")
    parser.add_argument("--xtrabackup_log_path", help="The path to the xtrabackup.log file.")
    parser.add_argument("--begin_time", help="The start time of the binlog.")
    parser.add_argument("--end_time", help="The end time of the binlog.")
    args = parser.parse_args()
    if args.access_key_id:
        print(f"access_key_id: ************")
    if args.access_key_secret:
        print(f"access_key_secret: ************")
    if args.endpoint:
        print(f"endpoint: {args.endpoint}")
    if args.region_code:
        print(f"region_code: {args.region_code}")
    if args.datasource_id:
        print(f"datasource_id: {args.datasource_id}")
    if args.data_type:
        print(f"data_type: {args.data_type}")
    if args.file_path:
        print(f"file_path: {args.file_path}")
    if args.xtrabackup_info_path:
        print(f"xtrabackup_info_path: {args.xtrabackup_info_path}")
    if args.xtrabackup_log_path:
        print(f"xtrabackup_log_path: {args.xtrabackup_log_path}")
    if args.begin_time:
        print(f"begin_time: {args.begin_time}")
    if args.end_time:
        print(f"end_time: {args.end_time}")
    print('\n')
    return args
def date_to_unix_timestamp(date_str):
    # Converts a date string to a Unix timestamp in milliseconds.
    dt_obj = datetime.strptime(date_str, "%Y-%m-%d %H:%M:%S")
    timestamp_seconds = time.mktime(dt_obj.timetuple())
    return int(timestamp_seconds) * 1000
def create_oss_client(params):
    # Authentication details for Alibaba Cloud OSS.
    access_key_id = params['AccessKeyId']
    access_key_secret = params['AccessKeySecret']
    security_token = params['SecurityToken']
    bucket_name = params['BucketName']
    endpoint = params['OssEndpoint']
    # Initialize the OSS client.
    auth = oss2.StsAuth(access_key_id, access_key_secret, security_token)
    return oss2.Bucket(auth, endpoint, bucket_name)
def upload_oss_file(oss_client, file_path, object_key):
    """
    Uploads a large file to OSS using multipart upload.
    :param oss_client: The OSS client instance.
    :param file_path: The path to the local file.
    :param object_key: The object key in OSS, which serves as the file name.
    """
    # Set the part size in bytes. Default is 5 MB.
    part_size = 1024 * 1024 * 5
    # Initiate the multipart upload.
    upload_id = oss_client.init_multipart_upload(object_key).upload_id
    # Open the file and read its content.
    with open(file_path, 'rb') as file_obj:
        parts = []
        while True:
            data = file_obj.read(part_size)
            if not data:
                break
            # Upload a part.
            result = oss_client.upload_part(object_key, upload_id, len(parts) + 1, data)
            parts.append(oss2.models.PartInfo(len(parts) + 1, result.etag))
        # Complete the multipart upload.
        oss_client.complete_multipart_upload(object_key, upload_id, parts)
class OssUploader:
    def __init__(self, access_key_id, access_key_secret, endpoint, region_code, datasource_id):
        self.access_key_id = access_key_id
        self.access_key_secret = access_key_secret
        self.endpoint = endpoint
        self.region_code = region_code
        self.datasource_id = datasource_id
        config = open_api_models.Config(access_key_id, access_key_secret)
        # For more information about endpoints, see https://api.aliyun.com/product/Dbs.
        config.endpoint = endpoint
        self.client = OpenApiClient(config)
    """
    Registers backup set metadata.
    """
    def configure_backup_set_info(self, req_param):
        params = open_api_models.Params(
            # The name of the API operation.
            action='ConfigureBackupSetInfo',
            # The version of the API operation.
            version='2021-01-01',
            # The protocol.
            protocol='HTTPS',
            # The HTTP method.
            method='POST',
            auth_type='AK',
            style='RPC',
            # The path of the API operation.
            pathname='/',
            # The format of the request body.
            req_body_type='json',
            # The format of the response body.
            body_type='json'
        )
        # Runtime options.
        runtime = util_models.RuntimeOptions()
        request = open_api_models.OpenApiRequest(
            query=OpenApiUtilClient.query(req_param)
        )
        # The response is a map that contains the response body, headers, and HTTP status code.
        print(f"ConfigureBackupSetInfo request: {req_param}")
        data = self.client.call_api(params, request, runtime)
        print(f"ConfigureBackupSetInfo response: {data}")
        return data['body']['Data']
    """
    Gets OSS upload information.
    """
    def describe_bak_datasource_storage_access_info(self, req_param):
        params = open_api_models.Params(
            # The name of the API operation.
            action='DescribeBakDataSourceStorageAccessInfo',
            # The version of the API operation.
            version='2021-01-01',
            # The protocol.
            protocol='HTTPS',
            # The HTTP method.
            method='POST',
            auth_type='AK',
            style='RPC',
            # The path of the API operation.
            pathname='/',
            # The format of the request body.
            req_body_type='json',
            # The format of the response body.
            body_type='json'
        )
        # Runtime options.
        runtime = util_models.RuntimeOptions()
        request = open_api_models.OpenApiRequest(
            query=OpenApiUtilClient.query(req_param)
        )
        # The response is a map that contains the response body, headers, and HTTP status code.
        print(f"DescribeBakDataSourceStorageAccessInfo request: {req_param}")
        data = self.client.call_api(params, request, runtime)
        print(f"DescribeBakDataSourceStorageAccessInfo response: {data}")
        return data['body']['Data']
    def _fetch_oss_access_info(self, params):
        info = self.describe_bak_datasource_storage_access_info({
            'RegionId': params['RegionId'],
            'DataSourceId': params['DataSourceId'],
            'RegionCode': params['RegionCode'],
            'BackupType': params['BackupType'],
            'BackupSetId': params['BackupSetId']
        })
        return info['OssAccessInfo']
    def upload_and_register_backup_set(self, file_path, data_type, extra_meta):
        filename = os.path.basename(file_path)
        params = {'BackupMode': 'Automated', 'BackupMethod': 'Physical', 'RegionId': self.region_code,
                  'RegionCode': self.region_code, 'DataSourceId': self.datasource_id, 'BackupSetName': filename,
                  'ExtraMeta': extra_meta, 'BackupType': data_type, 'UploadStatus': 'WaitingUpload'}
        # Register the backup set for the first time to obtain a backup set ID.
        data = self.configure_backup_set_info(params)
        params['BackupSetId'] = data['BackupSetId']
        print(f"------ configure_backup_set_info success: {file_path}, {data_type}, {params['BackupSetId']}, WaitingUpload\n")
        # Upload the file to OSS.
        oss_info = self._fetch_oss_access_info(params)
        oss_client = create_oss_client(oss_info)
        upload_oss_file(oss_client, file_path, oss_info['ObjectKey'])
        print(f"------ upload_oss_file success: {file_path}, {data_type}, {params['BackupSetId']}\n")
        # Mark the backup set as uploaded.
        params['UploadStatus'] = 'UploadSuccess'
        self.configure_backup_set_info(params)
        print(f"------ configure_backup_set_info success: {file_path}, {data_type}, {params['BackupSetId']}, UploadSuccess\n")
if __name__ == '__main__':
    args = init_command_args()
    uploader = OssUploader(args.access_key_id, args.access_key_secret,
                           args.endpoint, args.region_code, args.datasource_id)
    # Construct extra_meta differently for full backups and log backups.
    extra_meta = '{}'
    if args.data_type == 'FullBackup':
        obj = {}
        if args.xtrabackup_log_path is not None:
            obj = xtrabackup_log_parser.analyze_slave_status(logpath=args.xtrabackup_log_path)
        elif args.xtrabackup_info_path is not None:
            parser = xtrabackup_info_parser.ExtraMetaParser(file_path=args.xtrabackup_info_path)
            obj = parser.get_extra_meta()
        extra_meta = {'BINLOG_FILE': obj.get('BINLOG_FILE'),
                      'version': obj.get("SERVER_VERSION"),
                      'dataBegin': date_to_unix_timestamp(obj.get("START_TIME")),
                      'dataEnd': date_to_unix_timestamp(obj.get("END_TIME")),
                      'consistentTime': int(date_to_unix_timestamp(obj.get("END_TIME")) / 1000)}
        extra_meta = json.dumps(extra_meta)
    elif args.data_type == 'LogBackup':
        obj = {'dataBegin': date_to_unix_timestamp(args.begin_time),
               'dataEnd': date_to_unix_timestamp(args.end_time)}
        extra_meta = json.dumps(obj)
    print(f"get extra meta json string: {extra_meta}")
    # Upload the data and register the backup set metadata.
    uploader.upload_and_register_backup_set(file_path=args.file_path, data_type=args.data_type, extra_meta=extra_meta)

xtrabackup_info_parser.py: This script parses metadata from the xtrabackup_info file.

# -*- coding: utf-8 -*-
# This file is auto-generated, don't edit it. Thanks.
import re
import json
class ExtraMetaParser:
    def __init__(self, file_path):
        self.file_path = file_path
        pass
    def _parse_xtrabackup_info(self):
        config_data = {}
        with open(self.file_path, 'r') as file:
            for line in file:
                line = line.strip()
                if line and not line.startswith('#'):
                    key, value = line.split('=', 1)
                    config_data[key.strip()] = value.strip()
        return config_data
    def get_extra_meta(self):
        config_data = self._parse_xtrabackup_info()
        print(f"xtrabackup_info: {config_data}")
        binlog_pos = config_data.get("binlog_pos")
        pattern = f"filename '(.*)', position (.*)"
        match = re.search(pattern, binlog_pos)
        return {'BINLOG_FILE': match.group(1),
                'SERVER_VERSION': config_data.get("server_version"),
                'START_TIME': config_data.get("start_time"),
                'END_TIME': config_data.get("end_time")}

xtrabackup_log_parser.py: This script parses metadata from the xtrabackup.log file generated during the backup process.

#!/usr/bin/python
# coding:utf8
import io
import re
import sys
from datetime import datetime
from six import binary_type, text_type
def parse_date_part(date_part, time_part):
    """
    Parses and returns a full datetime string from the given date and time parts.
    """
    # Get the first two digits of the current year (the century).
    current_century = datetime.now().strftime("%Y")[:2]
    year_short = date_part[:2]
    # Construct the full year.
    year_full = current_century + year_short
    date_full = year_full + date_part[2:]
    datetime_str = date_full + " " + time_part
    dt = datetime.strptime(datetime_str, "%Y%m%d %H:%M:%S")
    formatted_datetime = dt.strftime("%Y-%m-%d %H:%M:%S")
    return text_type(formatted_datetime)
def analyze_slave_status(logpath=None):
    slave_status = {}
    start_time_pattern = (
        r"(\d{6}) (\d{2}:\d{2}:\d{2}) .*Connecting to MySQL server host:"
    )
    """
        240925 17:46:58 completed OK!
        240925 02:22:58 innobackupex: completed OK!
        240925 02:22:58 xtrabackup: completed OK!
    """
    end_time_pattern = r"(\d{6}) (\d{2}:\d{2}:\d{2}) .*completed OK!"
    with io.open(logpath, "rb") as fp:
        lines = fp.read().splitlines()
        for i in reversed(range(len(lines))):
            line = lines[i]
            if isinstance(line, binary_type):
                line = line.decode("utf-8")
            m = re.search(start_time_pattern, line)
            if m:
                # Extract the date and time parts.
                date_part = m.group(1)
                time_part = m.group(2)
                slave_status["START_TIME"] = parse_date_part(date_part, time_part)
                continue
            m = re.search(r"Using server version (\S*)", line)
            if m:
                slave_status["SERVER_VERSION"] = text_type(m.group(1))
                continue
            m = re.search("MySQL binlog position:", line)
            if m:
                binlog_line = line
                m = re.search(r"filename '(\S*)'", binlog_line)
                if m:
                    slave_status["BINLOG_FILE"] = text_type(m.group(1))
                m = re.search(r"position (\d+)", binlog_line)
                m2 = re.search(r"position '(\d+)'", binlog_line)
                if m:
                    try:
                        slave_status["BINLOG_POS"] = int(m.group(1))
                    except ValueError:
                        pass
                elif m2:
                    try:
                        slave_status["BINLOG_POS"] = int(m2.group(1))
                    except ValueError:
                        pass
                continue
            m = re.search("consistent_time (\d+)", line)
            if m:
                try:
                    slave_status["CONSISTENT_TIME"] = int(m.group(1))
                except ValueError:
                    pass
                continue
            m = re.search(end_time_pattern, line)
            if m:
                date_part = m.group(1)
                time_part = m.group(2)
                slave_status["END_TIME"] = parse_date_part(date_part, time_part)
                continue
    return slave_status
if __name__ == "__main__":
    logpath = sys.argv[1]
    slave_status = analyze_slave_status(logpath)
    print(slave_status)

Bash script workflow

Configure your AccessKey ID and AccessKey secret, the Database Backup Service (DBS) OpenAPI endpoint, and the data source ID.

#!/bin/bash
## The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations is a high-risk practice. We recommend that you create and use a RAM user for API access or routine O&M.
## For security reasons, do not hardcode your AccessKey ID and AccessKey secret in your code. This can lead to credential leaks. Instead, store them in a configuration file or use environment variables.
AK=<YOUR_ACCESS_KEY_ID>
SK=<YOUR_ACCESS_KEY_SECRET>
DBS_API=dbs-api.<your-region-id>.aliyuncs.com
DATASOURCE_ID=<your-data-source-id>

Run the Xtrabackup command to save the backup data to a specified directory and redirect the error log to the xtrabackup.log file.
```
## Get the current path of the script.
BASE_PATH=$(cd `dirname $0`; pwd)
TS=`date +%Y%m%d_%H%M%S`
XTRA_DATA_DIR=~/tool/xtrabackup_data/$TS
mkdir -p $XTRA_DATA_DIR
## Run the Xtrabackup command and redirect error logs to the xtrabackup.log file.
~/innobackupex --defaults-file=/etc/my.cnf --user='<your-db-account>' --password='<your-db-password>' --host='<your-db-host>' --port=<your-db-port> --socket='/var/lib/mysql/mysql.sock' --parallel=4 $XTRA_DATA_DIR/ 2>$XTRA_DATA_DIR/xtrabackup.log
```
- The script later parses the xtrabackup.log file to obtain metadata for the full backup set. The metadata includes the backup start time, end time, consistency point, and the corresponding binlog file name.
- Limitation:
  
  The Xtrabackup command currently does not support the compression and encryption parameters (--compress, --compress-threads, --encrypt, --encrypt-key-file, and --encrypt-threads). Remove these parameters before you run the command.

Compress and package the full backup data directory into a .tar.gz file.

## Get the current Xtrabackup backup directory.
backup_dir=`ls $XTRA_DATA_DIR | grep -v xtrabackup.log | head -n1`
echo -e "\033[33mexecute innobackupex success, backup_dir: $backup_dir" && echo -n -e "\033[0m" && chmod 755 $XTRA_DATA_DIR/$backup_dir
## Package the directory into a .tar.gz file.
cd $XTRA_DATA_DIR/$backup_dir && tar -czvf ../$backup_dir.tar.gz . && file ../$backup_dir.tar.gz
echo -e "\033[33mpackage to $backup_dir.tar.gz" && echo -n -e "\033[0m" && sleep 2

Limitation: DBS accepts backup data only in the following file formats:
- tar: The directory is packaged into a .tar archive.
- tar.gz: The directory is packaged into a .tar archive and then compressed using Gzip.
Usage notes:
- Before you run the tar command, run the chmod 755 command to set the directory permissions.
- Before running the tar command, navigate to the root of the directory that you want to package.

Run the upload_and_register_backup_set.py script to upload the backup data and register its metadata.

## Call the Python script to upload the backup file and register its metadata.
python3 $BASE_PATH/upload_and_register_backup_set.py --access_key_id $AK --access_key_secret $SK --endpoint $DBS_API --datasource_id $DATASOURCE_ID --region_code=cn-hangzhou --data_type=FullBackup --file_path=$XTRA_DATA_DIR/$backup_dir.tar.gz --xtrabackup_log_path=$XTRA_DATA_DIR/xtrabackup.log

Parameter	Description
`--access_key_id`	Your Alibaba Cloud AccessKey ID.
`--access_key_secret`	Your Alibaba Cloud AccessKey secret.
`--endpoint`	The API endpoint for Database Backup Service (DBS).
`--datasource_id`	The ID of the data source in DBS.
`--region_code`	The ID of the region where the data source resides.
`--data_type`	The type of backup data. Valid values: `FullBackup` for a full backup and `LogBackup` for a log backup.
`--file_path`	The path to the full backup data file.
`--xtrabackup_log_path`	The path to the `xtrabackup.log` file generated during the full backup.

Python script workflow

The main function is the entry point.

Depending on the data_type parameter (FullBackup or LogBackup), the script constructs the required extra_meta information for metadata registration.
Note
- BINLOG_FILE: The name of the binlog file.
- dataBegin: The backup start time, as a Unix timestamp in milliseconds.
- dataEnd: The backup end time, as a Unix timestamp in milliseconds.
- consistentTime: The consistency point, as a Unix timestamp in seconds.
- Format of extra_meta for a full backup:
```
{
  'BINLOG_FILE':'mysql-bin.001',
  'version':'5.5',
  'dataBegin':17274********,
  'dataEnd':17274********,
  'consistentTime':17274********
}
```
- Format of extra_meta for a log backup:
```
{
  'dataBegin':17274********,
  'dataEnd':17274********
}
```

Call the OssUploader.upload_and_register_backup_set method to upload the backup data and register its metadata.

if __name__ == '__main__':
    args = init_command_args()
    uploader = OssUploader(args.access_key_id, args.access_key_secret,
                           args.endpoint, args.region_code, args.datasource_id)
    # Construct extra_meta differently for full backups and log backups.
    extra_meta = '{}'
    if args.data_type == 'FullBackup':
        obj = {}
        if args.xtrabackup_log_path is not None:
            obj = xtrabackup_log_parser.analyze_slave_status(logpath=args.xtrabackup_log_path)
        elif args.xtrabackup_info_path is not None:
            parser = xtrabackup_info_parser.ExtraMetaParser(file_path=args.xtrabackup_info_path)
            obj = parser.get_extra_meta()
        extra_meta = {'BINLOG_FILE': obj.get('BINLOG_FILE'),
                      'version': obj.get("SERVER_VERSION"),
                      'dataBegin': date_to_unix_timestamp(obj.get("START_TIME")),
                      'dataEnd': date_to_unix_timestamp(obj.get("END_TIME")),
                      'consistentTime': int(date_to_unix_timestamp(obj.get("END_TIME")) / 1000)}
        extra_meta = json.dumps(extra_meta)
    elif args.data_type == 'LogBackup':
        obj = {'dataBegin': date_to_unix_timestamp(args.begin_time),
               'dataEnd': date_to_unix_timestamp(args.end_time)}
        extra_meta = json.dumps(obj)
    print(f"get extra meta json string: {extra_meta}")
    # Upload the data and register the backup set metadata.
    uploader.upload_and_register_backup_set(file_path=args.file_path, data_type=args.data_type, extra_meta=extra_meta)

The upload process within the OssUploader.upload_and_register_backup_set method.

class OssUploader:
    def __init__(self, access_key_id, access_key_secret, endpoint, region_code, datasource_id):
        self.access_key_id = access_key_id
        self.access_key_secret = access_key_secret
        self.endpoint = endpoint
        self.region_code = region_code
        self.datasource_id = datasource_id
        config = open_api_models.Config(access_key_id, access_key_secret)
        # For more information about endpoints, see https://api.aliyun.com/product/Dbs.
        config.endpoint = endpoint
        self.client = OpenApiClient(config)
    """
    Registers backup set metadata.
    """
    def configure_backup_set_info(self, req_param):
        params = open_api_models.Params(
            # The name of the API operation.
            action='ConfigureBackupSetInfo',
            # The version of the API operation.
            version='2021-01-01',
            # The protocol.
            protocol='HTTPS',
            # The HTTP method.
            method='POST',
            auth_type='AK',
            style='RPC',
            # The path of the API operation.
            pathname='/',
            # The format of the request body.
            req_body_type='json',
            # The format of the response body.
            body_type='json'
        )
        # Runtime options.
        runtime = util_models.RuntimeOptions()
        request = open_api_models.OpenApiRequest(
            query=OpenApiUtilClient.query(req_param)
        )
        # The response is a map that contains the response body, headers, and HTTP status code.
        print(f"ConfigureBackupSetInfo request: {req_param}")
        data = self.client.call_api(params, request, runtime)
        print(f"ConfigureBackupSetInfo response: {data}")
        return data['body']['Data']
    """
    Gets OSS upload information.
    """
    def describe_bak_datasource_storage_access_info(self, req_param):
        params = open_api_models.Params(
            # The name of the API operation.
            action='DescribeBakDataSourceStorageAccessInfo',
            # The version of the API operation.
            version='2021-01-01',
            # The protocol.
            protocol='HTTPS',
            # The HTTP method.
            method='POST',
            auth_type='AK',
            style='RPC',
            # The path of the API operation.
            pathname='/',
            # The format of the request body.
            req_body_type='json',
            # The format of the response body.
            body_type='json'
        )
        # Runtime options.
        runtime = util_models.RuntimeOptions()
        request = open_api_models.OpenApiRequest(
            query=OpenApiUtilClient.query(req_param)
        )
        # The response is a map that contains the response body, headers, and HTTP status code.
        print(f"DescribeBakDataSourceStorageAccessInfo request: {req_param}")
        data = self.client.call_api(params, request, runtime)
        print(f"DescribeBakDataSourceStorageAccessInfo response: {data}")
        return data['body']['Data']
    def _fetch_oss_access_info(self, params):
        info = self.describe_bak_datasource_storage_access_info({
            'RegionId': params['RegionId'],
            'DataSourceId': params['DataSourceId'],
            'RegionCode': params['RegionCode'],
            'BackupType': params['BackupType'],
            'BackupSetId': params['BackupSetId']
        })
        return info['OssAccessInfo']
    def upload_and_register_backup_set(self, file_path, data_type, extra_meta):
        filename = os.path.basename(file_path)
        params = {'BackupMode': 'Automated', 'BackupMethod': 'Physical', 'RegionId': self.region_code,
                  'RegionCode': self.region_code, 'DataSourceId': self.datasource_id, 'BackupSetName': filename,
                  'ExtraMeta': extra_meta, 'BackupType': data_type, 'UploadStatus': 'WaitingUpload'}
        # Register the backup set for the first time to obtain a backup set ID.
        data = self.configure_backup_set_info(params)
        params['BackupSetId'] = data['BackupSetId']
        print(f"------ configure_backup_set_info success: {file_path}, {data_type}, {params['BackupSetId']}, WaitingUpload\n")
        # Upload the file to OSS.
        oss_info = self._fetch_oss_access_info(params)
        oss_client = create_oss_client(oss_info)
        upload_oss_file(oss_client, file_path, oss_info['ObjectKey'])
        print(f"------ upload_oss_file success: {file_path}, {data_type}, {params['BackupSetId']}\n")
        # Mark the backup set as uploaded.
        params['UploadStatus'] = 'UploadSuccess'
        self.configure_backup_set_info(params)
        print(f"------ configure_backup_set_info success: {file_path}, {data_type}, {params['BackupSetId']}, UploadSuccess\n")