Big data cloud migration and inspection service description
1. Service description
1.1. Big data migration service description
As cloud computing adoption grows, an increasing number of businesses are migrating their computing platforms to the cloud. The Big Data Platform Migration Service offers technical support for designing and implementing migration solutions, which includes both data migration and job migration. We help you develop a technical solution to migrate and transform your big data platform, ensuring a fast and efficient migration of data and jobs while maintaining data consistency. This service addresses the following customer pain points:
Lack of familiarity with cloud platform products, experience in big data migration design, and the ability to create migration solutions.
Lack of familiarity with the overall data migration process and hands-on experience. This includes process control, migration implementation, ensuring data consistency, and troubleshooting migration tools. This results in a need for technical guidance and support.
Lack of familiarity with the overall job migration process and hands-on experience. This includes process control, migration implementation, and troubleshooting migration tools. This results in a need for technical guidance and support.
This service includes four sub-services. You can purchase them based on your business needs:
Big data migration solution design service (optional)
Basic big data migration implementation service (choose either the data migration or job migration sub-service)
Supplementary big data migration implementation service (optional)
Big data migration tool support service (optional)
Service notes:
A single purchase of this Statement of Work (SOW) includes only one of the following three services: migration consulting, data migration, or job migration.
The workload assessment for migration implementation, which includes data and job migration, assumes that migration resources are available on schedule.
The cost of this service package covers resource usage only, not the cost of migrated resources.
Unless otherwise specified, the default migration period is two months or longer.
This service package supports the following migration scenarios. A single purchase of a service package covers only one of the following scenarios:
Open source Hive/Hadoop to Alibaba Cloud EMR-Hive
Open source Spark to Alibaba Cloud EMR-Spark
Open source Hive/Hadoop to Alibaba Cloud MaxCompute
Open source Spark to Alibaba Cloud MaxCompute (non-pyodps)
Open source Flink to Alibaba Cloud Flink
ClickHouse to Alibaba Cloud ClickHouse
On-cloud/on-premises MaxCompute to on-cloud/on-premises MaxCompute
For scenarios not listed above, a specific project analysis is required.
The measurement of 'task volume' mentioned in this document depends on many factors, such as the scenario, task type, and scheduling. You can consult with the service package Project Director (PD) for a calculation based on your specific situation.
1.2. Big data inspection service description
Based on the best practices for operations and maintenance (O&M) and inspection of Alibaba Cloud's big data platform, this service provides monitoring and collection configuration for cloud-native and EMR commercial open source platforms. We provide proactive O&M inspections and health checks based on the O&M cycle and generate a health analysis report for your current data business. This service addresses pain points such as unreasonable big data product configurations, low resource utilization, potential architecture availability issues, frequent production job failures, and the inability to detect problems in advance.
Service notes:
This service supports the following products:
Open source EMR
Alibaba Cloud EMR
Alibaba Cloud MaxCompute/DataWorks/Hologres
For products not listed above, a specific project analysis is required.
2. Scope of service
The scope for each sub-service is as follows:
2.1. Scope of migration solution design service
This service provides a complete technical solution for migrating your business system's data links to the Alibaba Cloud platform. We survey your current business systems, data link architecture, and cloud adoption requirements. Then, we design a complete migration data link that meets your needs. The solution covers incremental migration, full migration, merging incremental and full data, data consistency verification, and SQL transformation for both data and jobs. The design covers the data link from your business system to cloud products, the required migration tools, implementation instructions for key steps, and cutover instructions.
The technical parameters for the solution design have the following constraints:
Number of source clusters | Number of scheduling components | Data volume | Number of tables | Task volume |
<=4 | <=2 | 10 PB | 10,000 | 3,000 |
2.2. Scope of basic migration implementation service - data migration
Based on the migration plan, we provide training on tool deployment and usage, technical support, and problem resolution during the data migration of your big data platform. This service ensures a smooth implementation process and helps you complete the data migration cutover.
The migration scale has the following constraints:
Number of source clusters | Number of scheduling components | Data volume | Number of tables | Task volume |
<=2 | First-party | 1,000 |
2.3. Scope of basic migration implementation service - job migration technical support
Based on the migration plan, we provide training on tool deployment and usage, technical support, and problem resolution during the job migration of your big data platform. This service ensures a smooth implementation process and helps you complete the job migration cutover.
The migration scale has the following constraints. The calculation of task volume is affected by multiple factors, such as the heterogeneity of migrated components and whether jobs need to be refactored. The final volume is determined during the pre-sales survey.
Number of source clusters | Number of scheduling components | Data volume | Number of Tables | Task volume (for reference only) |
<=2 | <=1 | Determined by pre-sales survey |
2.4. Scope of migration implementation supplementary service
As a supplement to the basic big data platform migration implementation service, this service extends the basic service based on workload. It also provides premium services beyond the basic offerings, such as on-site support, technical Q&A, and technical training.
The premium services have the following constraints. The calculation of task volume is affected by multiple factors, such as the heterogeneity of migrated components and whether jobs need to be refactored. The final volume is determined during the pre-sales survey.
Service type | Service content | Scope of service |
Workload extension for data or job migration technical support | Extends the migration workload not covered by the basic service package for the same scenario. | Consistent with solution design, data migration, and job migration. |
Technical Q&A | Corresponds to the landing usage of a single big data product, including resource assessment, user permission design, and network allocation. |
|
Technical training | Corresponds to usage training for a single big data product, not exceeding 2 person-days for the course and 3 person-days for post-course Q&A. |
|
On-site support | Provides on-site support during major customer events or projects (such as platform transformation, platform migration, or sales promotions), not exceeding 10 person-days of on-site support. |
|
2.5. Scope of big data platform inspection service
No. | Main work content | Detailed description |
Requirements phase | Current state analysis | Survey the customer's current big data architecture and data links. Investigate the scope of the customer's big data platform inspection needs and daily pain points. |
Requirements confirmation | Understand the customer's needs, create a checklist of key metrics and inspection goals, and confirm the requirements with the customer. | |
System inspection | Inspection implementation | Deploy inspection and collection services using tools such as Prometheus or self-developed tools to obtain the list of metrics from the requirements phase. These metrics include cluster operating environment (CPU, memory, disk, network, etc.) and big data service operating information (service CPU and memory usage, intermediate directories, etc.). |
Analysis and diagnosis | Based on the collected metrics and configuration information, generate a cluster health inspection report and diagnose issues such as abnormal resource usage and service exceptions. | |
Report generation | Report generation | 1) Evaluate the inspection results from multiple dimensions, such as environment configuration, runtime performance, storage rationality, and security, and create a report document. 2) Provide problem diagnosis and optimization suggestions for high-priority or high-risk scenarios, such as resource consumption by big data cluster configuration tasks, high-frequency inefficient computations, small file accumulation, and partition abnormalities. |
2.6. Scope of big data migration tool support service
Based on the migration plan, we provide training on the deployment and usage of the LHM big data migration tool, technical support, and problem resolution during your big data platform migration. This service ensures a smooth implementation process.
The migration scale has the following constraints:
Number of source clusters | Number of scheduling components | Data volume | Number of tables | Task volume (for reference only) |
<=2 | <=2 | <=5 PB | <=5,000 |
The scope of service is as follows:
Phase name | Main work content | Task details |
Requirements survey | Requirements survey | Survey the customer's technology stack and application scenarios before and after the big data migration. This includes migration data sources, migration engines, network links, scheduling engines, and typical SQL code styles. |
Migration tool proof of concept (POC) | Support the customer in deploying the tool and testing simple scenarios to confirm if it meets their migration needs. | |
Migration support | Tool usage support | Assist the customer with deploying the LHM big data migration tool and establishing data links. Provide functional Q&A and usage support during the tool's use. Identify and help the product team resolve abnormal issues encountered with LHM during migration, including product bugs and usage problems. |
Tool best practices | Support best practices for big data migration using LHM, including performance tuning and migration task planning, to ensure efficient migration work. | |
Tool support | Tool support | Support the continuous and stable operation of the tool, especially ensuring tool stability during critical phases such as dual-run and cutover. |
2.7. This service does not include the following
This service does not include application migration, like-for-like database migration, application refactoring, application design, or database logical schema design.
Migration of application systems not explicitly identified as "in scope".
Migration of data not explicitly identified as "in scope".
Design and setup of cloud platform infrastructure and environments.
Communication infrastructure and components required for implementation.
Installation, configuration, or support for electrical, electronic communication, and cabling infrastructure and components.
Installation and configuration of non-IT equipment and desktop devices, such as desks, chairs, printers, air conditioners, UPS, and tapes.
This service does not include application refactoring. We are only responsible for data and job migration within the defined scope. You are responsible for resolving compatibility issues that arise from the platform replacement.
Alibaba Cloud is not responsible for the specific project schedule. Alibaba Cloud is not liable for delays caused by the customer.
3. Prerequisites
You must request this service at least 15 calendar days in advance. This allows Alibaba Cloud to assess your business goals, evaluate the feasibility of the timeline, and confirm whether to accept the service request.
If your request involves a large volume of resources, you must submit it one month in advance. The final timeline will be negotiated based on a supply chain assessment.
After Alibaba Cloud accepts the service request, you must provide the site, equipment, necessary non-production environments, remote access channels, permissions, and clear business objectives, such as the migration scope and product solution. This is required for Alibaba Cloud to deliver the service.
You must review the project implementation plan created by Alibaba Cloud and confirm the specific implementation plan in writing, for example, by email. To ensure the timely completion of this service, you must not reject a confirmed technical proposal or solution without a valid technical reason.
You must implement the specific migration service at the time agreed upon by both parties.
4. Division of responsibilities
4.1. Customer and Alibaba Cloud
Project phase | Customer | Party B |
Project preparation | 1. Appoint a project manager with the appropriate skills and experience to be the primary contact for Alibaba Cloud. This person will be responsible for planning, coordinating, supervising, and controlling the project implementation, along with escalating issues and risks. They must also have the authority to make decisions on all aspects of this project on behalf of the customer. 2. Work with Alibaba Cloud personnel to confirm all matters in the project preparation phase. For more information, see Section 2.1 Project Preparation of this SOW. 3. Prepare the office environment and approve access permissions for personnel. 4. Manage your project team members, communicate with personnel involved in each phase, and secure their commitment and time for the project. 5. You can manage the members of the Party A project group. | 1. Assign an experienced project manager to manage the project, onboard and manage the Party B project team, and communicate with the Party A project manager. 2. Propose solutions and plans for all matters in the project preparation phase, confirm them with your project manager, and document them in writing. For more information, see Section 2.1 Project Preparation of this SOW. |
Current state analysis phase | 1. Organize research interviews with key users according to the project and interview plans. 2. Provide an overview of your existing business operations by completing the requirements survey form from Alibaba Cloud. This information is used for the assessment and must include details such as the components used, the types and number of data sources, the number and types of migration jobs, job execution methods, the target Alibaba Cloud components for the migration, the data migration method (full or incremental), the job migration method, and consistency verification standards. 3. Confirm the cloud adoption strategy and risk governance policy provided by Alibaba Cloud. 4. Deploy the data discovery and collection tool provided by Alibaba Cloud, and upload the collected results for assessment and analysis. 5. Designate a person to review deliverables from Alibaba Cloud and provide feedback and confirmation. | 1. Provide an interview plan and assess the existing system architecture and deployment architecture based on the interview results. 2. Propose a cloud adoption strategy and risk governance strategy based on the survey and assessment results, and reach an agreement with you. 3. Assist you in analyzing the data migration assessment results. 4. Ensure the final deliverables meet the acceptance criteria based on your feedback. |
Solution design phase | 1. Assist the provider in selecting and designing a data architecture. 2. Collaborate with the service provider to design the data migration plan. 3. Review the SQL code to confirm the scope and plan for job refactoring based on the service provider's recommendations. 4. Appoint an acceptance officer to review Alibaba Cloud's deliverables and provide feedback and confirmation for acceptance. | 1. Select components and design the architecture based on the scope of service and your specific scenario. 2. Design the data migration solution and recommend suitable migration tools based on your current system, cloud products, data volume, and business characteristics. 3. Analyze the compatibility between source and target jobs and provide suggestions for job refactoring and SQL code inspection. 4. Ensure the final deliverables meet the acceptance criteria based on your feedback. |
Solution validation phase | 1. Based on the job compatibility analysis and SQL refactoring suggestions, refactor one sample job and perform business validation. 2. Validate the data migration solution. | 1. Assist and guide you in activating or purchasing cloud resources and in setting up and configuring the basic infrastructure. 2. Provide relevant technical support during the job refactoring process. 3. Assist you in validating the data migration solution by providing Q&A, tool training, and technical support during the migration process. 4. In this phase, only one sample is selected to validate the consulting solution. |
Migration implementation phase | 1. Based on the job compatibility analysis and refactoring suggestions, execute implementation tasks such as job code refactoring. 2. Execute data migration and cutover drills. 3. Prepare for business cutover based on the results of application testing and cutover drills. | 1. Assist and guide you in activating or purchasing cloud resources and in setting up and configuring the basic infrastructure. 2. Provide relevant technical support for job refactoring. 3. Assist you with data migration by providing Q&A, tool training, and technical support during the migration process. |
Cutover and support | 1. Confirm the business cutover plan for the Alibaba Cloud data platform and assign personnel to coordinate the cutover. 2. Clean up test data and confirm that production resources are ready. 3. Complete the system cutover within the planned time window and validate the business tests after the cutover. 4. If issues arise that are not related to Alibaba Cloud systems or services, you must assist Alibaba Cloud in coordinating with third-party personnel to resolve the problems. 5. Appoint an O&M contact to receive the post-launch operational notes from Alibaba Cloud. 6. Appoint an acceptance officer to review Alibaba Cloud's deliverables and provide feedback and confirmation for acceptance. | 1. Assist you in preparing for the system cutover and reach a consensus with you on the system cutover plan. 2. Provide technical support and problem resolution related to the computing platform during the cutover drill for the business systems that you designate. 3. Assist you with the system cutover and provide technical support, troubleshooting, and solutions related to the Alibaba Cloud computing platform during the process. 4. After the system cutover is complete, provide O&M troubleshooting and offer fixes or temporary solutions for issues within the project scope. 5. Conduct an O&M handover with your O&M contact. |
Inspection implementation | 1. Cooperate with Alibaba Cloud to implement the inspection configuration. This includes product activation, deployment, and necessary configuration collaboration. | 1. Implement the inspection configuration according to the scope of service, covering the metric requirements identified in the survey phase. 2. Collect inspection metric results, build a data analysis model, and generate an inspection report. |
Inspection report | 1. Confirm acceptance. | 1. Produce the results report and provide necessary suggestions. |
4.2. Completion criteria
The service is considered complete if one of the following conditions is met:
The workload defined in Section 2. Scope of service is completed and accepted by you.
If a project is paused or stopped for reasons attributable to the customer, and Alibaba Cloud has already provided 20 person-days of service within the specified timeframe, the project is considered complete.
5. Service items
Alibaba Cloud provides the following services to meet your business objectives:
5.1. Solution design
Main work content | Detailed description |
Current state analysis | 1. Survey relevant information about the customer's big data systems within the specified migration scope. This includes current components, data source types, number of data sources, number of migration jobs, job types, and job execution methods. 2. Corresponding to the pre-sales solution, determine the information of the Alibaba Cloud big data components at the migration destination (such as specifications, versions, and network connectivity). Also, determine the data migration method (full/incremental), job migration method, migration tools, and data consistency verification standards. |
Data migration design | Design the data migration solution for the [XXX] project based on the results of the detailed requirements survey.
|
Job migration design | Design the job migration solution for the [XXX] project based on the results of the detailed requirements survey.
|
Key point validation and cutover plan design | 1. Validate key points in the solution based on project requirements, such as the introduction of new migration tools or data volumes exceeding normal ranges. 2. Design the cutover plan, including descriptions of work steps and risk points for dual-running clusters. |
5.2. Data migration
Main work content | Content description |
Migrating existing data | Provide support for full data migration and validation. |
Incremental data migration | Provide support for incremental data migration and validation. |
Cutover drill | Assist in designing the cutover plan and conduct cutover drills to ensure the plan is feasible. |
Cutover implementation | Assist with the final cutover implementation for cloud migration. |
5.3. Job migration
Main work content | Content description |
Job migration support | Provide support for job migration and SQL job compatibility refactoring. |
Data consistency verification support | Provide technical support for data consistency verification during the migration process. |
Cutover drill support | Assist in designing the cutover plan and conduct cutover drills to ensure the plan is feasible. |
Cutover implementation support | Assist with the final cutover implementation for cloud migration. |
5.4. Platform inspection
Main work content | Detailed description |
Current state analysis | Survey the customer's current big data architecture and data links. Investigate the scope of the customer's big data platform inspection needs and daily pain points. |
Requirements confirmation | Understand the customer's needs, create a checklist of key metrics and inspection goals, and confirm the requirements with the customer. |
Inspection implementation | Deploy inspection and collection services using tools such as Prometheus or self-developed tools to obtain the list of metrics from the requirements phase. These metrics include cluster operating environment (CPU, memory, disk, network, etc.) and big data service operating information (service CPU and memory usage, intermediate directories, etc.). |
Analysis and diagnosis | Based on the collected metrics and configuration information, generate a cluster health inspection report and diagnose issues such as abnormal resource usage and service exceptions. |
Report generation | 1. Evaluate the inspection results from multiple dimensions, such as environment configuration, runtime performance, storage rationality, and security, and create a report document. 2. Provide problem diagnosis and optimization suggestions for high-priority or high-risk scenarios, such as resource consumption by big data cluster configuration tasks, high-frequency inefficient computations, small file accumulation, and partition abnormalities. |
5.5. Tool support
Main work content | Task details |
Requirements survey | Survey the customer's technology stack and application scenarios before and after the big data migration. This includes migration data sources, migration engines, network links, scheduling engines, and typical SQL code styles. |
Migration tool proof of concept (POC) | Support the customer in deploying the tool and testing simple scenarios to confirm if it meets their migration needs. |
Tool usage support | Assist the customer with deploying the LHM big data migration tool and establishing data links. Provide functional Q&A and usage support during the tool's use. Identify and help the product team resolve abnormal issues encountered with LHM during migration, including product bugs and usage problems. |
Tool best practices | Support best practices for big data migration using LHM, including performance tuning and migration task planning, to ensure efficient migration work. |
Tool support | Support the continuous and stable operation of the tool, especially ensuring tool stability during critical phases such as dual-run and cutover. |
6. Service process
6.1. Big data migration service process
Application time limit: You must submit your request at least 15 calendar days before the service start date.
Requirements survey (data links, platform architecture, scope definition, plan creation) → Solution design (data migration plan, job migration plan, data validation plan, cutover plan) → Development and implementation (environment setup, solution validation, data migration, job migration, dual-run validation) → System cutover (monitoring and alerting, cutover preparation) → Go-live and maintenance (official launch, knowledge transfer, project acceptance)
The process described above is for the complete big data migration service package, which includes solution design, data migration, and job migration support. The actual service process depends on the specific service items that you purchase.
6.2. Big data platform inspection service process
Application deadline: You must submit your request at least 15 days before the service start date.
Project preparation → Requirements survey → System inspection → Report generation
6.3. Big data migration tool support service process
Application time limit: You must submit your request at least 15 calendar days before the service start date.
Project preparation → Requirements survey → Migration support → Tool support
7. Acceptance criteria
For the acceptance of the Big Data Migration Service, the provider must, based on the service purchased by the customer (solution design, data migration, or task migration), deliver either the service content specified in Section 2 (Scope of Service) and Section 3.2 (Service Items), which consists of the workload for migrating specified data and task volumes including any supplementary service volume, or a migration solution design for a specified scale. The provider must then submit the following acceptance deliverables for the customer to confirm.
Deliverable content | Deliverable | Deliverable type |
Migration solution design service | "Big Data Platform Migration Solution Design", "Big Data Platform Data Migration Solution Validation Issue Checklist" | Document |
Data migration service | "Big Data Platform Data Migration Implementation Report" | Document |
Job migration service | "Big Data Platform Job Migration Implementation Support Report" | Document |
As specified in the service scope agreement, the service provider must complete the service items in Section 3.2 and submit the following acceptance deliverables for customer confirmation as part of the big data platform inspection service.
Deliverable content | Deliverable | Deliverable type |
Migration solution design | "Big Data Cluster Information Summary", "Customer Requirements Checklist", "Big Data Metrics Checklist and Inspection Goals" | Document |
Inspection implementation | "Big Data Cluster Metrics Inspection Dashboard" | Report |
Report generation | "Big Data Cluster Inspection Report" | Document |
Project phase | Deliverable content | Deliverable | Deliverable type |
Requirements survey | Migration requirements survey | "Big Data Migration Solution (Based on LHM)" | Document |
Migration support | Migration content checklist | "Big Data Migration Tool Usage Report" | Document |
Tool support | Tool usage manual | Document |
8. Completion flag
The implementation is finished and your acceptance is complete.