Migration solution

The following figure shows the technical architecture.

The following figure shows the migration process.

The migration involves the following steps:

1. Create a workspace

   • Estimate resources

     Before you create a workspace for Realtime Compute for Apache Flink, you must estimate the required resources.

     The basic unit of measurement for a Flink workspace is the Compute Unit (CU). One CU is equal to 1 CPU core, 4 GiB of memory, and 20 GB of local storage. The local storage holds information such as logs and system checkpoints. A CU represents the computing power of the underlying real-time computing system. The CU usage of a Realtime Compute job depends on the queries per second (QPS) of its input data stream, its computing complexity, and the specific data distribution. You can estimate the number of resources to purchase based on your business scale and the computing power of Realtime Compute. The following table shows the processing power of 1 CU for real-time computing.

     Processing scenario	Processing power
     Simple stream stress testing such as filtering and cleansing operations.	1 CU can process 40,000 to 55,000 data records per second.
     Complex stream stress testing such as aggregate operations and complex user-defined function (UDF) calculations.	1 CU can process 5,000 to 10,000 data records per second.

     Note

     • The preceding computing power estimates apply only to the internal processing of Realtime Compute. They do not include external data reads and writes. The efficiency of external data reads and writes can affect the actual computing power. For example:

       • If Realtime Compute needs to read data from Simple Log Service (SLS), but SLS has a quota limit on API calls, the overall computing power of Realtime Compute is limited by the SLS quota.

       • If the RDS data storage that Realtime Compute references has connection or Transaction Per Second (TPS) limits, the throughput capacity of Realtime Compute is limited by the throttling of RDS.

     • If a job uses window functions, CU usage is higher than for simple jobs. You must purchase at least 4 CUs.

   • Create

     To migrate your self-managed Flink jobs, use an Alibaba Cloud account to create a Realtime Compute for Apache Flink workspace. To ensure network connectivity with upstream and downstream systems after migration, select the same VPC as your self-managed Flink cluster when you create the workspace. For more information about how to create a workspace, see Activate Realtime Compute for Apache Flink. For more information about billing, see Billing.

2. Migrate the job

   Modify the configurations in the package of the job on your self-managed Flink cluster. Then, you can manually upload the job package to the Realtime Compute for Apache Flink console and create the corresponding job. For more information about how to migrate different types of jobs, see Overview.

3. Verify data accuracy

   The quality of output data is critical for your business. As with any change to a real-time job, you must verify the data quality of the new job after migration. You can run the new job and the original job in parallel for a dual run. After a period, once the data comparison conditions are met, check if the output from the new job is consistent with the original job and meets your quality expectations. If the output data is identical, no further analysis is needed.

   

   Real-time jobs run 24/7. However, in most cases, the output data has periodic characteristics, which makes data comparison feasible. For example, an aggregation job calculates aggregate values on an hourly or daily basis, and a data transformation job partitions data by day. You can compare data based on the corresponding period. For example, after an hourly job has fully processed data for several hours, you can compare the processed hourly data. For a daily aggregate value, you typically need to wait until the new job has processed a full day of data before you can compare.

   Based on the generation cycle and data scale of the job output, use a suitable comparison method for your business. The following table describes the comparison methods.

   Data scale	Comparison
   Small to medium data scale	Compare the full data. For example, for data from window aggregation logic, compare the aggregation results for each key and each window one by one to verify their correctness.
   Large data scale	If a full data comparison is too costly or impractical, consider using sampling. For example, for data from window aggregation logic, randomly sample the aggregation results of some keys and windows and compare them to verify correctness. Ensure that your sampling method is reasonable to avoid gaps in comparison that could lead to misjudging data quality.

4. Verify business stability

   Business stability is as important as data quality. To be considered stable, a job must run smoothly for an extended period. A period of at least 7 days is recommended. During the stability observation period, enable monitoring and alerting at the same level as the original job. During this period, primarily observe the job's runtime processing latency, any abnormal failovers, and whether checkpoints are healthy. If the new job achieves the same or higher stability than the original job, the stability verification is complete.

5. Perform business migration

   After you confirm the data accuracy, business performance, and stability, you can begin the business migration. Replace the original job's sink table with the backup sink table from the new Flink job and make it available to your business users. Then, you can stop and unpublish the original production pipeline. This completes the migration.