Long-tail computing optimization-MaxCompute(MaxCompute)-阿里云帮助中心

Long tails may occur in JOIN operations and other computing jobs. Learn about common long-tail scenarios and how to resolve them.

Long tails are one of the most common issues in distributed computing. They are primarily caused by uneven data distribution, which results in different workloads across nodes. The entire job can complete only after the slowest node finishes processing its data.

To avoid overloading a single worker, distribute the work across multiple workers.

GROUP BY long tail

Causes

A GROUP BY key has a heavily skewed data distribution.

Solution

You can resolve this issue by using either of the following methods:

Rewrite the SQL statement to split the heavily skewed key by adding a random value (a technique known as salting). For example:
```
SELECT Key,COUNT(*) AS Cnt FROM TableName GROUP BY Key;
```
Without a combiner, the mapper shuffles data to the reducer, which performs the COUNT operation. The execution plan is mapper > reducer. To redistribute the work for the skewed key, rewrite the query as follows.
```
-- Assume that the skewed key is 'KEY001'.
SELECT a.Key
  , SUM(a.Cnt) AS Cnt
FROM (
  SELECT Key
    , COUNT(*) AS Cnt
FROM TableName
GROUP BY Key, 
  CASE 
    WHEN Key = 'KEY001' THEN Hash(Random()) % 50
    ELSE 0
   END
) a
GROUP BY a.Key;
```
The execution plan becomes mapper > reducer > reducer. Although this adds an extra stage, splitting the skewed key across two stages can reduce the overall execution time.

Note If data skew is not severe, this method adds an unnecessary reducer stage and may increase the overall execution time.
Resolve the long-tail issue by setting a system parameter.
```
set odps.sql.groupby.skewindata=true;
```
This setting enables a generic optimization. Because it does not analyze your business logic, the resulting execution plan may not be optimal. For the best performance, rewrite your SQL query based on your data distribution.

DISTINCT long tail

If a long tail occurs with the DISTINCT keyword, key splitting does not apply. Use the following alternative approach instead.

Solution

-- The original SQL statement, regardless of the case where uid is not specified.
SELECT COUNT(uid) AS Pv
    , COUNT(DISTINCT uid) AS Uv
FROM UserLog;

The preceding statement can be rewritten as follows:

SELECT SUM(PV) AS Pv
    , COUNT(*) AS UV
FROM (
    SELECT COUNT(*) AS Pv
      , uid
    FROM UserLog
    GROUP BY uid
) a;

This approach replaces DISTINCT with COUNT, distributing the workload across different reducers. The rewritten statement also benefits from GROUP BY optimizations, and the combiner participates in the computation, which significantly improves performance.

Long tail of dynamic partitions

Causes

The dynamic partition feature starts a reducer at the final stage to sort small files. If the data written through dynamic partitions is skewed, a long tail occurs.
In most cases, incorrect use of dynamic partitions causes long tails.

Solution

If you know the target partition, specify it explicitly when inserting data instead of using dynamic partitions.

Use combiners to handle long tails

Combiners are frequently used to handle long tails in MapReduce jobs. They reduce the amount of data shuffled from mappers to reducers, which significantly lowers network transmission overhead. MaxCompute SQL implements this optimization automatically.

Note A combiner is an optimization that runs on the map side. You must ensure that the result is the same whether or not the combiner is used. For example, in a WordCount job, the final result is the same whether a reducer receives two (key, 1) pairs or a combiner pre-aggregates them into a single (key, 2) pair. However, for an operation like calculating an average, a combiner cannot combine (key, 1) and (key, 2) into (key, 1.5) before the final reduction.

Optimize the system to handle long tails

Beyond combiners, MaxCompute includes built-in system-level optimizations. For example, the following logs show the (+N backups) indicator during job execution.

M1_Stg1_job0:0/521/521[100%] M2_Stg1_job0:0/1/1[100%] J9_1_2_Stg5_job0:0/523/523[100%] J3_1_2_Stg1_job0:0/523/523[100%] R6_3_9_Stg2_job0:1/1046/1047[100%] 
M1_Stg1_job0:0/521/521[100%] M2_Stg1_job0:0/1/1[100%] J9_1_2_Stg5_job0:0/523/523[100%] J3_1_2_Stg1_job0:0/523/523[100%] R6_3_9_Stg2_job0:1/1046/1047[100%] 
M1_Stg1_job0:0/521/521[100%] M2_Stg1_job0:0/1/1[100%] J9_1_2_Stg5_job0:0/523/523[100%] J3_1_2_Stg1_job0:0/523/523[100%] R6_3_9_Stg2_job0:1/1046/1047(+1 backups)[100%] 
M1_Stg1_job0:0/521/521[100%] M2_Stg1_job0:0/1/1[100%] J9_1_2_Stg5_job0:0/523/523[100%] J3_1_2_Stg1_job0:0/523/523[100%] R6_3_9_Stg2_job0:1/1046/1047(+1 backups)[100%]

In this example, 1,047 reducers are used. Of these, 1,046 have completed, but one remains running. When MaxCompute detects this, it automatically launches a backup reducer to process the same data. The result from whichever reducer finishes first is aggregated into the final result set.

Optimize business logic to handle long tails

The preceding optimization methods cannot address all long-tail scenarios. In some cases, you must optimize your business logic directly.

Noisy data can cause skew. For example, when you compute per-user access records by visitor ID, you must filter out crawler traffic. Otherwise, crawler data may produce a long tail. Identifying crawlers is increasingly difficult. Similarly, when joining on a field such as xxid, verify that the field is not empty.
Certain business scenarios are inherently skewed. For example, independent software vendors (ISVs) generate far more operation records than individual users. In such cases, apply dedicated analysis methods for high-volume accounts.
If data is unevenly distributed, avoid using constants as the DISTRIBUTE BY key to sort all data records.