RDS MySQL出现“The incident LOST_EVENTS occured on the master”报错

更新时间:
复制 MD 格式

本文主要针对RDS MySQL 5.65.7版本,解决主备复制因批量执行GRANTREVOKE权限时失败而引发的LOST_EVENTS中断问题。

现象描述

RDS MySQL主备复制中断,出现如下报错:

The incident LOST_EVENTS occured on the master. Message: REVOKE/GRANT failed while granting/revoking privileges in databases. 
Error_code: 1590
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: xxx
                  Master_User: replicator
                  Master_Port: 3070
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000005
          Read_Master_Log_Pos: 4838
               Relay_Log_File: slave-relay.000011
                Relay_Log_Pos: 50251
        Relay_Master_Log_File: mysql-bin.000004
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1590
                   Last_Error: The incident LOST_EVENTS occured on the master. Message: REVOKE/GRANT failed while granting/revoking privileges in databases.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 68321
              Relay_Log_Space: 57092

问题原因

RDS MySQL 5.65.7版本中,GRANTREVOKE 操作对多个用户的权限变更是非原子性的。如果其中一个用户因不存在或语法错误等原因导致操作失败,主库会认为整个操作不可靠,为了避免将失败的语句同步到备库,主库会在Binlog中插入一个LOST_EVENTS事件,导致备库在应用日志时中断复制关系。

例如,如下命令会导致主备复制中断:

如果testuser1存在但testuser2不存在,主库会记录LOST_EVENTS事件,从而中断主备复制。

REVOKE SELECT ON db1.* FROM 'testuser1'@'%','testuser2'@'%';
show grants for 'testuser1'@'%';
+--------------------------------------------+
| Grants for testuser1@%                     |
+--------------------------------------------+
| GRANT USAGE ON *.* TO 'testuser1'@'%'     |
| GRANT SELECT ON `db1`.* TO 'testuser1'@'%' |
+--------------------------------------------+
2 rows in set (0.01 sec)
show grants for 'testuser2'@'%';
+----------------------------------------+
| Grants for testuser2@%                 |
+----------------------------------------+
| GRANT USAGE ON *.* TO 'testuser2'@'%' |
+----------------------------------------+
1 row in set (0.01 sec)
revoke select on db1.* from 'testuser1'@'%','testuser2'@'%';
ERROR 1141 (42000): There is no such grant defined for user 'testuser2' on host '%'

解决方案

若主备复制已中断,系统将自动尝试修复,但这一过程可能需一定时间。您可稍后通过如下SQL查询主备复制状态是否已恢复正常:

SHOW SLAVE STATUS;
  • 若已恢复正常,在后面操作中,请勿一次性对多个用户执行GRANTREVOKE操作,改为逐一处理每个用户。示例如下:

    REVOKE SELECT ON db1.* FROM 'testuser1'@'%';
    REVOKE SELECT ON db1.* FROM 'testuser2'@'%';
  • 若未恢复正常,请联系阿里云技术支持人员处理。