Known issues for public images

CentOS 8.0: Public image naming issue

Symptoms: After you create a CentOS instance by using the centos_8_0_x64_20G_alibase_20200218.vhd public image, you connect to the instance and find that the system version is CentOS 8.1.

testuser@ecshost:~$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID:    CentOS
Description:    CentOS Linux version 8.1.1911 (Core)
Version:    8.1.1911
Codename:    Core

Cause: This public image was updated with the latest community packages, which upgraded its version to 8.1.

Affected image ID: centos_8_0_x64_20G_alibase_20200218.vhd.

Resolution: If you require CentOS 8.0, call the API operation and set the ImageId parameter to centos_8_0_x64_20G_alibase_20191225.vhd to create an ECS instance.

CentOS 7: Image ID change issues

Symptoms: The image IDs of some CentOS 7 public images have changed. This change may affect automated processes that depend on specific image IDs.

Affected images: CentOS 7.5 and CentOS 7.6

Cause: The latest versions of the CentOS 7.5 and CentOS 7.6 public images use the image ID format %OS_Type%_%Major_Version%_%Minor_Version%_%Special_Field%_alibase_%Date%.%Format%. For example, the image ID prefix for CentOS 7.5 is updated from centos_7_05_64 to centos_7_5_x64. You must adjust your automated O&M policies accordingly. For more information about image IDs, see Release notes for 2023.

CentOS 7: Hostname case changes after restart

Symptoms: On some CentOS 7 instances, uppercase letters in a hostname are converted to lowercase after the first restart.

Affected images: The following CentOS public images and any custom images created from them.

• centos_7_2_64_40G_base_20170222.vhd

• centos_7_3_64_40G_base_20170322.vhd

• centos_7_03_64_40G_alibase_20170503.vhd

• centos_7_03_64_40G_alibase_20170523.vhd

• centos_7_03_64_40G_alibase_20170625.vhd

• centos_7_03_64_40G_alibase_20170710.vhd

• centos_7_02_64_20G_alibase_20170818.vhd

• centos_7_03_64_20G_alibase_20170818.vhd

• centos_7_04_64_20G_alibase_201701015.vhd

  Affected applications: If your application is sensitive to hostname case, its services may be affected by an instance restart. Use the following table to determine if you are affected.

Resolution: To retain uppercase letters in the hostname after a restart, follow these steps.

1. Connect to the instance remotely.

   For more information, see Connection methods.

2. View the current hostname.

   [testuser@izbp193*3i161uynzzx ~]# hostname
   izbp193*3i161uynzzx

3. Run the following command to make the hostname persistent.

   hostnamectl set-hostname --static iZbp193*3i161uynzzX

4. Run the following command to view the updated hostname.

   [testuser@izbp193*3i161uynzzx ~]# hostname
   iZbp193*3i161uynzzX

   Next steps: If you use a custom image, update cloud-init to the latest version and then create a new custom image. This prevents the issue from occurring in new instances created from the image. For more information, see Install cloud-init and Create a custom image from an instance.

CentOS 6.8: NFS client instance crash

Symptoms: A CentOS 6.8 instance with the NFS client loaded may hang, requiring a restart to recover.

Cause: When you use the NFS service with a kernel version from 2.6.32-696 to 2.6.32-696.10, the kernel nfsclient proactively disconnects the TCP connection if a communication latency glitch occurs. If the NFS server is slow to respond, the connection initiated by the nfsclient may get stuck in the FIN_WAIT2 state. Normally, a connection in the FIN_WAIT2 state times out and is reclaimed after one minute, allowing the nfsclient to reinitiate the connection. However, due to a defect in the TCP implementation of these kernel versions, the connection in the FIN_WAIT2 state never times out. As a result, the TCP connection of the nfsclient can never be closed, which blocks new connections and causes user requests to hang indefinitely. Restarting the ECS instance is the only way to recover.

Affected image IDs: centos_6_08_32_40G_alibase_20170710.vhd and centos_6_08_64_20G_alibase_20170824.vhd.

Resolution: Run the yum update command to upgrade the system kernel to version 2.6.32-696.11 or later.

Ubuntu 5.15 kernel: Disk hot-unplug triggers hung task

Symptoms: When you hot-unplug a disk from an instance that runs Ubuntu kernel version 5.15.0-144-generic, a hung task may be intermittently triggered with a timeout of about 120 seconds. Common stuck processes include:

• kworker (ACPI hot-plug thread)

• udev-worker (device event handler process)

  Cause: This issue is caused by a logical flaw in the kernel's del_gendisk() function. A race condition between freezing the queue and releasing the sysfs reference leads to an ABBA deadlock.

• The kworker process holds the queue freeze lock and waits for the sysfs reference to be released.

• The udev-worker process holds the sysfs reference and waits for the queue to be unfrozen or to exit.

  Because the kernel does not set the QUEUE_FLAG_DYING flag, blk_queue_enter() cannot exit, which results in a deadlock.

Resolution:

• Method 1: Upgrade the kernel (Recommended)

  Upgrade to a kernel version that includes the fix (5.19 or later, or a distribution kernel that includes the patch).

• Method 2: Apply a patch (Temporary fix)

  In the 5.15 kernel, modify the del_gendisk() function by replacing blk_queue_start_drain(q); with blk_set_queue_dying(q);. This change sets the QUEUE_FLAG_DYING flag, which allows pending I/O requests to exit promptly and prevents the deadlock.

Ubuntu 24.04 instances affected by CVE-2024-57843 may experience kernel panics or memory corruption

• Symptoms: CVE-2024-57843 is a 16-byte cross-page overflow that occurs when the virtio-net driver fills mergeable RX buffers. Under specific conditions, such as an MTU of 4084 or higher, Ubuntu 24.04 LTS instances may experience a kernel panic, fail to transfer large files, or silently corrupt kernel data structures.

• Affected scope and resolution: For more information, see Impact and remediation of CVE-2024-57843 for ECS official images.

CentOS Stream 9 instances affected by CVE-2024-57843 may experience kernel panics or memory corruption

• Symptoms: CVE-2024-57843 is a 16-byte cross-page overflow that occurs when the virtio-net driver fills mergeable RX buffers. Under specific conditions, such as an MTU of 4084 or higher, CentOS Stream 9 instances may experience a kernel panic, fail to transfer large files, or silently corrupt kernel data structures.

• Affected scope and resolution: For more information, see Impact and remediation of CVE-2024-57843 for ECS official images.

Fedora 40 instances affected by CVE-2024-57843 may experience kernel panics or memory corruption

• Symptoms: CVE-2024-57843 is a 16-byte cross-page overflow that occurs when the virtio-net driver fills mergeable RX buffers. Under specific conditions, such as an MTU of 4084 or higher, Fedora 40 instances may experience a kernel panic, fail to transfer large files, or silently corrupt kernel data structures. Fedora 40 has reached end of life (EOL). We recommend that you migrate to Fedora 41 or later.

• Affected scope and resolution: For more information, see Impact and remediation of CVE-2024-57843 for ECS official images.

Fedora CoreOS: Hostname not applied from custom images

Symptoms: When you create an ECS instance (instance B) from a custom image that was created from another Fedora CoreOS instance (instance A), the new hostname specified for instance B is not applied. Instance B retains the hostname of instance A.

For example, you have an ECS instance (instance A) that runs the Fedora CoreOS operating system and has the hostname test001. You then use a custom image from this instance to create a new ECS instance (instance B). During the creation process, you set the hostname of instance B to test002. After you create and remotely connect to instance B, the hostname of instance B is still test001.

Cause: The Fedora CoreOS public images provided by Alibaba Cloud use the official Ignition service for instance initialization. Ignition is a utility used by Fedora CoreOS and Red Hat Enterprise Linux CoreOS to manipulate disks during the initramfs phase of system startup. When an ECS instance starts for the first time, the coreos-ignition-firstboot-complete.service in Ignition checks for the existence of the /boot/ignition.firstboot file to determine whether to initialize the instance. If this empty file exists, Ignition proceeds with initialization, which includes configuring the hostname, and then deletes the /boot/ignition.firstboot file.

Since the original Fedora CoreOS instance has been started at least once, the /boot/ignition.firstboot file is no longer present in the custom image. When you use this custom image to create a new ECS instance, Ignition does not run the initialization process on first boot, and the new hostname is not applied.

Resolution:

Before you create a custom image from a Fedora CoreOS instance, use root permissions to create the /ignition.firstboot file in the /boot directory:

1. Remount /boot in read-write mode.

   sudo mount /boot -o rw,remount

2. Create the /ignition.firstboot file.

   sudo touch /boot/ignition.firstboot

3. Remount /boot in read-only mode.

   sudo mount /boot -o ro,remount

   Ignition Configuration Specification.

OpenSUSE 15: Startup hang after kernel update

Symptoms: After you upgrade the OpenSUSE kernel to 4.12.14-lp151.28.52-default, an instance may hang on startup on certain CPU types. The known affected CPU type is Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz. The following is the call trace:

[    0.901281] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.901281] CR2: ffffc90000d68000 CR3: 000000000200a001 CR4: 00000000003606e0
[    0.901281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.901281] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    0.901281] Call Trace:
[    0.901281]  cpuidle_enter_state+0x6f/0x2e0
[    0.901281]  do_idle+0x183/0x1e0
[    0.901281]  cpu_startup_entry+0x5d/0x60
[    0.901281]  start_secondary+0x1b0/0x200
[    0.901281]  secondary_startup_64+0xa5/0xb0
[    0.901281] Code: 6c 01 00 0f ae 38 0f ae f0 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 90 31 d2 65 48 8b 34 25 40 6c 01 00 48 89 d1 48 89 f0 <0f> 01 c8 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00  

Cause: The new kernel version is incompatible with the CPU Microcode. For more information, see Bug 1162092.

Affected image: opensuse_15_1_x64_20G_alibase_20200520.vhd.

Solution: In the /boot/grub2/grub.cfg file, add the kernel parameter idle=nomwait to the line that starts with linux. An example of the modification is as follows:

menuentry 'openSUSE Leap 15.1'  --class opensuse --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-20f5f35a-fbab-4c9c-8532-bb6c66ce' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  20f5f35a-fbab-4c9c-8532-bb6c66ce
        else
          search --no-floppy --fs-uuid --set=root 20f5f35a-fbab-4c9c-8532-bb6c66ce
        fi
        echo    'Loading Linux 4.12.14-lp151.28.52-default ...'
        linux   /boot/vmlinuz-4.12.14-lp151.28.52-default root=UUID=20f5f35a-fbab-4c9c-8532-bb6c66ce  net.ifnames=0 console=tty0 console=ttyS0,115200n8 splash=silent mitigations=auto quiet idle=nomwait
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd-4.12.14-lp151.28.52-default
}

Red Hat Enterprise Linux 8: Kernel update failure

Symptoms: On a Red Hat Enterprise Linux 8 64-bit ECS instance, you run the yum update command to update the kernel and restart the instance. After the restart, you find that the kernel version remains unchanged.

Cause: In Red Hat Enterprise Linux 8 64-bit, the /boot/grub2/grubenv file that stores GRUB2 environment variables has an abnormal size. The file is not the standard 1,024 bytes, which causes the kernel update to fail.

Resolution: After updating the kernel, you must manually set the new version as the default for booting. Follow these steps:

1. Update the kernel.

   yum update kernel -y

2. Obtain the kernel boot parameters of the current operating system.

   grub2-editenv list | grep kernelopts

3. Back up the old /grubenv file.

   mv /boot/grub2/grubenv /home/grubenv.bak

4. Generate a new /grubenv file.

   grub2-editenv /boot/grub2/grubenv create

5. Set the new kernel version as the default boot version.

   In this example, the updated kernel version is /boot/vmlinuz-4.18.0-305.19.1.el8_4.x86_64.

   grubby --set-default /boot/vmlinuz-4.18.0-305.19.1.el8_4.x86_64

6. Set the kernel boot parameters.

   Set the kernelopts parameter to the value that you obtained in step 2.

   grub2-editenv - set kernelopts="root=UUID=0dd6268d-9bde-40e1-b010-0d3574b4 ro crashkernel=auto net.ifnames=0 vga=792 console=tty0 console=ttyS0,115200n8 noibrs nosmt"

7. Restart the ECS instance to boot into the new kernel.

   reboot

   Warning

   The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

   The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

   The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

   The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

   The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

   The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

   The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

SUSE Linux Enterprise Server: SMT server connection failure

Symptoms: When you use a paid SUSE Linux Enterprise Server or SUSE Linux Enterprise Server for SAP image, you may encounter connection timeouts or other issues with the Subscription Management Tool (SMT) server. When you try to download or update components, an error message similar to one of the following is returned:

• Registration server returned 'This server could not verify that you are authorized to access this service.' (500)

• Problem retrieving the repository index file for service 'SMT-http_mirrors_cloud_aliyuncs_com' location

  Affected images: SUSE Linux Enterprise Server, SUSE Linux Enterprise Server for SAP

Resolution: You must re-register and activate the SMT service.

1. Run the following commands in sequence to re-register and activate the SMT service.

   SUSEConnect -d
   SUSEConnect --cleanup
   systemctl restart guestregister

2. Run the following command to verify the activation status of the SMT service.

   SUSEConnect -s

   [{"identifier":"SLES_SAP","version":"12.5","arch":"x86_64","status":"Registered"}]

SUSE Linux Enterprise Server 12 SP5: Startup hangs after kernel update

SymptomsIntel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHzIntel(R) Xeon(R) CPU E7-8880 v4 @ 2.20GHz

[    0.901281] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.901281] CR2: ffffc90000d68000 CR3: 000000000200a001 CR4: 00000000003606e0
[    0.901281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.901281] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    0.901281] Call Trace:
[    0.901281]  cpuidle_enter_state+0x6f/0x2e0
[    0.901281]  do_idle+0x183/0x1e0
[    0.901281]  cpu_startup_entry+0x5d/0x60
[    0.901281]  start_secondary+0x1b0/0x200
[    0.901281]  secondary_startup_64+0xa5/0xb0
[    0.901281] Code: 6c 01 00 0f ae 38 0f ae f0 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 90 31 d2 65 48 8b 34 25 40 6c 01 00 48 89 d1 48 89 f0 <0f> 01 c8 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00  

Cause: The new kernel version is incompatible with the CPU Microcode.

Resolution: In the /boot/grub2/grub.cfg file, add the kernel parameter idle=nomwait to the line that starts with linux. The following provides an example of the modified file:

menuentry 'SLES 12-SP5'  --class sles --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-fd7bda55-42d3-4fe9-a2b0-45efdced' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  fd7bda55-42d3-4fe9-a2b0-45efdced
        else
          search --no-floppy --fs-uuid --set=root fd7bda55-42d3-4fe9-a2b0-45efdced
        fi
        echo    'Loading Linux 4.12.14-122.26-default ...'
        linux   /boot/vmlinuz-4.12.14-122.26-default root=UUID=fd7bda55-42d3-4fe9-a2b0-45efdced  net.ifnames=0 console=tty0 console=ttyS0,115200n8 mitigations=auto splash=silent quiet showopts idle=nomwait
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd-4.12.14-122.26-default
}

AnolisOS 8.9 RHCK: Startup failure on ecs.ebmc8i and ecs.ebmg8i

Symptoms: Due to a compatibility issue between AnolisOS 8.9 RHCK and Intel QAT, the system crashes during startup on ecs.ebmc8i and ecs.ebmg8i instances. If you need to use AnolisOS 8 RHCK on these instance types, we recommend that you use AnolisOS 8.10 RHCK.

The following is an example of the kernel crash log triggered during startup:

[   31.165923] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
[   31.174877] PGD 80a74ee067 P4D 0
[   31.178620] Oops: 0000 [#1] SMP NOPTI
[   31.182761] CPU: 134 PID: 2746 Comm: systemd-udevd Not tainted 4.18.0-513.18.1.0.1.an8.x86_64 #1
[   31.192672] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.0.ES.AL.P.087.05 04/07/2024
[   31.192673] RIP: 0010:qat_rsa_exit_tfm+0x14/0x40 [intel_qat]
[   31.209951] Code: 00 c7 83 80 00 00 00 00 00 00 00 5b 5d 41 5c 41 5d c3 cc cc cc cc 0f 1f 44 00 00 53 48 8b 87 c5 00

[   31.209952] RSP: 0018:ff5c35909f277ab8 EFLAGS: 00010282
[   31.209954] RAX: 0000000000000000 RBX: ff2aa603110ee100 RCX: 0000000000000091
[   31.209955] RDX: 0000000000000090 RSI: ff2aa603110ee140 RDI: ff2aa603110ee100
[   31.209956] RBP: ff2aa603110ee100 R08: ff5c35909f277a88 R09: ff2aa603110ee000
[   31.209957] R10: 0000000000000000 R11: 000000000006000c0 R12: ff2aa603110ee100
[   31.255995] 4xxx 0001:ed:00.0: qat_dev1 started 9 acceleration engines
[   31.261146] R13: ff2aa603110ee000 R14: ff5c35909f277b50 R15: ff5c35909f277b30
[   31.261148] FS:  00007fb8e8256280(0000) GS:ff2aa6807fd80000(0000) knlGS:0000000000000000
[   31.276913] WARNING: CPU: 175 PID: 0 at kernel/workqueue.c:1650 __queue_delayed_work+0x68/0x80
[   31.284627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   31.293749] Modules linked in:
[   31.303466] CR2: 0000000000000020 CR3: 00000080a8aaa003 CR4: 00000000000771ee0
[   31.309953]  iTCO_wdt
[   31.313408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   31.321457]  pmt_crashlog
[   31.324032] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   31.332282]  pmt_telemetry
[   31.335566] PKRU: 55555554
[   31.343894]  intel_sdsi
[   31.347296] Call Trace:
[   31.350659]  iTCO_vendor_support
[   31.353751]  ? __die_body+0x1a/0x60
[   31.356805]  pmt_class
[   31.360774]  ? no_context+0x1ba/0x3f0
[   31.360779]  ? __bad_area_nosemaphore+0x16c/0x1c0
[   31.360781]  ? do_page_fault+0x37/0x12d
[   31.365016]  joydev
[   31.368003]  ? page_fault+0x1e/0x30
[   31.372437]  ipmi_ssif
[   31.378064]  ? qat_rsa_exit_tfm+0x14/0x40 [intel_qat]
[   31.382679]  intel_uncore
[   31.385367]  ? public_key_verify_signature+0x249/0x320
[   31.389592]  cdc_ether
[   31.392570]  crypto_destroy_tfm+0x40/0xc0
[   31.398562]  pcspkr
[   31.401833]  crypto_destroy_tfm+0x40/0xc0
[   31.407910]  usbnet
[   31.410885]  public_key_verify_signature+0x254/0x320

Cause: A compatibility issue between AnolisOS 8.9 RHCK and Intel QAT causes the system to crash during startup on ecs.ebmc8i and ecs.ebmg8i instances.

Resolution: If you need to use AnolisOS 8 RHCK on these instance types, use AnolisOS 8.10 RHCK instead.

Startup call trace with recent kernels

Symptoms: A call trace may occur when you start certain instance types, such as ecs.i2.4xlarge, that run an operating system with a recent kernel version, such as RHEL 8.3 or CentOS 8.3 with kernel 4.18.0-240.1.1.el8_3.x86_64. The following provides an example of the call trace:

Dec 28 17:43:45 localhost SELinux:  Initializing.
Dec 28 17:43:45 localhost kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Dec 28 17:43:45 localhost kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Dec 28 17:43:45 localhost kernel: Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Dec 28 17:43:45 localhost kernel: Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Dec 28 17:43:45 localhost kernel: unchecked MSR access error: WRMSR to 0x3a (tried to write 0x000000000000) at rIP: 0xffffffff8f26 (native_write_msr+0x4/0x20)
Dec 28 17:43:45 localhost kernel: Call Trace:
Dec 28 17:43:45 localhost kernel:  init_ia32_feat_ctl+0x73/0x28b
Dec 28 17:43:45 localhost kernel:  init_intel+0xdf/0x400
Dec 28 17:43:45 localhost kernel:  identify_cpu+0x1f1/0x510
Dec 28 17:43:45 localhost kernel:  identify_boot_cpu+0xc/0x77
Dec 28 17:43:45 localhost kernel:  check_bugs+0x28/0xa9a
Dec 28 17:43:45 localhost kernel:  ? __slab_alloc+0x29/0x30
Dec 28 17:43:45 localhost kernel:  ? kmem_cache_alloc+0x1aa/0x1b0
Dec 28 17:43:45 localhost kernel:  start_kernel+0x4fa/0x53e
Dec 28 17:43:45 localhost kernel:  secondary_startup_64+0xb7/0xc0
Dec 28 17:43:45 localhost kernel: Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
Dec 28 17:43:45 localhost kernel: Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
Dec 28 17:43:45 localhost kernel: FEATURE SPEC_CTRL Present
Dec 28 17:43:45 localhost kernel: FEATURE IBPB_SUPPORT Present

Cause: Community updates for these kernel versions include a patch that attempts to write to Model-Specific Registers (MSRs). However, some instance types, such as ecs.i2.4xlarge, run on a virtualization version that does not support MSR writes, which causes the call trace.

Resolution: You can safely ignore this call trace because it does not affect system operation or stability.

hfg6 instance family: Kernel incompatibility causes panic

Symptoms: On instances that belong to the hfg6 instance family, upgrading to a new kernel may cause a kernel panic on some Linux distributions, such as CentOS 8, SUSE Linux Enterprise Server 15 SP2, and OpenSUSE 15.2. The following is an example call trace:

[    0.005000]  apic_timer_interrupt+0xf/0x20
[    0.005000]  </IRQ>
[    0.005000] RIP: 0010:smp_call_functioxx
[    0.005000] Code: 8b 4c 24 38 65 48 3xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx
3 e2 01 75 f5 eb ca 8b 05 b1 37 c0 01 85 xxx
[    0.005000] RSP: 000xxx      cfd80 EFLAGS: 00xxx    ORIG_RAX: ffxxx
[    0.005000] RAX: 000xxx            RBX: fffxxx        RCX: 00000000
[    0.005000] RDX: 000xxx            RSI: 000xxx        RDI: 00000000
[    0.005000] RBP: fffxxx            R08: 000xxx        R09: 00000000
[    0.005000] R10: fffxxx            R11: 000xxx        R12: 00000000
[    0.005000] R13: fffxxx            R14: 000xxx        R15: ffffffff
[    0.005000]  ? sort_range+0x20/0x20
[    0.005000]  ? poke_int3_handler+0xe0/0xe0
[    0.005000]  ? poke_int3_handler+0xe0/0xe0
[    0.005000]  ? poke_int3_handler+0xe0/0xe0
[    0.005000]  on_each_cpu+0x28/0x60
[    0.005000]  text_poke_bp_batch+0xcd/0x160
[    0.005000]  ? set_rq_offline+0x60/0x60
[    0.005000]  arch_jump_label_transform_apply+0x2e/0x50
[    0.005000]  static_key_slow_inc_cpuslocked+0x88/0x90
[    0.005000]  sched_cpu_activate+0xf1/0x100
[    0.005000]  ? refresh_zone_stat_thresholds+0x140/0x140
[    0.005000]  cpuhp_invoke_callback+0x8d/0x500
[    0.005000]  ? sort_range+0x20/0x20
[    0.005000]  cpuhp_thread_fun+0xb0/0x110
[    0.005000]  smpboot_thread_fn+0xc5/0x160
[    0.005000]  kthread+0x112/0x130
[    0.005000]  ? kthread_flush_work_fn+0x10/0x10
[    0.005000]  ret_from_fork+0x35/0x40
[    0.005000] Modules linked in:
[    0.005000] ---[ end trace 79c5ba462cfc4c1b ]---
[    0.005000] RIP: 0010:arch_scale_freq_tick+0x67/0x7e

Cause: A compatibility issue exists between the hfg6 instance family and some Linux kernel versions.

Resolution:

• The latest kernel versions for SUSE Linux Enterprise Server 15 SP2 and OpenSUSE 15.2 include a fix for this issue. If your kernel includes the following commits, it is compatible with the hfg6 instance family.

  commit 1e33d5975b49472e286bd7002ad0f689af33fab8
  Author: Giovanni Gherdovich <ggherdovich@suse.cz>
  Date:   Thu Sep 24 16:51:09 2020 +0200
  
      x86, sched: Bail out of frequency invariance if
      turbo_freq/base_freq gives 0 (bsc#1176925).
  
      suse-commit: a66109f44265ff3f3278fb34646152bc2b3224a5
      
      
  commit dafb858aa4c0e6b0ce6a7ebec5e206f4b3cfc11c
  Author: Giovanni Gherdovich <ggherdovich@suse.cz>
  Date:   Thu Sep 24 16:16:50 2020 +0200
  
      x86, sched: Bail out of frequency invariance if turbo frequency
      is unknown (bsc#1176925).
  
      suse-commit: 53cd83ab2b10e7a524cb5a287cd61f38ce06aab7
  
  commit 22d60a7b159c7851c33c45ada126be8139d68b87
  Author: Giovanni Gherdovich <ggherdovich@suse.cz>
  Date:   Thu Sep 24 16:10:30 2020 +0200
  
      x86, sched: check for counters overflow in frequency invariant
      accounting (bsc#1176925).

  yum update command to upgrade to kernel version kernel-4.18.0-240 or later on an instance from the hfg6 instance family, a kernel panic may occur. If this happens, roll back to the previous kernel version.

pip: Request timeouts

Symptoms: pip requests occasionally time out or fail.

Affected images: CentOS, Debian, Ubuntu, SUSE, OpenSUSE, and Alibaba Cloud Linux.

Cause: Alibaba Cloud provides the following pip source endpoints. The default endpoint, mirrors.aliyun.com, requires a public Internet connection. If your instance does not have a public IP address, pip requests may time out.

• (Default) Public network: mirrors.aliyun.com

• VPC internal network: mirrors.cloud.aliyuncs.com

  Resolution: Use one of the following methods to resolve the issue.

• Method 1: Assign a public IP address to your instance by associating an Elastic IP (EIP) address. For more information, see Associate an EIP with an instance.

  For subscription instances, you can also assign a new public IP address when you change the instance type.

• Method 2:

  If pip responses are delayed, run the fix_pypi.sh script on the ECS instance and retry the operation. Follow these steps:

  1. Connect to the instance remotely.

     For more information, see Connect to an instance by using a VNC client.

  2. Run the following command to download the script file.

     wget http://image-offline.oss-cn-hangzhou.aliyuncs.com/fix/fix_pypi.sh

  3. Run the script.

     For instances in a VPC, run the command bash fix_pypi.sh "mirrors.cloud.aliyuncs.com".

  4. Retry the pip operation.

     The fix_pypi.sh script contains the following:

  #!/bin/bash
  
  function config_pip() {
      pypi_source=$1
  
      if [[ ! -f ~/.pydistutils.cfg ]]; then
  cat > ~/.pydistutils.cfg << EOF
  [easy_install]
  index-url=http://$pypi_source/pypi/simple/
  EOF
      else
          sed -i "s#index-url.*#index-url=http://$pypi_source/pypi/simple/#" ~/.pydistutils.cfg
      fi
  
      if [[ ! -f ~/.pip/pip.conf ]]; then
      mkdir -p ~/.pip
  cat > ~/.pip/pip.conf << EOF
  [global]
  index-url=http://$pypi_source/pypi/simple/
  [install]
  trusted-host=$pypi_source
  EOF
      else
          sed -i "s#index-url.*#index-url=http://$pypi_source/pypi/simple/#" ~/.pip/pip.conf
          sed -i "s#trusted-host.*#trusted-host=$pypi_source#" ~/.pip/pip.conf
      fi
  }
  
  config_pip $1

Alibaba Cloud Linux 2: Upgrade GCC version

1. Install the experimental repository.

   sudo yum install -y alinux-release-experimentals

2. Install the developer toolset.

   sudo yum install -y devtoolset-9

3. Activate the new environment.

   sudo scl enable devtoolset-9 bash

4. Verify the upgrade.

   sudo gcc --version