Troubleshoot PAM-related logon failures on Linux instances

Updated at:

Problem description

Incorrect PAM configuration or missing PAM module files on a Linux instance can prevent logon. Check /var/log/secure for the following error messages:

  • requirement "uid >= 1000" not met by user "xxxx"

  • user xxxx (0) has 5 failed attempts, failed attempts threshold is 3

  • Refused user xxxx for service sshd

  • PAM unable to dlopen(/usr/lib64/security/pam_lastlog.so): /usr/lib64/security/pam_lastlog.so: cannot open shared object file: No such file or directory

Solution

Important

You must log on to the instance with an administrative account (such as root or a user with sudo privileges) to modify PAM configuration files.

Problem identification

  1. Log on to the ECS instance via VNC as an administrator.

    1. Go to the ECS console - Instances page. Select your region and resource group.

    2. On the instance details page, click Connect and select VNC. Enter the username and password to log on.

  2. Check the authentication logs for key error messages.

    • The account is locked due to failed attempts : If the log contains Maximum amount of failed attempts was reached, the account is locked after multiple failed logon attempts. Unlock the account.

      sudo grep -i "Maximum amount of failed attempts" /var/log/secure
    • User logon is restricted: If the log contains requirement "uid >= 1000" not met by user, a rule prevents users with a UID less than 1000 from logging on. Remove the UID restriction.

      sudo grep -i "not met by user" /var/log/secure
    • User is denylisted : If the log contains Refused user , this indicates that an allowlist or denylist is controlling user access. The user is either on the denylist or not on the allowlist. You must modify the allowlist or denylist.

      sudo grep -i "Refused user" /var/log/secure
    • A PAM module file is missing: If the log contains PAM unable to dlopen and No such file or directory, a PAM module file referenced by the system (such as pam_lastlog.so) is missing. You must restore the missing PAM module.

      sudo grep -i "unable to dlopen" /var/log/secure

Unlock the account

  1. Check the number of failed logon attempts. Replace <username> with the affected username.

    sudo pam_tally2 -u <username>

    If the failed attempt count is greater than 0, the account is locked.

  2. Unlock the account.

    sudo pam_tally2 -u <username> -r
  3. (Optional) Disable the locking policy permanently.

    Edit /etc/pam.d/system-auth and comment out the rule that contains pam_tally2.so.

    sudo vim /etc/pam.d/system-auth

    This rule locks both regular and root accounts after three consecutive incorrect password attempts, with automatic unlock after 50 seconds.

    auth required pam_tally2.so deny=3 unlock_time=50  
    This example uses the pam_tally2 module. The actual module may vary by PAM version. See the Linux-PAM System Administrators' Guide.
  4. Log on to the instance again to verify the logon works.

Remove the UID restriction

  1. Locate the configuration file.

    Find files that contain pam_succeed_if, such as /etc/pam.d/sshd, /etc/pam.d/login, or /etc/pam.d/system-auth.

    sudo grep -r "pam_succeed_if" /etc/pam.d/
  2. Modify the configuration.

    sudo vim /etc/pam.d/sshd

    Comment out the line containing auth required pam_succeed_if.so uid >= 1000 by adding # to the beginning.

    # auth required pam_succeed_if.so uid >= 1000
  3. Log on to the instance again to verify the logon works.

Modify the allowlist or denylist

  1. Locate the configuration file.

    Find files that contain pam_listfile, such as /etc/pam.d/sshd, /etc/pam.d/login, or /etc/pam.d/system-auth.

    sudo grep -r "pam_listfile" /etc/pam.d/
  2. Examine the configuration.

    sudo vim /etc/pam.d/sshd

    The sense parameter indicates an allowlist (sense=allow) or denylist (sense=deny). Note the file path in the file= parameter, such as /etc/ssh/whitelist or /etc/ssh/blacklist.

    # A whitelist is configured. Only users in the whitelist are allowed to log on.
    auth required pam_listfile.so item=user sense=allow file=/etc/ssh/whitelist onerr=fail
    
    # A blacklist is configured. Users in the blacklist are prohibited from logging on.
    auth required pam_listfile.so item=user sense=deny  file=/etc/ssh/blacklist onerr=fail
  3. Modify the list file.

    • For an allowlist: Add the username on a new line. Save and exit.

      sudo vim /etc/ssh/whitelist
    • For a denylist: Delete the line that contains the target username. Save and exit.

      sudo vim /etc/ssh/blacklist
  4. Log on to the instance again to verify the logon works.

Restore a missing PAM module

  1. Identify the package that owns the missing module.

    The diagnostic report or system log specifies the path of the missing module, for example /usr/lib64/security/pam_lastlog.so. Run the following command to identify the package that owns the file. The rpm -qf command queries the RPM database and works even if the file is already missing.

    rpm -qf /usr/lib64/security/pam_lastlog.so

    The following sample output indicates that the module belongs to the pam package.

    pam-1.3.1-39.1.al8.x86_64

    You can also run dnf provides "*/pam_lastlog.so" to identify the owning package.

  2. Reinstall the package to restore the missing module file.

    Replace pam with the package name identified in the previous step.

    sudo dnf reinstall pam -y

    On systems that use yum as the package manager, such as CentOS 7, replace dnf with yum in the preceding command.

  3. Verify that the module is restored.

    ls -l /usr/lib64/security/pam_lastlog.so

    If the module file exists, the restoration is complete.

  4. Log on to the instance again to verify the logon works.

Note

After the restoration, the instance health diagnostics may still report this issue based on the historical error logs generated before the restoration. Wait a while and run the diagnostics again.