Troubleshoot remote connection failures to a Windows instance

Use the self-service troubleshooting tool

The Alibaba Cloud self-service troubleshooting tool helps you quickly check security group configurations, the internal firewall of the instance, and the listener status of common application ports. The tool provides a detailed diagnostic report.

If the self-service troubleshooting tool cannot identify the issue, follow the steps below to troubleshoot the issue manually.

Manual troubleshooting

Check the status of the ECS instance, and then use Cloud Assistant to send commands to the Windows instance or log on to the instance using VNC.

Step 1: Test the remote connection by using Workbench

You can use Workbench provided by Alibaba Cloud to connect remotely. If the remote connection fails, Workbench returns a specific error message and a solution.

1. Go to ECS console - Instances.

2. In the upper-left corner of the page, select a region and resource group.

3. On the Instances page, find the instance that you want to connect to and click Connect in the Actions column.

4. In the Connect dialog box, click Sign in now next to Workbench.

5. Workbench automatically fills in the basic information required to log on to the target instance. Confirm that the information is correct and enter the username and authentication credentials. Then, take one of the following actions based on the result:

   • If you still cannot log on, Workbench returns an error message and a solution. Follow the on-screen instructions to resolve the issue and then try to connect again. You can connect to the instance using VNC to resolve common issues that occur when you use Workbench.

   • If you can log on to the instance with Workbench but not from your local computer, the instance's remote connection port and service are working correctly. The issue is likely with your local client, which you must troubleshoot.

Step 2: Check for blackhole filtering notifications

Check for notifications indicating that blackhole filtering has been triggered for the instance. This feature blocks all internet access to the instance. For more information, see Alibaba Cloud blackhole policy.

Step 3: Check ports and security groups

Check whether any security group rules are blocking the connection. Follow these steps:

1. Go to ECS console - Instances.

2. In the upper-left corner of the page, select a region and resource group.

3. On the Instances page, click the ID of the target instance.

4. Click the Security Groups tab. In the Security Group List section, find the security group that you want to manage and click Manage Rules in the Actions column.

5. Select the direction for the security group rule.

6. On the Security Group Details tab, use one of the following methods to add a security group rule.

   • Method 1: Quickly add a security group rule

     • Select RDP Connection to a Windows Instance

   • Method 2: Manually add a security group rule

     • Action: Allow

     • Priority: 1 (A smaller value indicates a higher priority.)

     • Protocol Type: Custom TCP

     • Source > /Destination IPv4 CIDR Block: 0.0.0.0/0 (represents all IP addresses)

     • Destination > /Source Port Range: Set this to the RDP port (default 3389).

7. Connect to the remote desktop using the IP:port format.

   In the Computer text box of the Remote Desktop Connection dialog box, enter the address in the IP:port format, such as 192.168.0.1:4389, and click Connect.

8. Run the following command to test if the port is working correctly.

   telnet <IP> <Port>

   Note

   • <IP> specifies the IP address of the Windows instance.

   • <Port> specifies the RDP port number of the Windows instance.

   For example, when you run the telnet 192.168.0.1 4389 command, the output is normally similar to the following.

   Trying 192.168.0.1 ...
   Connected to 192.168.0.1  4389.
   Escape character is '^]'

   If the port test fails, see Check port availability when the ping command is successful but the port is unreachable for troubleshooting.

Step 4: Check the firewall configuration

1. Connect to a Windows instance using VNC.

2. In the menu bar, choose Start > Control Panel.

3. Set View by to Small icons and click Windows Firewall.

4. In the Windows Firewall window, click Advanced settings.

5. Enable the firewall configuration.

   1. In the Windows Firewall with Advanced Security, window, click Windows Firewall Properties.

   2. Select On (recommended) and click Apply(A).

      We recommend that you enable the firewall on the Domain Profile, Private Profile, and Public Profile tabs.

6. In the Windows Firewall with Advanced Security window, click Inbound Rules. In the pane on the right, scroll down to the bottom, right-click Remote Desktop - User Mode(TCP-In), and select Enable Rule.

Step 5: Check Remote Desktop Services

Check whether Remote Desktop Services is enabled on the Windows server.

1. Connect to a Windows instance using VNC.

2. Right-click the Start menu and click System.

3. In the System window, click Remote settings.

4. In the Remote Desktop section, select Allow remote connections to this computer (L), and click OK.

5. Start the Remote Desktop Services service.

   In the Start menu, choose Administrative Tools > Component Services > Services (Local). In the pane on the right, find the Remote Desktop Services service and check its status. If the service is not running, start it.

6. Load the drivers and services that Remote Desktop Services depends on.

   Key services that Remote Desktop Services depends on are sometimes disabled by mistake, causing connection failures. To resolve this, perform the following check.

   1. Right-click the Start menu, click Run, enter msconfig, and click OK.

   2. In the System Configuration dialog box, click the General tab, select Normal startup(N), and click OK.

   3. Restart the ECS instance.

Step 6: Check remote terminal service

The remote desktop of a Windows instance may be unreachable because of an incorrect remote terminal service configuration.

Step 7: Check the network

If you cannot remotely connect to a Windows instance, first check whether the network is working correctly.

1. Test the connection from other network environments, such as different network segments or from different carriers, to determine if the issue is with your local network or the server.

   • If the issue is with your local network or carrier, contact your local IT staff or the carrier to resolve it.

   • If the network adapter or its driver is not working correctly, ensure the network adapter is enabled and update its driver. Follow these steps:

     Fix a faulty network adapter or driver

     Note

     This operation uses Windows Server 2016 as an example. The user interface (UI) may vary by OS version. Adjust the steps accordingly.

     1. Connect to a Windows instance using VNC.

     2. In the notification area of the taskbar, right-click the icon and select Open Network and Sharing Center.

     3. Click Change adapter settings to check whether the network adapter is enabled.

        • If the network adapter is disabled, right-click the adapter name, select Enable, and then check whether the remote connection to the Windows instance is restored.

        • If the network adapter is enabled but still unavailable, proceed to the next step.

     4. Open the Run window, enter regedit, and click OK.

     5. In Registry Editor, navigate to HKEY_LOCAL_MACHINE > SYSTEM > CurrentControlSet > Control > Class > {4d36e972-e325-11ce-bfc1-08002be10318}. Check whether the following information exists in the pane on the right. If not, right-click a blank area and select New to add it.

        Important

        After you modify the registry, you must restart the system for the changes to take effect.

        • Name: Installer32

        • Type: REG_SZ

        • Data: NetCfgx.dll,NetClassInstaller

     6. Open the Run window, enter devmgmt.msc, and click OK.

     7. On the Device Manager page, expand Network adapters, right-click the name of the network adapter, and select Update Driver Software....

     8. In the dialog box that appears, click Search automatically for updated driver software. After the update is complete, click Close.

     9. Check whether the remote connection to the Windows instance is restored.

2. Use the ping command on your local client to test the network connectivity to the instance.

   • If a network issue occurs, see Use a packet capture tool to capture network packets for troubleshooting.

   • If ping packets are lost or the ping command fails, see Use MTR to analyze network paths for troubleshooting.

   • If intermittent packet loss occurs and the network of your ECS instance is unstable, see Link interruption to resolve the issue.

3. If you receive a General failure error when you run the ping command from your instance to test the connectivity to the client, see The "General failure" error is reported when I ping a public IP address from a Windows instance to resolve the issue.

Step 8: Check CPU, bandwidth, and memory

A remote connection to a Windows instance may fail due to high CPU load, insufficient bandwidth, or insufficient memory.

1. Select the appropriate operation based on the CPU load.

   • If the CPU load is not high, continue to the next check.

   • If the CPU load is high, resolve the issue by using the methods in this step.

     • If you suspect high CPU load, log on to the instance from the terminal on the Instance Details page and check whether Windows Update is running in the background. A high CPU load is expected if Windows Update is running. Wait for the update to complete.

     • If applications on the instance cause high disk I/O, frequent network requests, or compute-intensive workloads, a high CPU load is expected. In this case, you can upgrade the instance type to resolve resource bottlenecks.

       Note

       For more information about how to resolve high CPU loads, see Troubleshoot high CPU utilization on Windows instances.

     • If the CPU usage remains abnormally high after you rule out the preceding causes, the instance may be infected with malware. For example, cryptomining malware maliciously consumes CPU resources, which may cause the instance to lag, respond slowly, or even become unreachable. For more information about how to troubleshoot and handle this issue, see Scan for and handle cryptomining programs. If the instance is infected with ransomware, system files are encrypted and locked, which may also cause logon failures. For more information, see Protect your servers from ransomware.

2. Check for insufficient public bandwidth.

   A remote connection may fail due to insufficient public bandwidth. Follow these steps to troubleshoot the issue.

   1. Go to ECS console - Instances.

   2. In the upper-left corner of the page, select a region and resource group.

   3. On the Instances page, click the instance ID. On the Instance Details tab, view the Public Bandwidth in the Configuration Information section.

      If the public bandwidth is 0 Mbit/s, the instance was created without public bandwidth. You can resolve this issue by upgrading the public bandwidth.

3. Check for insufficient memory.

   After you remotely connect to a Windows instance, the desktop may not display correctly and the connection may close without an error message. Insufficient server memory can cause this issue. Follow these steps to check the memory usage.

   1. Connect to a Windows instance using VNC.

   2. Go to Start > Administrative Tools > Event Viewer and check for warning logs that indicate insufficient memory resources. For more information, see Troubleshoot insufficient virtual memory on Windows Server 2003/2008 ECS instances.

Step 9: Check system security policies

Check whether any security policies on the Windows server are blocking remote desktop connections. Follow these steps.

1. Connect to a Windows instance using VNC.

2. Choose Start > Control Panel > Administrative Tools and double-click Local Security Policy.

3. In the Local Security Policy window, click IP Security Policies on Local Computer. The next step depends on whether a relevant security policy already exists.

   1. If a relevant security policy exists, delete or edit it.

      • To delete the security policy, right-click the policy and select Delete(D). In the dialog box that appears, click Yes.

      • Double-click the IP security policy to open it, reconfigure it to allow remote desktop connections, and then try to connect again by using Remote Desktop.

   2. If no relevant security policies exist, proceed to Step 10 to continue troubleshooting.

Step 10: Check the antivirus software

Remote connection failures may be caused by third-party antivirus software settings. Use the following methods to resolve this issue. This section provides two case studies of how SafeDog configuration can cause remote access to fail.

• If antivirus software is running in the background, connect to the instance using VNC to upgrade the antivirus software to the latest version or uninstall it. For more information about how to log on to an instance using VNC, see Connect to an instance by using VNC.

• Use a commercial version of antivirus software, or use the free Microsoft Safety Scanner to scan for and remove viruses in safe mode. For more information about Security Scanner, see Microsoft Safety Scanner.

Step 11: Check Windows registry configurations

Incorrect configurations in the Windows registry may block RDP connections. Follow these steps to fix the issue.

1. Connect to a Windows instance using VNC.

2. In the Run dialog box, enter regedit and click OK to open the Registry Editor.

3. In Registry Editor, modify the following parameter values.

   • In HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp, modify the value of the fEnableWinStation parameter to 1.

   • Change the value of the fDenyTSConnections parameter in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server to 0.

Step 12: Check for expired RDP self-signed certificates

An expired RDP self-signed certificate may cause remote logon failures. Follow these steps to fix the issue.

1. Connect to a Windows instance using VNC.

2. Run Windows PowerShell as an administrator.

3. In the Windows PowerShell window, run the following command to check whether the current certificate has expired.

   Get-Item 'Cert:\LocalMachine\Remote Desktop\*' | Select-Object NotAfter

4. If the certificate has expired, run the following command to delete the self-signed certificate and restart the TermService service.

   Remove-Item -Path 'Cert:\LocalMachine\Remote Desktop\*' -Force -ErrorAction SilentlyContinue
   Restart-Service TermService -Force

   When the TermService service restarts, the system automatically generates a new self-signed certificate.

5. Run the following command to confirm that the new self-signed certificate's timestamp has been updated.

   Get-Item 'Cert:\LocalMachine\Remote Desktop\*' | Select-Object NotAfter

   Note

   The default validity period of an RDP self-signed certificate is six months.

Specific error messages

• A protocol error occurs when you remotely connect to a Windows ECS instance

• Authorization-related issues:

  • Enabling password-protected sharing causes the 'Your credentials did not work' error during remote access to a Windows system

  • The "The connection was denied because the user account is not authorized for remote login" error occurs when you connect to an ECS instance using Remote Desktop

  • Solution for an unspecified Remote Desktop license server in a Windows instance

  • What do I do if an error message that indicates the lack of Terminal Server User Access permissions is reported when I connect to a Windows instance by using Remote Desktop?

• Connection count-related issues:

  • "The terminal server has exceeded the maximum number of allowed connections" error is reported when connecting to a Windows instance

Related topics

• Troubleshoot connection failures to a Linux instance

• VNC connection issues

• SSH connection issues

• RDP connection issues