Network connection failure in Linux due to an incorrect route

更新时间:
复制 MD 格式

Problem description

An incorrectly configured route on a Linux server can cause network connectivity issues. The symptoms are as follows:

  • Pinging the server gateway fails.

  • Pinging other services in the same CIDR block fails.

Cause

Services such as Docker have modified the route table.

Solution

Note

Alibaba Cloud reminds you of the following:

  • If you perform risky operations, such as modifying an instance or data, ensure that the instance has disaster recovery and fault tolerance capabilities to protect data security.

  • If you modify the configuration or data of an instance, such as an Elastic Compute Service (ECS) or RDS instance, create a snapshot or enable a feature such as RDS log backup beforehand.

  • If you have granted permissions or submitted security information, such as a logon username and password, on the Alibaba Cloud platform, change them promptly.

  1. Run the following command on the server to view the route table information.

    route -n

    The system displays output similar to the following.

  2. Based on the route table information, identify the affected network interface card (NIC).

  3. Run the following command to capture packets on the affected NIC. After the packet capture is complete, press Ctrl+C to stop.

    Note

    Note: The NIC may be eth0 or eth1. The actual name depends on your environment.

    tcpdump -v -i [$Iface] -w [$tcpdump]
    Note

    Note:

    [$Iface] is the affected NIC. [$tcpdump] is the file in which the captured packet data is saved.

    The system displays output similar to the following. In this example, the packets show that the local machine continuously sends MAC broadcast messages from the internal NIC to find the public gateway. Because the requests originate from the internal NIC, the public gateway never receives them. Therefore, the local machine receives no response, and the connection to the gateway fails.

  4. Use the route del command to delete the incorrect route.

  5. Use the route add command to add the correct route.

More information

  • The parameters are described as follows:

    • Destination: The destination CIDR block or host.

    • Gateway: The gateway address. An asterisk (*) indicates that the destination is on the local network and no routing is needed.

    • Genmask: The network mask. It specifies which bits of an IP address identify the subnet and which bits identify the host.

    • The meanings of the flags are as follows:

      • U: Up. The route is active.

      • H: Host. The route is to a specific host.

      • G: Gateway. The route is to a gateway.

      • R: Reinstate Route. The route was reinstated by dynamic routing.

      • D: Dynamically. The route was created by a redirection message and is dynamically modified by a routing daemon.

      • !: Reject. The route is disabled.

    • Metric: The routing distance, which is the number of hops to the destination network. This is not used by the Linux kernel.

    • Ref: The number of references to this route. This is not used by the Linux kernel.

    • Use: The number of times the route has been looked up by routing software.

    • Iface: The outgoing interface for the route.

  • A router selects a route based on the destination address, longest prefix match, administrative distance (priority), and metric. The route selection process is as follows:

    Note

    Note: Longest prefix match means that when a router looks up a route, it uses the route with the longest subnet mask for the same destination.

    1. A route is selected based on the destination address and the longest prefix match.

    2. If multiple routes match, the route with the lowest administrative distance is selected. The administrative distance value varies depending on the routing protocol.

    3. If the administrative distances are the same, the route with the lowest metric is selected.

Application scope

  • Elastic Compute Service (ECS)