Resolve the HSF "invalid call is removed" error-Enterprise Distributed Application Service(EDAS)-阿里云帮助中心

Symptom

After a client initiates a connection request to the server, the following error is returned:

invalid call is removed because of connection closed

This error means the TCP connection between the client and server closed while a request was still in flight. The client detected the broken connection and discarded the pending call.

Causes

Three conditions commonly trigger this error:

Cause	What happens
Transient network interruption	The client establishes a session with the server and sends a call request. A network disruption closes the session before the server finishes processing, and the client timeout has not yet expired.
Server restart	The client sends a request, but the server restarts before it can respond. The server OS sends a "connection closed" callback to the client through the socket disconnection.
Server out of memory (OOM)	The server enters a FullGC cycle that triggers an OOM error, making it unresponsive and causing the connection to drop.

Diagnose the root cause

Work through the following checks to identify which condition applies.

Check 1: Server-side GC activity

FullGC pauses can make the server unresponsive long enough for connections to time out.

Review the server GC logs for long pause times around the time of the error:

   # Locate GC log entries near the error timestamp
   grep -A 5 "Full GC" <gc-log-path> | grep -i "pause"

If the GC logs show prolonged FullGC pauses (typically several seconds), the server likely ran out of memory.
Check heap usage to confirm: If the old generation (O column) is consistently near 100%, the server is running out of heap space.
```
   jstat -gcutil <pid> 1000 5
```

Check 2: Server restart events

Check server startup logs for recent restarts around the time of the error:
```
   grep -i "started" <server-log-path> | tail -5
```
Review system logs for unexpected process termination: If the OS killed the server process (OOM Killer), you will see entries in dmesg.
```
   dmesg | grep -i "killed process"
```

Check 3: Network conditions

If neither GC issues nor server restarts explain the error, investigate the network.

Check TCP retransmission rates on the server: A high retransmission rate indicates network instability.
```
   netstat -s | grep -i retransmit
```

Test connectivity between the client and server:

   ping <server-ip>
   traceroute <server-ip>

Review network device logs (switches, load balancers, firewalls) for connection resets or drops during the error window.

Solutions

Apply the solution that matches your diagnosed root cause.

Transient network interruption

Investigate and resolve the underlying network issue (faulty hardware, misconfigured firewall rules, or bandwidth saturation).
If the service is idempotent, configure retry logic on the client.
Note
Only retry idempotent operations. Retrying non-idempotent operations (such as order creation) can cause duplicate side effects.
Monitor network retransmission rates to detect degradation early.

Server restart

For planned restarts, drain in-flight requests before stopping the server. Use graceful shutdown to let the server finish processing active requests.
For unplanned restarts, check server logs and system logs to identify the cause (crash, OOM Killer, hardware failure).
For idempotent services, configure client-side retries to handle brief unavailability during restarts.

Server OOM

Increase the JVM heap size (-Xmx) to give the server more memory.
Analyze heap dumps to find memory leaks: Use a tool such as Eclipse MAT or VisualVM to analyze the dump.
```
  jmap -dump:format=b,file=heap_dump.hprof <pid>
```
Tune GC parameters to reduce FullGC frequency. For example, switch to G1GC if you are still using the default collector:
```
  -XX:+UseG1GC -XX:MaxGCPauseMillis=200
```
Add JVM monitoring to detect rising heap usage before it causes an OOM error.

Prevent recurrence

Measure	Purpose
Enable server health checks	Detect unresponsive servers before they affect traffic.
Configure client-side retries (for idempotent services)	Automatically recover from transient connection failures.
Set up JVM monitoring and alerts	Catch memory pressure and long GC pauses before they cause OOM errors.
Use graceful shutdown for deployments	Drain active connections before stopping a server.
Monitor network metrics	Track retransmission rates and connection errors at the infrastructure level.