DocsICP FilingConsole
官方文档
首页

RPC FAQ

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic summarizes common issues and troubleshooting methods for using Remote Procedure Call (RPC).

Errors occur when using an RPC client to call a service

When you call a service, the error "RPC-02306: No service address is available for service [{0}]. Check whether the service has been pushed." is reported.

To troubleshoot the issue, follow these steps:

  1. Check whether the service endpoint was pushed.

    Log on to the client and view the /home/admin/logs/rpc/sofa-registry.log file. You can filter the log by the service interface name to find the last push record. If the server-side endpoint was not pushed to the client, first check whether the service was registered successfully. For example, the following log record shows that zero callable destination addresses are available. This indicates that the server-side endpoint for com.alipay.share.rpc.facade.SampleService was not pushed to the client.

    RPC-REGISTRY - RPC-00204:Receiving RPC service address: ServiceName[com.alipay.share.rpc.facade.SampleService:1.0@DEFAULT]
Number of callable destination addresses[0]

  2. Check whether the client received the RPC Config push at startup.

    View the /home/admin/logs/rpc/rpc-registry.log file to determine the last startup time of the RPC client. Filter the log by the client's last startup time and the service interface name to check whether a Receive Rpc Config info record exists for the interface. If not, subsequent service calls will fail. In this case, you can consider restarting the client.

  3. Check whether the service is registered successfully in the SOFA Application Center.

    Check the service registration status, or log on to the server and view the /home/admin/logs/confreg/config.client.log file. If there are errors related to service publishing, use the log information for further troubleshooting.

  4. Check whether the service was called before the endpoint was pushed.

    If the client's sofa-registry.log file shows that the service endpoint has been pushed, but the RPC-02306 error occurred before the push time, the client application was likely not fully started when the service was called. This issue is often caused by a business system that calls the service through a scheduled task or starts calling the service immediately after a bean is initialized. To resolve this issue, you can configure the address-wait-time parameter.

  5. Check whether the configurations of the RPC server and client applications match.

    Open the application.properties configuration file for both the server and client applications. Check whether the com.alipay.instanceid and com.antcloud.antvip.endpoint parameters are configured with the same values. If the configurations are different, the RPC client cannot detect the RPC server.

  6. Check the connection to the service registry.

    Run the following command to check the connection status between the client, the server, and the service registry:

    netstat -a |grep 9600

    Port 9600 is the listening port for the service registry. The client and server establish persistent connections to port 9600 to publish and subscribe to services. If the connection from the client or server to port 9600 is broken, restart the application to recover the connection and investigate the cause of the disconnection.

  7. Check the RPC server-side endpoint binding.

    Log on to the RPC server and run the following command:

    ps -ef|grep java

    Check whether the process startup parameters rpc_bind_network_interface or rpc_enabled_ip_range are bound to the correct IP address.

The error log contains "Rpc invocation timeout[responseCommand TIMEOUT]"

The following time series chart shows an RPC call:DRM推送.png

Note

For more information about the time consumed at each stage on the client and server, see Tracing Analysis.

If your RPC service call times out, you will see the following exception in the client's logs/tracelog/middleware_error.log file:

2018-07-0613:21:20.463,sofa2-rpc-client,707c27b9153085447746110464663,0,main,timeout_error,rpc,invokeType=sync&uid=&protocol=bolt&targetApp=sofa2-rpc-server&targetIdc=&targetCity=&paramTypes=&methodName=message&serviceName=com.alipay.share.rpc.facade.SampleService:1.0&targetUrl=10.160.34.141:12200&targetZone=&,,com.alipay.sofa.rpc.core.exception.SofaTimeOutException: com.alipay.remoting.rpc.exception.InvokeTimeoutException:Rpc invocation timeout[responseCommand TIMEOUT]! the address is10.160.34.141:12200

Troubleshoot the issue by following these steps:

  1. Check whether the timeout is caused by an issue with the service itself, such as a long processing time in the business code.

    By default, the RPC timeout period is 3 seconds. To determine the actual processing time for a request, log on to the server and view the logs/tracelog/rpc-server-digest.log file. Use the traceID from the client timeout log, such as 707c27b9153085447746110464663, to find the log entry for the corresponding request on the server. The log format is as follows:

    2018-07-0613:21:22.441,sofa2-rpc-server,707c27b9153085447746110464663,0,com.alipay.share.rpc.facade.SampleService:1.0,message,bolt,,10.160.33.96,sofa2-rpc-client,,,4001ms,0ms,SofaBizProcessor-12200-0-T46,02,,,1ms,,

    In the preceding log, the server-side business code processing time is 4001 ms.

    The default RPC call timeout is 3 seconds. If the time consumed in the log is greater than or very close to 3 seconds, first investigate the server. Possible causes include the following:

    • The server-side business code executes slowly.

    • The server itself calls an external service, or the server calls another RPC service (client > RPC Server A > RPC Server B). In this case, you must investigate both Server A and Server B to identify the problem.

    • The server performs database operations, such as slow database connections or slow SQL queries.

    If the timeout is caused by the server itself, adjust the code.

  2. Check whether the timeout is caused by a depleted RPC thread pool on the server.

    Log on to the server and view the rpc/tr-threadpool log. If the RPC thread pool queue is blocked, check whether there was a peak in business requests during the timeout period. You can also use jstack to check for waiting threads or deadlocks that may have exhausted the RPC thread pool. For more information, see Application-level configuration extensions.

  3. Check whether threads were stopped due to Garbage Collection (GC) issues.

    Some GC types can trigger a "stop-the-world" event, which suspends all threads. To check whether the timeout is caused by GC, you can enable GC logs using one of the following methods.

    • Method 1:

      Add the following startup parameters to the config/java_opts file and then repackage and publish the application.

      -verbose:gc -XX:+PrintGCDetails-XX:+PrintGCDateStamps-Xloggc:/home/admin/logs/gc.log

    • Method 2:

      1. Run the kill -15 command to end the server-side process.

      2. Manually start the RPC service.

        Run su admin to switch to the admin user. Then, start the RPC service using the nohup command as follows:

        $ nohup java -verbose:gc -XX:+PrintGCDetails-XX:+PrintGCDateStamps-Xloggc:/home/admin/logs/gc.log -Drpc_bind_network_interface=eth0 -Dspring.profiles.active=&{environment_id}-jar /home/admin/app-run/sofa2-rpcserver-service-1.0-SNAPSHOT-executable.jar &
        Note

        You can find the workspace ID by logging on to the SOFAStack console and choosing Resource Management > Workspace in the navigation pane on the left.

      3. After the next RPC timeout occurs, view the gc.log file to verify whether a long-running GC, especially a Full GC, occurred during the timeout period.

  4. Check whether the timeout is caused by network latency or jitter.

    Troubleshoot the issue by following these steps:

    1. Run tsar -i 1 on the client and server to check for network retransmissions at the time of the issue.

    2. Deploy tcpdump on both the client and server to capture packets in a loop. Analyze the network packets after the issue occurs.

    3. Run ping on the client and server to check for network latency.

  5. Check whether other external factors are affecting server performance, such as task scheduling, batch processing, or resource contention with other virtual machines or containers on the host.

How to print client-side RPC call statistics

The following example statement prints the total number of requests to the sofa2-rpc-server application that took longer than 3 seconds, the server IP address, the service application, and the client IP address:

$ grep sofa2-rpc-server rpc-client-digest.log | awk -F,'{if(int($18)>3000)print $9,$10,$27}'|sort | uniq -c | sort -n

When you use this command, replace sofa2-rpc-server with the actual server-side application name. Adjust the $18 value based on the column position of the processing time in your log. You can also adjust the printed information as needed.

Why does a service fail to publish after the SOFABoot application has started?

Troubleshoot the issue based on the following situations:

  • Abnormal application startup

    Typically, you can view the health-check log. If there are error logs, use the related information for troubleshooting. Common fault information includes:

    • Redis is not configured correctly.

    • Multiple instances of a service are running locally.

    • The Bolt service failed to start, and a port conflict was detected.

  • Service registry issues

    If the application has started but the service failed to publish, follow these steps to troubleshoot:

    1. Check whether any services are registered in the service registry. If not, rule out a service registry fault.

    2. Check for ACVIP issues. If you rule them out and the service still has problems, follow these steps to troubleshoot:

      1. Check whether the application container has a file such as /Users/xxx/conf/acvip-java-client-cache/domains/0000X-DSR_HTTP.json. If it does, open the file and view its contents. It usually contains the locally cached DSR service registry address. Check for any abnormalities, such as a failed health check or an incorrectly obtained IP address.

      2. Use a command to determine whether the current service registry is normal. Example: curl -i -XPOST {antvip}:9003/antcloud/antvip/instances/get -d '{"vipDomainName2ChecksumMap":{"000001-DSR_CLOUD":"N"}}'. If it is not normal, check whether the service registry is configured correctly.

  • Service provider's run mode

    When publishing to the cloud, the run.mode=DEV parameter was not changed. In DEV mode, the service is registered only locally and not in the service registry.

How to migrate an internal Dubbo project to SOFABoot

Problem description:

  • How can I migrate an internal Dubbo project to SOFABoot?

  • If a third party needs to keep Dubbo, how should the system be designed?

Solutions:

During a system transformation, you cannot guarantee that all associated systems will be upgraded at once. You may face scenarios where you need to maintain compatibility with legacy systems. For example, after a service is converted to a SOFA Bolt service, you might find that some callers still depend on Dubbo. A simple compatibility solution is to have the service expose both BOLT and Dubbo services.

To expose a Dubbo service in SOFABoot, follow these steps:

  1. Add the Dubbo starter dependency.

    Example:

    <dependency>
    <groupId>com.alibaba.boot</groupId>
    <artifactId>dubbo-spring-boot-starter</artifactId>
    <version>0.1.1</version>
</dependency>
<!-- Dubbo -->
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>dubbo</artifactId>
    <version>2.6.4</version>
</dependency>
<!-- Spring Context Extras -->
<dependency>
    <groupId>com.alibaba.spring</groupId>
    <artifactId>spring-context-support</artifactId>
    <version>1.0.2</version>
</dependency>

  2. Configure application.properties.

    Example:

    ################ common configuration ##############
spring.application.name=bank-dubbo-provider
logging.level.com.dubbo.example=INFO
logging.path=./logs
################ dubbo configuration ##############
demo.service.version =1.0.0
dubbo.application.id = bank-dubbo-provider
dubbo.application.name = bank-dubbo-provider
################ sofa configuration ##############
run.mode=DEV
com.alipay.sofa.rpc.bolt-port=12201
# shared middleware
com.alipay.env=shared
com.alipay.instanceid=IPYJUBMB231N
com.antcloud.antvip.endpoint=100.103.1.174
com.antcloud.mw.access=uPxHLxsMmstcQCNWEh
com.antcloud.mw.secret=TyMlUB9uGRMzcc2pG0dMv6xzUXCM****

  3. Add the Dubbo service publication.

    Example:

    <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:sofa="http://schema.alipay.com/sofa/schema/slite"
    xmlns:dubbo="http://dubbo.apache.org/schema/dubbo"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
    http://www.springframework.org/schema/beans/spring-beans-3.0.xs
    http://schema.alipay.com/sofa/schema/slite http://schema.alipay.com/sofa/slite.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://dubbo.apache.org/schema/dubbo http://dubbo.apache.org/schema/dubbo/dubbo.xsd">
    <!-- dubbo zookeeper configuration -->
    <dubbo:registry address="zookeeper://127.0.0.1:2181"/>
    <dubbo:protocol name="dubbo" port="20880"/>
    <!-- bean define -->
    <bean id="dubboService" class="com.dubbo.example.service.DubboServiceImpl"/>
    <!-- sofa service -->
    <sofa:service interface="com.dubbo.example.facade.DubboService" ref="dubboService" unique-id="sofaDubboService">
        <sofa:binding.bolt/>
    </sofa:service>
    <!-- dubbo service -->
    <dubbo:service interface="com.dubbo.example.facade.DubboService" ref="dubboService" version="1.0.0"/>
</beans>
    Note

    The service needs to import the Dubbo schema so that the Dubbo definitions are displayed. The Dubbo service registry is Zookeeper (ZK), which also needs to be configured.

  4. Update the main function to enable Dubbo.

    Example:

    @ImportResource({"classpath*:META-INF/bank-dubbo-provider/*.xml"})
@org.springframework.boot.autoconfigure.SpringBootApplication
@EnableDubbo
public class SOFABootSpringApplication{
     private static final Logger logger =LoggerFactory.getLogger(SOFABootSpringApplication.class);
     public static void main(String[] args){
          SpringApplication springApplication =new SpringApplication(SOFABootSpringApplication.class);
          ApplicationContext applicationContext = springApplication.run(args);
     }
}

An "02306, Service cannot be published to the registry" error occurs during an RPC call

Phenomenon:

The error log for the call is as follows:日志报错.pngWhen you check the RPC service registry log, the registry is not found. Example:未发现注册中心.pngThe service publication is invalid. The health check log shows the following problem:error.pngCause:

Bolt port 12200 is occupied.

Solutions:

Change the port or shut down the service that is occupying the local port.

How to resolve application startup failures and service registration issues

Problem description:

An application depends on some SOFA components, but you only want to test RPC in the local environment. How do you handle the following issues?

  • Application startup failure

  • Service registration failure

Solutions:

The health check mechanism checks the status of all components when the project starts. If you reference Distributed Dynamic Configuration Service (DDCS) or other components, the application may start normally, but the RPC service cannot be registered locally. If your business does not use these components, you can skip these checks during the health check. The root cause is that these components look for component addresses in antvip, but the application is not in the cloud, so the check fails. To resolve this issue, perform the following operation:

Add the following configuration to application.properties to skip the health check for all components.

com.alipay.sofa.healthcheck.skip.component=true
Important

This solution is recommended for testing purposes only. Always enable health checks in a production environment.

How to handle errors when uploading a file larger than 10 MB through a SOFARest interface

Phenomenon:

When you upload a file larger than 10 MB through a SOFARest interface, an error occurs. The error message is as follows:

ERROR org.jboss.resteasy.core.ExceptionHandler- failed to execute
javax.ws.rs.NotFoundException:Couldnot find resource for full path: http://unknown/bad-request
    at org.jboss.resteasy.core.registry.ClassNode.match(ClassNode.java:73)
    at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:48)

Troubleshooting steps:

  1. Enable the error in the log and change the log mode for HTTP-related classes in Netty to debug.

    Example:

    logging.level.io.netty.handler.codec.http.HttpObjectAggregator=DEBUG

  2. Obtain the specific cause of the error, for example, Failed to send a 413 Request Entity Too Large.

  3. Set the corresponding parameter in the application.properties file.

    com.alipay.sofa.rpc.RestMaxRequestSize=104857600

    If the application's default memory capacity is small, such as 1 GB or 2 GB, the payload requested by Netty REST is placed in DirectMemory. This DirectMemory has a maximum value, which defaults to the memory size requested by the system JVM during initialization. If an out-of-memory (OOM) or DirectMemory overflow error occurs when you upload a large file, perform the following steps:

    1. Determine whether the current system memory size can handle the large file.

    2. Confirm the maximum value of the runtime memory or DirectMemory.

    3. If optimization is possible, use multipart upload.

How to troubleshoot frequent timeouts when a client calls a remote service, but the server responds quickly and normally

You can investigate the following aspects:

  • The thread pool is blocked

    This issue usually occurs only in multi-level link calls. For example, if A calls B, which then calls C, and the server-side thread of B is blocked, you need to check the tr-threadpool.log file of B.

  • A critical fault occurred during GC processing

    Although GC faults are rare in the framework, they can occur in specific scenarios. Generally, check the logs/stdout.log file and examine metrics such as CMS-remark, YG, and ParNew. These metrics indicate a stop-the-world (STW) event, which can cause the JVM to pause.

  • Hardware disk I/O fault

    Generally, you can use tsar -I 1 to view the number of I/O requests in the next minute. In some scenarios, high disk I/O can affect the performance of the entire system.

  • Network fault

    Network issues are often difficult to locate. A common network device fault is that when a firewall removes an inactive (90s) link or a Server Load Balancer (SLB) failover removes a link, it does not send an RST packet to the client socket. This can leave a stale socket on the client.

    A machine requests a service from a specific IP address. The service has low traffic, so the request frequency is very low, sometimes only once every few tens of minutes or even hours. When a timeout occurs, it may time out once and then disconnect, or it may time out n times consecutively before disconnecting. A single timeout is likely caused by a firewall disconnecting the link. Multiple timeouts are likely caused by an SLB disconnecting the link.

    Why does the connection break only after n timeouts? Because the socket is already stale, when data is written, it is only written to the TCP buffer and not actually sent. The operating system (OS) does not interrupt the upper-layer user. An org.apache.mina.common.WriteTimeoutException occurs only when the TCP buffer is full. The TCP buffer on a server is typically 64 KB. To troubleshoot this type of issue, you can configure heartbeats or use the fault removal feature.

What should I be aware of when a SOFA client calls a long-running service?

Problem description:

  • What should I be aware of when a SOFA client calls a long-running service?

  • If I set a very long timeout period and it does not take effect, how can I troubleshoot the error?

Troubleshooting approach:

If the service initiator finds that the target is a long-running service, it needs to configure a reasonable timeout period. Otherwise, it must determine whether the interface needs to be executed in a oneway manner. If it must wait for a result and finds that the timeout period does not take effect regardless of the configuration, check whether a firewall or load balancer has configured connection timeout control upstream.

How can I change the service registry path for local RPC calls?

Currently, the only way to do this is to set the environment variable: System.setProperty("user.home","local_directory"). The Enterprise Edition wraps the service registry path configuration. SOFA reads this user.home by default at startup. When configuring, you need to add two forward slashes (//) because the framework skips the first two characters and starts reading from the third. For example: user.home=//c://hulu.

Important

This issue occurs only during local development. You do not need to worry about the service registry in a cloud environment. The Enterprise Edition uses antvip to obtain the address of a healthy service registry and then constructs dsr://ip:port. The Enterprise Edition wraps this construction process in the framework.

Is there an example of a Resteasy key-value method request?

Examples are as follows:

  • SpringMVC Controller request method:

    • URL: http://localhost:8080/test?str=aaa

    • Code configuration example:

      @GetMapping("/test")
public String testParam(@RequestParam("str") String str){
    return str;
}

  • Resteasy GET request:

    • Type 1:

      • URL: http://localhost:8341/webapi/users/test/xiaoming

      • Code configuration example:

        @Path("/webapi/users")
public interface SampleRestFacade{
    @GET
    @Path("/test/{userName}")
    public RestSampleFacadeResp<DemoUserModel> user(@PathParam("userName")String userName)throws CommonException;
}

    • Type 2 (key-value)

      • URL: http://localhost:8341/webapi/users/test?userName=xiaoming

      • Code configuration example:

        @Path("/webapi/users")
public interface SampleRestFacade{
      @GET
      @Path("/test")
      public RestSampleFacadeResp<DemoUserModel> userInfo(@QueryParam("userName") String userName) throws CommonException;
}

  • @FormParam: Maps a field from a form to a method call. This type of submission is usually a POST request.

RPC Tracer log format

For more information about the log format, see SOFARPC logs.

Is there an example of a generalized call?

SOFARPC provides generic interface methods and types at the framework level:

  • Service interface name: Set through the service definition.

  • Method name and parameter list: Passed in through $invoke or $genericInvoke.

  • Custom type: Use GenericObject.

Note the following points:

  • $invoke method: Use this method only if the parameter types can be loaded by the current application's class loader, such as if only basic types are used.

  • $genericInvoke with GenericObject: Use this method when the parameter types cannot be loaded by the current application's class loader.

  • argTypes must pass the parameter types declared in the interface. You cannot use subclass types.

  • When you call the $genericInvoke interface, classes from packages other than the following are serialized into GenericObject: "com.sun","java","javax","org.ietf","org.ogm","org.w3c","org.xml","sunw.io","sunw.util"

  • GenericContext is temporarily used only for unitization scenarios.

  • The value of GenericObject and fields can also be a GenericObject.

The following is an example of a SOFARPC generalized call:

  • Service provider:

    • Service, interface, and type definitions:

      // Service definition
<sofa:reference interface="com.alipay.sofa.rpc.api.GenericService" id="xxxGenericService">
    <sofa:binding.tr>
        <sofa:global-attrs generic-interface="full_name_of_target_service_interface"/>
    </sofa:binding.tr>
</sofa:reference>
    
    // Interface method definition
    public interface GenericService{
        Object $invoke(String methodName,String[] argTypes,Object[] args) throws GenericException;
        Object $genericInvoke(String methodName,String[] argTypes,Object[] args) throws GenericException;
        Object $genericInvoke(String methodName,String[] argTypes,Object[] args,GenericContext context) throws GenericException;
        <T> T $genericInvoke(String methodName,String[] argTypes,Object[] args,Class<T> clazz) throws GenericException;
        <T> T $genericInvoke(String methodName,String[] argTypes,Object[] args,Class<T> clazz,GenericContext context) throws GenericException;
    }
    // Type definition
    public final class GenericObject implements Serializable{
        private String type;
        private Map<String,Object> fields =new HashMap<String,Object>();
    }

    • Server-side interface, custom type, and configuration for publishing a generalized call:

      public interface PeopleService{
    String hello();
    String hello(String arg);
    People hello(People people);
    String[] hello(String[] args);
    People[] hello(People[] peoples);
}
public class People{
    private String name;
    private int age;
   //getter and setter methods
}
<!--Configuration for publishing a generalized interface-->
<bean id="genericService" class="com.aliyun.gts.financial.product.demo.rpc.server.service.PeopleServiceImpl"/>
<sofa:service ref="genericService" interface="com.aliyun.gts.financial.product.demo.service.facade.PeopleService">
     <sofa:binding.bolt/>
</sofa:service>

  • Client side:

    1. Define the generalized service and set the correct target service interface.

      <!--Configuration for calling a generalized interface-->
<sofa:reference interface="com.alipay.sofa.rpc.api.GenericService" id="genericFacade">
     <sofa:binding.bolt>
          <sofa:global-attrs generic-interface="com.aliyun.gts.financial.product.demo.service.facade.PeopleService"/>
     </sofa:binding.bolt>
</sofa:reference>
      Important

      The interface in the reference tag must be the GenericService interface defined by the framework. The generic-interface in the global-attrs tag is where you specify the actual target service interface. Because the interface in the reference tag is always GenericService, you can use the id of the reference tag to distinguish between generalized calls to different service interfaces.

    2. Call the target method through the methods of GenericService.

          @Controller
    public class TestController{
        private String peoplePath ="com.aliyun.gts.financial.product.demo.rpc.bean.People";
        private static final Logger logger =LoggerFactory.getLogger(TestController.class);
        /**
         * Injected by name by default
         */
        @Autowired
        private GenericService genericFacade;
        /**
         * Use $invoke for scenarios without parameters
         * The $invoke method is used only when the parameter types can be loaded by the current application's class loader. Use this method if only basic types are involved.
         * Generalized call to the String hello() method
         */
        @GetMapping("/test/invokeWithoutArgs")
        @ResponseBody
        @Produces("application/json;charset=UTF-8")
        public void invokeWithoutArgs(){
            String result =(String) genericFacade.$invoke("hello",
                    newString[]{},
            new Object[]{});
            if(logger.isInfoEnabled()){
                logger.info("Generic invoke result: {}", result);
            }
        }
        /**
         * Call $invoke with parameters
         * Generalized call to the String hello(String arg) method;
         */
        @GetMapping("/test/invokeBasicTypeMethod")
        @ResponseBody
        @Produces("application/json;charset=UTF-8")
        public void invokeBasicTypeMethod(){
            String result =(String) genericFacade.$invoke(
                    "hello",
                    new String[]{String.class.getName()},
            new Object[]{"BasicType"});
            if(logger.isInfoEnabled()){
                logger.info("Generic invoke result: {}", result);
            }
        }
        /**
         * Call $genericInvoke for scenarios where parameter types cannot be loaded by the current application's class loader
         * Generalized call to the People hello(People people) method;
         */
        @GetMapping("/test/invokeCustomTypeMethod")
        @ResponseBody
        @Produces("application/json;charset=UTF-8")
        public void invokeCustomTypeMethod(){
            // Specify the full path class name in the constructor
            GenericObject genericPeopleObject =new GenericObject(peoplePath);
            // Call putField to specify the field value
            genericPeopleObject.putField("name","Lilei");
            genericPeopleObject.putField("age",15);
            Object result = genericFacade.$genericInvoke(
                    "hello",
                    newString[]{peoplePath},
            new Object[]{genericPeopleObject});
            // The returned type is still GenericObject
            if(logger.isInfoEnabled()){
                logger.info("Type of result: {}", result.getClass().getName());
            }
        }
        /**
         * Call $genericInvoke with an array parameter
         * Generalized call to the String[] hello(String[] args) method;
         */
        @GetMapping("/test/invokeBasicArrayTypeMethod")
        @ResponseBody
        @Produces("application/json;charset=UTF-8")
        public void invokeBasicArrayTypeMethod(){
            String[] results =(String[]) genericFacade.$genericInvoke(
                    "hello",
                    new String[]{new String[]{}.getClass().getName()},
            new Object[]{new String[]{"BasicArrayType"}});
            // The returned type is still GenericObject
            if(logger.isInfoEnabled()){
                for(String result : results){
                    logger.info("Generic invoke result: {}", result);
                }
            }
        }
        /**
         * Call $genericInvoke with a custom type array
         * People[] hello(People[] peoples);
         */
        @GetMapping("/test/invokeCustomArrayTypeMethod")
        @ResponseBody
        @Produces("application/json;charset=UTF-8")
        public void invokeCustomArrayTypeMethod(){
            GenericObject genericObject =new GenericObject(peoplePath);
            // Call putField to specify the field value
            genericObject.putField("name","HanMeimei");
            genericObject.putField("age",14);
            // For server-side reflection, class.forName has specific format requirements for array types
            String genericObjArrayType ="[L"+ peoplePath +";";
            GenericObject[] genericObjArray =new GenericObject[]{genericObject};
            GenericArray resultArray =(GenericArray) genericFacade.$genericInvoke("hello",
                    new String[]{genericObjArrayType},
            new Object[]{genericObjArray});
            for(Object result : resultArray.getObjects()){
                logger.info(result.toString());
            }
        }
    }

How to use generalized calls

Two methods are currently available:

  • $invoke: Supported only when the method parameter types exist in the current application's ClassLoader.

  • $genericInvoke: Supported when the method parameter types do not exist in the current application's ClassLoader.

The following is a usage example:

  • Service provider:

    • Service reference example:

      <!-- Reference a BOLT service -->
<sofa:referenceinterface="com.alipay.sofa.rpc.api.GenericService"id="genericService">
    <sofa:binding.bolt>
        <sofa:global-attrsgeneric-interface="com.alipay.test.SampleService"/>
    </sofa:binding.bolt>
</sofa:reference>

    • Server-side service definition example:

      /*** Java Bean*/
public class People{
    private String name;
    private int    age;
    // getters and setters
}
/** * Interface provided by the server */
interface SampleService{
    String hello(String arg);
    People hello(People people);
}

  • Client side:

    • Generalized call example:

          /** * Consumer test class. */
    public class ConsumerClass{
        GenericService genericService;
        public void do(){
            // $invoke is supported only when the method parameter types exist in the current application's ClassLoader.
            genericService.$invoke("hello",new String[]{String.class.getName()},new Object[]{"I'm an arg"});

            // $genericInvoke is supported when the method parameter types do not exist in the current application's ClassLoader.      
            //Construct the parameter.
            GenericObject genericObject =new GenericObject("com.alipay.sofa.rpc.test.generic.bean.People");//Specify the full path class name in the constructor.
            genericObject.putField("name","Lilei");// Call putField to specify the field value.
            genericObject.putField("age",15);
            // Make the call without specifying a return type. The result type is GenericObject.
            Object obj = genericService.$genericInvoke("hello",new String[]{"com.alipay.sofa.rpc.test.generic.bean.People"},new Object[]{ genericObject });
            Assert.assertTrue(obj.getClass()==GenericObject.class);
            // Make the call and specify the return type.
            People people = genericService.$genericInvoke("hello",new String[]{"com.alipay.sofa.rpc.test.generic.bean.People"},new Object[]{ genericObject },People.class);

            // Use of generalized calls in an LDC architecture.
            // Construct a GenericContext object.
            AlipayGenericContext genericContext =new AlipayGenericContext();
            genericContext.setUid("33");
            // Make the call.
            People people = genericService.$genericInvoke("hello",new String[]{"com.alipay.sofa.rpc.test.generic.bean.People"},new Object[]{ genericObject },People.class, genericContext);
}
      Important

      When you call the $genericInvoke(String methodName, String[] argTypes, Object[] args) interface, classes from packages other than com.sun, java, javax, org.ietf, org.ogm, org.w3c, org.xml, sunw.io, and sunw.util are serialized into GenericObject.

What are the scenarios for generalized calls?

Generalized calls allow a client to invoke a service without depending on the service's interface. Currently, the only supported method for SOFARPC generalized calls is using the Bolt communication protocol with Hessian2 as the serialization protocol.

Common scenarios for generalized calls:

During development, some third-party applications may not want to depend on your custom interface JAR files but still want to invoke your service. Another scenario is to create a simple microservice gateway that does not depend on JAR files.

How to perform custom processing for RPC requests, such as whitelist filtering

You can use filters to filter RPC interfaces, for example, for IP blacklists and whitelists or token validation.

To implement whitelist filtering, follow these steps:

  1. Inherit the SOFA Filter abstract class and implement its invoke method:

        @Component
    public class WhiteIpFilter extends Filter{
        @Value("${security.firewall.whiteIps}")
        private String whiteIpList;

        @Override
        public boolean needToLoad(FilterInvoker invoker){
               return true;
        }
        @Override
        public SofaResponse invoke(FilterInvoker invoker,SofaRequest request) throws SofaRpcException{
            RpcInternalContext context =RpcInternalContext.getContext();
            InetSocketAddress remoteAddress = context.getRemoteAddress();
            finalString remoteIp = remoteAddress.getHostString();
            if(whiteIpList.contains(remoteIp)){
                return invoker.invoke(request);
            }else{
                SofaResponse sofaResponse =new SofaResponse();
                sofaResponse.setErrorMsg("Illegal IP: "+ remoteIp +" access. Contact the administrator.");
                return sofaResponse;
            }
        }
    }

  2. Configure the filter on the interface that requires whitelist validation:

    <sofa:service interface="cloud.provider.facade.CallerService" ref="callerService">
   <sofa:binding.bolt>
        <sofa:global-attrs filter="whiteIpFilter"/>
   </sofa:binding.bolt>
   <sofa:binding.rest/>
</sofa:service>

Is there example code for file uploads and downloads?

The SOFARPC REST protocol uses Resteasy at its core, which can be used for file uploads and downloads.

The main steps are as follows:

  1. Declare the Facade interface:

    public interface FileServiceFacade{
   @GET
   @Path("/files/{fileName}")
   @Produces("text/plain")
   Response downloadFile(@PathParam("fileName")String fileName) throws Exception;
   @POST
   @Path("/files")
   @Consumes("multipart/form-data")
   Response uploadFile(MultipartFormDataInput input) throws IOException;
}

  2. Implement the download method:

    @Override
    public Response downloadFile(String fileName) throws Exception {
          // Specify a directory to store the file. 
          final String dir = "/destdir/";
          // Check if the filename is null or empty.
          if (fileName == null || fileName.isEmpty()) {
                ResponseBuilder response = Response.status(Status.BAD_REQUEST);
                return response.build();
          }
    // The filename is encoded in UTF-8.
    String utfFileName = URLDecoder.decode(fileName, "utf-8");
    File file = new File(dir + utfFileName);
   // Check if the file exists.
    if (!file.exists()) {
         ResponseBuilder response = Response.ok((Object) file);
         // Set the request header.
         response.header("Content-Disposition", "attachment; filename=" + utfFileName);
         // Return the download response.
         return response.build();
    }
    Note

    If the filename contains Chinese characters, ensure that the character encoding is consistent on both the server and the client.

  3. Implement the upload method:

    @Override
    public Response uploadFile(MultipartFormDataInput input)throws IOException{
        final String UPLOAD_FILE_PATH ="/Users/yuanshaopeng/Desktop/temp/";
        Map<String,List<InputPart>> uploadForm = input.getFormDataMap();
        // httpclient
        // Get file name
        //String fileName = uploadForm.get("fileName").get(0).getBodyAsString();
        // Get file data to save
        //List<InputPart> inputParts = uploadForm.get("attachment");
        // http mode
        List<InputPart> inputParts = uploadForm.get("uploadedFile");
        String fileName ="";
        for(InputPart inputPart : inputParts){
            try{
                @SuppressWarnings("unused")
                MultivaluedMap<String,String> header = inputPart.getHeaders();
                fileName = getFileName(header);
                byte[] bytes =IOUtils.toByteArray(inputPart.getBody(InputStream.class,null));
                log.info("Uploaded file size: "+ bytes.length);
                File desFile =new File(UPLOAD_FILE_PATH + fileName);
                FileUtils.writeByteArrayToFile(desFile, bytes);
                System.out.println("Success !!!!!");
            }catch(Exception e){
                e.printStackTrace();
            }
        }
        return Response.status(200).entity("Upload file name : "+ fileName).build();
    }

    private String getFileName(MultivaluedMap<String,String> header){
        String[] contentDisposition = header.getFirst("Content-Disposition").split(";");
        for(String filename : contentDisposition){
            if((filename.trim().startsWith("filename"))){
                String[] name = filename.split("=");
                String finalFileName = name[1].trim().replaceAll("\"","");
                return finalFileName;
            }
        }
        return "unknown";
    }
}

How to implement a service gateway that does not depend on a Facade interface

The general approach is as follows:

  1. Use a generic RESTful protocol for access.

  2. The backend parses the incoming data based on the generalized interface.

  3. Specify the routing configuration through the agent's filter.

For details, see the filter design of spring-cloud-gateway and the generalization design of the SOFABolt protocol. The following extensions may be required:

  • Dynamic routing

  • Filter interaction with the service registry

  • Cache design

  • Automatic code completion

How to pass data with an RPC request

Important

This feature is disabled by default. Enabling it affects performance, so use it with caution.

To implement this, follow these steps:

  1. Enable the configuration.

    Add the rpc-config.json file to the resource directory.

    {"invoke.baggage.enable":true}

  2. Implement the code.

    Example:

    System.out.println(RpcInvokeContext.isBaggageEnable());
RpcInvokeContext context =RpcInvokeContext.getContext();
context.putRequestBaggage("hellod","lolo");
    Note

    RpcInvokeContext is an RPC execution context. In this context, you can add data to RequestBaggage. The data must be of the string type.

How to set the port for an exposed SOFARPC service

In SOFARPC services, the default ports for protocols are as follows:

  • BOLT protocol: 12200

  • REST protocol: 8341

Therefore, you can set parameters in the /resources/config/application.properties file to specify the port for the exposed service. Example:

com.alipay.sofa.rpc.bolt.port=12202
com.alipay.sofa.rpc.rest.port=8765

If a service has multiple implementations, how should it be published and referenced?

If a service has multiple implementations, you can configure a unique-id as its unique identifier when exposing and referencing the service. Example:

<!-- Service 1 -->
<sofa:service ref="sampleServiceBean1" interface="com.alipay.APPNAME.facade.SampleService" unique-id="service1">
     <sofa:binding.bolt/>
</sofa:service>
<!-- Service 2 -->
<sofa:service ref="sampleServiceBean2" interface="com.alipay.APPNAME.facade.SampleService" unique-id="service2">
    <sofa:binding.bolt/>
</sofa:service>

What parameters are available for RPC service timeout control?

Both service publishing and referencing have timeout control configurations. Methods can also have timeout controls. The priority of timeout periods is as follows: referenced service method timeout > referenced service global timeout > service publisher method timeout > service publisher global timeout. Example:

<sofa:binding.bolt>
     <sofa:global-attrs timeout="5000"/>
     <sofa:method name="message" type="future" timeout="25000"/>
</sofa:binding.bolt>

What should I be aware of when exposing an interface with the RESTful protocol?

Note the following:

  • First, clarify the middleware version you are using. The recommended version for sofaboot-enterprise is 3.4.*. When you use this version to develop with RESTful interfaces, you only need to import the RPC starter package dependency. You do not need to import the REST starter dependency. Otherwise, two RESTful servers will be started, causing port 8341 to be occupied.

  • When developing with REST interfaces, you need to expose Path and WS-related annotations on the interface. Otherwise, the service will not know what the specific request is. For more information about using WS annotations, see Using @Path and @GET, @PUT, etc.

How to expose a service with both RESTful and Bolt protocols simultaneously

When you publish a service, you can simply declare the binding for both protocols at the same time, as follows:

<bean id="demoServiceImpl" class="com.alipay.sofa.samples.rpc.DemoServiceImpl"/>
<sofa:service ref="demoServiceImpl" interface="com.alipay.sofa.samples.rpc.DemoService">
     <sofa:binding.rest/>
     <sofa:binding.bolt/>
</sofa:service>

RPC calls in the dev environment fail with an "RPC-02306: Can not get the service address of service" error

Symptoms

An RPC call in the dev environment failed, and the middleware_error.log file shows an error that the service endpoint URL cannot be found.

Cause:

The direct connection switch is enabled in the property configuration, but the URL configured in the code does not match the actual RPC service address.

  • run.mode=test is configured in the application-dev.properties file.

  • test-url="${servicename_tr_service_ur}" is configured in the code.

  • The address that servicename_tr_service_ur points to does not match the actual RPC service address.

  • The RPC provider and consumer projects use different SOFABoot versions.

The following figure shows a code example:Code example

Solution:

Comment out run.mode=test in the application-dev.properties file, or upgrade the SOFABoot versions of the RPC provider and consumer projects to the same version. For more information, see SOFABoot Version Guide.

RPC calls are slow and time out without reporting a timeout exception

Symptoms:

SOFA RPC uses a REST API to trigger generalized RPC calls. Each call takes 30 seconds and does not report a timeout exception. The business logs confirm that the process took 30 seconds from start to finish.

Cause:

A DNS configuration error may cause a timeout.

Solutions:

Add an IP-to-hostname mapping in the /etc/hosts file to resolve the issue.

RPC Registration Failure

Symptoms

RPC registration failed.

Cause

In the application.properties file, run.mode is set to dev.

Solutions

Delete the run.mode configuration or set it to normal.

Error during RPC application startup: "Can't find BindingConverter of type binding.tr"

Symptoms:

The following error occurs when an RPC application starts:

Causedby: org.springframework.beans.factory.BeanCreationException:Error creating bean with name 'secretFacade':Invocation of init method failed; nested exception is com.alipay.sofa.runtime.api.ServiceRuntimeException:Can't find BindingConverter of type binding.tr

Cause:

The rpc-enterprise-sofa-boot-starter is commented out, and this JAR package provides the following bindings:

  • rpc-enterprise-sofa-boot/3.2.2/rpc-enterprise-sofa-boot-3.2.2.jar!/com/alipay/boot/sofarpc/converter/TrBindingConverter.class

  • rpc-sofa-boot/3.2.2/rpc-sofa-boot-3.2.2.jar!/com/alipay/sofa/rpc/boot/runtime/converter/BoltBindingConverter.class

Solution:

Import the rpc-enterprise-sofa-boot-starter JAR package.

The Tr interface cannot find a service endpoint

Cause:

Only the client migrated to the shared middleware, but the server-side did not.

Solution:

Migrate the server-side to shared middleware.

RPC Duplicate Publishing Error

Symptoms:

The specific error message is as follows:

java.util.concurrent.ExecutionException: com.alipay.sofa.rpc.core.exception.SofaRpcRuntimeException: RPC-010010014: The Consumer config with KEY [bolt://com.aliyun.fsi.insurance.aboss.rule.facade.AbossRuleFacade:] was published more than [3] times. This may be caused by a faulty configuration. Please check. 
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
  at SingelFormula.main(SingelFormula.java:47)
Caused by: com.alipay.sofa.rpc.core.exception.SofaRpcRuntimeException: RPC-010010014: The Consumer config with KEY [bolt://com.aliyun.fsi.insurance.aboss.rule.facade.AbossRuleFacade:] was published more than [3] times. This may be caused by a faulty configuration. Please check. 
  at com.alipay.sofa.rpc.bootstrap.DefaultConsumerBootstrap.refer(DefaultConsumerBootstrap.java:136)
  at com.alipay.sofa.rpc.config.ConsumerConfig.refer(ConsumerConfig.java:926)
  at SingelFormula$RuleSingle.call(SingelFormula.java:87)
  at SingelFormula$RuleSingle.call(SingelFormula.java:62)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

Solution:

This problem is usually caused by registering the service multiple times. We recommend that you check the service registration and delete any redundant registration information.

Is there a data volume limit for a single RPC transmission?

There is no inherent limit on the data size of a single RPC transfer. However, for performance reasons, we recommend that you set the size to 4 KB or less. If the data size exceeds this limit, overflow issues may occur in high-concurrency scenarios. The error keyword is "maybe write overflow". We recommend that you calculate the actual required size using the following system parameters:

Parameter

Default value

-Dbolt.netty.buffer.low.watermark

32 × 1024

-Dbolt.netty.buffer.high.watermark

64 × 1024

What client scenarios does rate limiting support?

The following are the main clients currently supported:

  • Spring MVC

    • Code intrusion: None

    • Rate limiting method: Web URL

  • SOFA RPC Bolt

    • Code intrusion: None

    • Rate limiting method: Interface method

  • Standard Spring Bean

    • Code intrusion: Uses Aspect-Oriented Programming (AOP)

    • Rate limiting method: Interface method

The following clients are not supported:

  • SOFA RPC REST

    • Code changes required: None

    • Rate limiting methods: Interface method, Web URL

  • SOFA REST (RESTEASY)

    • Code changes required: None

    • Rate limiting methods: Interface method, Web URL

    For SOFA REST, you can implement rate limiting using Aspect-Oriented Programming (AOP) to intercept the REST bean. To do this, follow these steps:

    1. Define the interface method to intercept. The following code provides an example:

      @Path(URLConstants.REST_API_PEFFIX +"/users")
@Consumes(RestConstants.DEFAULT_CONTENT_TYPE)
@Produces(RestConstants.DEFAULT_CONTENT_TYPE)
public interface SampleRestFacade{
    @GET
    @Path("/{userName}")
    public RestSampleFacadeResp<DemoUserModel> userInfo(@PathParam("userName") String userName)throws CommonException;
}

    2. Define the bean. The following code provides an example:

      <bean id="sampleRestFacadeRest" class="com.hula.sofa.demos.guardian.endpoint.impl.SampleRestFacadeRestImpl"/>

    3. Configure the AOP. The following code provides an example:

      <import resource="classpath:META-INF/spring/guardian-sofalite.xml"/>
<bean class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
      <property name="interceptorNames">
            <list>
                <value>guardianExtendInterceptor</value>
            </list>
      </property>
     <property name="beanNames">
            <list>
                 <!-- Configure the bean to intercept -->
                 <value>sampleRestFacadeRest</value>
            </list>
     </property>
    <!-- To use a CGLIB proxy, uncomment the following line -->
    <!-- <property name="optimize" value="true" /> -->
</bean>

    4. Configure rate limiting. The following figure shows an example:xml配置.png

Where is the generated cache address file for SOFARPC in DEV mode?

DEV mode does not register with the registry and instead generates a temporary file locally. You can find the "Write backup file to" field in /logs/rpc/rpc-registry.log to confirm the name of the temporary file. This file is generated only on the server-side.

How to check if a project has been registered or started successfully?

You can check in the following ways:

  • On the project server, run the ps -ef | grep java command to check for the Java process. If the process exists, the project is running. Otherwise, it is not.

  • On the project server, run the following command and check the output:

    • For RPC versions 3.0 and later

      curl http://localhost:8080/actuator/readiness

      Replace 8080 with the actual port number of your project.

    • For RPC versions earlier than 3.0

      curl http://localhost:8080/health/readiness

    If the output does not contain a down field, the project is running. Otherwise, it is not.

  • On the project server, run the ps -ef | grep 9600 command to check if port 9600 is in use. If the port is in use, the service registry is connected. Otherwise, the connection is not established.

RPC Call Timeout

To fix the issue, perform the following steps:

  1. Check the rpc-server-digest.log log file.

    Confirm whether the processing time of the service provider is normal during the time segment of the call timeout.

  2. Check the line after the abnormal log for a log with the same traceID to confirm whether other calls exist.

    • If one exists, troubleshoot other calls that have the same traceID.

    • If one does not exist, optimize the features of your service.

Previous：Service Throttling FAQNext：Service Mesh