Service Mesh uses SOFATracer and Zipkin for distributed tracing. It records network calls in the invocation chain as logs to provide visibility into the calls. These traces can be used for quick fault detection and service administration. This topic describes the log formats and provides examples for SOFATracer.
SOFATracer
Service Mesh supports three types of microservices: SOFA, Dubbo, and Spring Cloud. SOFATracer records microservice logs to the
The following sections describe the SOFATracer logs and provide examples for various microservice types. The logs are stored in the /home/admin/logs/tracelog/mosn directory.
SOFA
Service caller
Log file name: rpc-client-digest.log.
The following is an example log:
2021-02-05 13:46:44.264, //Log printing time
sofa-echo-server, //Current application name
1e49a2be16125040042601246****2,//TraceId
0,//RpcId
com.alipay.sofa.ms.service.SofaEchoService:1.0,//Service name
echo,//Method name
bolt,//Protocol
SYNC,//Invocation method
30.**.**.190:12200,//Target address
sofa-echo-server,//Target system name
GZ00B,//Target zone
,//Target IDC
,//Target city
,//uid
00,//Result code
388B,//Request size
105B,//Response size
2ms,//Invocation duration
0ms,//Connection setup duration
0ms,//Request serialization duration
0ms,//Timeout reference duration
,//Current thread name
,//Routing record
,//Flexible data bit
,//Whether flexibility is required
,//Forwarded service name
127.0.0.1,//Client IP
61181,//Client port
,//Current zone
F,//Whether it is a physical machine
,//systemMap
mosn_cluster=cb7aaa7a31fd0490299c530e83a868231
&mosn_namespace=default
&mosn_log=true
&mosn_data_id=com.alipay.sofa.ms.service.SofaEchoService:1.0@DEFAULT
&mosn_data_ver=&,//System passthrough data
606.52µs//MOSN processing timeThe log format is as follows:
Log content | Description |
2021-02-05 13:46:44.264 | The time when the log was printed. The format is |
sofa-echo-server | The name of the current application. |
1e49a2be16125040042601246**** | TraceId. The unique identifier for distributed tracing. |
0 | RpcId. |
com.alipay.sofa.ms.service.SofaEchoService:1.0 | The service name. |
echo | The method name. |
bolt | The protocol. Valid values are bolt and rest. |
SYNC | The invocation method.
|
30.**.**.190:12200 | The target address. |
sofa-echo-server | The name of the target application. |
GZ00B | The target zone. |
- | The target Internet Data Center (IDC). |
- | The target city. Example: beijing. |
- | The user ID (uid). |
00 | The result code.
|
388B | The request size in bytes (B). |
105B | The response size in bytes (B). |
2ms | The invocation duration in milliseconds (ms). |
0ms | The connection setup duration in milliseconds (ms). |
0ms | The request serialization duration in milliseconds (ms). |
0ms | The timeout reference duration in milliseconds (ms). |
- | The name of the current thread. |
- | The routing record. |
- | The flexible data bit. |
- | Indicates whether flexibility is required. |
- | The name of the forwarded service. Example: |
127.0.0.1 | The client IP address. |
61181 | The client port. |
- | The current zone. Example: Gzone. |
F | Indicates whether the host is a physical machine. |
- | systemMap |
mosn_cluster=cb7aaa7a31fd0490299c53****a868231 &mosn_namespace=default &mosn_log=true &mosn_data_id=com.alipay.sofa.ms.service.SofaEchoService:1.0@DEFAULT &mosn_data_ver=& | The system supports data pass-through. |
606.52µs | The MOSN processing time in microseconds (µs). |
Service provider
Log file name: rpc-server-digest.log.
The following is an example log:
2021-02-05 13:47:13.311,//Log printing time
sofa-echo-server,//Current application name
0ba685081612504033235341410,//TraceId
0,//RpcId
com.alipay.sofa.ms.service.SofaEchoService:1.0,//Service name
echo,//Method name
bolt,//Protocol
,//Invocation method
11.**.**.8:41711,//Caller URL
,//Caller application name
,//Caller zone
,//Caller IDC
2ms,//Request processing duration (ms)
0ms,//Server-side response serialization duration (ms)
,//Current thread name
00,//Result code
,//Indicates that this is a forwarded call. The name and method of the forwarded service. Example: com.test.service.testservice.TestService:1.0:biztest---doProcess
,//Indicates that the call was not forwarded.
0,//RPC thread pool wait time
,//System passthrough data
mosn_tls_state=off&mosn_cluster=cb7aaa7a31fd0490299c530e83a868231
&mosn_namespace=default
&mosn_log=true&mosn_tls_state=off
&mosn_data_id=com.alipay.sofa.ms.service.SofaEchoService:1.0@DEFAULT
&mosn_data_ver=&,//Passthrough data (key-value format)
681.534µs,//MOSN processing time
373B,//Request size
105B//Response sizeThe log format is as follows:
Log content | Description |
2021-02-05 13:47:13.311 | The time when the log was printed. The format is |
sofa-echo-server | The name of the current application. |
0ba68508161250403323534**** | TraceId. The unique identifier for distributed tracing. |
0 | RpcId. |
com.alipay.sofa.ms.service.SofaEchoService:1.0 | The service name. |
echo | The method name. |
bolt | The protocol. Valid values are bolt and rest. |
- | The invocation method.
|
11.**.**.8:41711 | The caller URL. |
- | The name of the caller application. |
- | The caller zone. |
- | The caller IDC. |
2ms | The request processing duration in milliseconds (ms). |
0ms | The server-side response serialization duration in milliseconds (ms). |
- | The name of the current thread. |
00 | The result code.
|
- | Indicates that this is a forwarded call. The name and method of the forwarded service. Example: |
- | Indicates that the call was not forwarded. |
0 | The RPC thread pool wait time. |
- | The system passthrough data in key-value format. This data is used to transmit system disaster recovery information. |
mosn_tls_state=off&mosn_cluster=cb7aaa7a31fd0490299c530e83a86**** &mosn_namespace=default &mosn_log=true&mosn_tls_state=off &mosn_data_id=com.alipay.sofa.ms.service.SofaEchoService:1.0@DEFAULT &mosn_data_ver=& | The passthrough data in key-value format. |
681.534µs | The MOSN processing time in microseconds (µs). |
373B | The request size in bytes (B). |
105B | The response size in bytes (B). |
Dubbo
Service caller
Log file name: rpc-client-digest.log.
The following is a sample log:
2021-02-05 11:57:28.284,//Log printing time
reservation-service,//Current application name
1e49a2be1612497448281145168886,//TraceId
0,//RpcId
com.alipay.sofa.ms.service.EchoService,//Service name
echo,//Method name
Dubbo,//Protocol
SYNC,//Invocation method
30.**.**.190:30800,//Target address
reservation-service,//Target system name
GZ00B,//Target zone
,//Target IDC
,//Target city
,//uid
00,//Result code (00=Success, 01=Business exception, 02=RPC logic error, 03=Timeout, 04=Routing failed)
242B,//Request size
118B,//Response size
2ms,//Invocation duration
0ms,//Connection setup duration
0ms,//Request serialization duration
0ms,//Timeout reference duration
,//Current thread name
,//Routing record
,//Flexible data bit
,//Whether flexibility is required
,//Forwarded service name
30.**.**.190,//Client IP
57510,//Client port
,//Current zone
F,//Whether it is a physical machine
,//systemMap
mosn_cluster=cb7aaa7a31fd0490299c530e83a868231
&mosn_namespace=default&mosn_log=true
&mosn_data_id=com.alipay.sofa.ms.service.EchoService:aaa:bbb@dubbo
&mosn_data_ver=&,//System passthrough data
325.799µs//MOSN processing timeThe Dubbo log format is the same as the SOFA log format.
Service provider
Log file name: rpc-server-digest.log.
The following is an example log:
2021-02-05 11:57:53.431,//Log printing time
reservation-service,//Current application name
1e49a2be161249747343014806****,//TraceId
0,//RpcId
com.alipay.sofa.ms.service.EchoService,//Service name
echo,//Method name
Dubbo,//Protocol
,//Invocation method
30.**.**.190:57523,//Caller URL
,//Caller application name
,//Caller zone
,//Caller IDC
1ms,//Request processing duration (ms)
0ms,//Server-side response serialization duration (ms)
,//Current thread name
00,//Result code (00=Success, 01=Business exception, 02=RPC logic error)
,//Indicates that this is a forwarded call. The name and method of the forwarded service. Example:
com.test.service.testservice.TestService:1.0:biztest---doProcess
,//Indicates that the call was not forwarded.
0,//RPC thread pool wait time
,//System passthrough data (key-value format, used to transmit system disaster recovery information)
mosn_tls_state=off&mosn_cluster=cb7aaa7a31fd0490299c530e83a868231
&mosn_namespace=default&mosn_log=true&mosn_tls_state=off
&mosn_data_id=com.alipay.sofa.ms.service.EchoService:aaa:bbb@dubbo
&mosn_data_ver=&,//Passthrough data (key-value format)
285.268µs,//MOSN processing time
242B,//Request size
118B//Response sizeThe Dubbo log format is the same as the SOFA log format.
SpringCloud
Service caller
Log file name: springcloud-client-digest.log
The following is an example of a log:
2021-02-05 11:55:33.61,//Log printing time
reservation-service,//Current application name
1e49a2be161249733360713416****,//TraceId
0,//RpcId
127.0.0.1:10088,//host
/echo/name,//uri
HTTP,//Protocol
SYNC,//Invocation method
30.**.**.190:10080,//Target address
reservation-service,//Target system name
GZ00B,//Target zone
,//Target IDC
,//Target city
,//uid
00,//Result code (00=Success, 01=Business exception, 02=RPC logic error, 03=Timeout, 04=Routing failed)
18B,//Request size
20B,//Response size
2ms,//Invocation duration
0ms,//Connection setup duration
0ms,//Request serialization duration
0ms,//Timeout reference duration
,//Current thread name
,//Routing record
,//Flexible data bit
,//Whether flexibility is required
,//Forwarded service name
127.0.0.1,//Client IP
57817,//Client port
,//Current zone
F,//Whether it is a physical machine
,//systemMap
mosn_cluster=cb7aaa7a31fd0490299c530e83a868231
&mosn_namespace=default&mosn_log=true&,//System passthrough data
426.311µs//MOSN processing timeThe log is as follows:
Log content | Description |
2021-02-05 11:55:33.61 | The time when the log was printed. The format is |
reservation-service | The name of the current application. |
1e49a2be161249733360713416**** | TraceId. The unique identifier for distributed tracing. |
0 | RpcId. |
127.0.0.1:10088 | The host. |
/echo/name | The URI. |
HTTP | The protocol. Valid values are bolt and rest. |
SYNC | The invocation method.
|
30.**.**.190:10080 | The target address. |
reservation-service | The name of the target system. |
GZ00B | The target zone. |
- | The target IDC. |
- | The target city. |
- | User ID. |
00 | The result code.
|
18B | The request size in bytes (B). |
20B | The response size in bytes (B). |
2ms | The invocation duration in milliseconds (ms). |
0ms | The connection setup duration in milliseconds (ms). |
0ms | The request serialization duration in milliseconds (ms). |
0ms | The timeout reference duration in milliseconds (ms). |
- | The name of the current thread. |
- | The routing record. |
- | The flexible data bit. |
- | Indicates whether flexibility is required. |
- | The name of the forwarded service. |
127.0.0.1 | The client IP address. |
57817 | The client port. |
- | The current zone. |
F | Indicates whether the host is a physical machine. |
- | systemMap. |
mosn_cluster=cb7aaa7a31fd0490299c5****a868231 &mosn_namespace=default&mosn_log=true& | The system passthrough data. |
426.311µs | The MOSN processing time in microseconds (µs). |
Service provider
Log file name: springcloud-server-digest.log
The following is a log example:
2021-02-05 11:56:30.962,//Log printing time
reservation-service,//Current application name
1e49a2be1612497390960139868886,//TraceId
0,//RpcId
127.0.0.1:10088,//host
/echo/name/aaa,//uri
HTTP,//Protocol
,//Invocation method
30.**.**.190:57818,//Caller URL
reservation-service,//Caller application name
,//Caller zone
,//Caller IDC
1ms,//Request processing duration (ms)
0ms,//Server-side response serialization duration (ms)
,//Current thread name
00,//Result code (00=Success, 01=Business exception, 02=RPC logic error)
,//Indicates that this is a forwarded call. The name and method of the forwarded service. Example:
com.test.service.testservice.TestService:1.0:biztest---doProcess
,//Indicates that the call was not forwarded.
0,//RPC thread pool wait time
,//System passthrough data (key-value format, used to transmit system disaster recovery information)
mosn_tls_state=off&mosn_cluster=cb7aaa7a31fd0490299c530e83a868231
&mosn_namespace=default&mosn_log=true&,//Passthrough data (key-value format)
262.787µs,//MOSN processing time
0B,//Request size
16B//Response sizeThe log content includes the following:
Log content | Description |
2021-02-05 11:56:30.962 | The time when the log was printed. The format is |
reservation-service | The name of the current application. |
1e49a2be161249739096013986**** | TraceId. The unique identifier for distributed tracing. |
127.0.0.1:10088 | The host. |
/echo/name/aaa | The URI. |
HTTP | The protocol. Valid values are bolt and rest. |
- | The invocation method.
|
30.**.**.190:57818 | The caller URL. |
reservation-service | The name of the caller application. |
- | The caller zone. |
- | The caller IDC. |
1ms | The request processing duration in milliseconds (ms). |
0ms | The server-side response serialization duration in milliseconds (ms). |
- | The name of the current thread. |
00 | The result code.
|
- | Indicates that this is a forwarded call. The name and method of the forwarded service. Example: |
- | Indicates that the call was not forwarded. |
0 | The RPC thread pool wait time. |
- | The system passthrough data in key-value format. This data is used to transmit system disaster recovery information. |
mosn_tls_state=off&mosn_cluster=cb7aaa7a31fd0490299c530e83a868231 &mosn_namespace=default&mosn_log=true& | The passthrough data in key-value format. |
262.787µs | The MOSN processing time in microseconds (µs). |
0B | The request size in bytes (B). |
16B | The response size in bytes (B). |