Training video
Watch the following video to learn how Service Mesh works:
Current state of MOSN
MOSN is a data plane product. It is deployed as a sidecar in the same pod or virtual machine as the application and runs as an independent process. Initially, MOSN supported traffic interception using a lightweight software development kit (SDK) in a mesh architecture. It currently supports three methods for traffic interception:
Intercept traffic using lightweight SDK port spoofing.
Intercept traffic by modifying the configuration from IP+Port to 127.0.0.1.
Intercept traffic using transparent iptables hijacking.
MOSN supports major frameworks, such as Ant Group's SOFA RPC, Dubbo, and the Spring Cloud technology stack. For service discovery, it uses the classic registry mechanism. The internal registry used is SOFARegistry.
MOSN can be deployed in virtual machine and container scenarios, which allows applications to connect to it. When acting as a service provider, MOSN writes its port number to the registry. When acting as a consumer, MOSN subscribes to services from the registry. This means it retrieves the IP address and port of the provider's MOSN to enable service communication.
MOSN configuration relationships
This section explains the configuration relationships in MOSN within the context of the call chain. In a static configuration, the configuration is divided into several main categories, such as listeners, routers, and clusters. The following figure shows a sample configuration:
Listeners are configurations for listening on ports, such as TCP and UDP. This includes specifying the binding relationships between ports and protocols.
The RPC category supports request-response communication models for proprietary protocols, such as Dubbo, SOFA, and Spring Cloud.
The MQ scenario handles communication for messaging frameworks.
Routers store protocol-related routing configuration mappings. They match the service information in the header to a specific cluster.
Clusters store the list of actual service endpoints in a cluster. For RPC scenarios, the cluster name is typically the interface, based on the mapping between the interface and its list of endpoints.
The numbers in the figure represent the steps in the process:
The client MOSN listens on port 30530 for calls from the client application.
In the MOSN configuration, port 30530 is assigned to the proprietary protocol SK. Therefore, the protocol is named SK.
The MOSN static configuration specifies the routing configuration mapping name as
egress_sk_router. The naming convention is typicallyegress_<protocol_name>_router. For a service provider, the convention isingress_<protocol_name>_router. This configuration mapping functions as an array. It stores various values for the `Service` key from the header, which are usually interface names. It then maps requests to specific clusters based on these values. The.*value is a wildcard. It indicates that if the header contains the `Service` key, the request is forwarded to the corresponding target cluster.If the request header contains any value for `Service`, the request is forwarded to the `skClientToWsServer` cluster.
MOSN can select and invoke any address in the address list of the skClientToWsServer cluster.
When a service provider is online, the MOSN client receives a pushed list of endpoints. MOSN then finds the cluster based on the `dataId`, which is usually the interface name, and dynamically updates the endpoint list.
Note that the SOFARPC folder in the listeners contains many proprietary protocol extensions. By default, MOSN loads them and starts listening on the corresponding ports at startup. For protocols that are not needed in an Apsara Stack deployment, you can remove them using a Dockerfile.
MOSN call relationships
To understand the MOSN call chain, it is important to understand the difference between ingress and egress.
From a sidecar's perspective, ingress and egress depend on whether MOSN forwards traffic to a remote or local service:
egress: Outbound traffic. MOSN receives traffic and forwards it to a remote service provider.
ingress: Inbound traffic. MOSN receives traffic and forwards it to the local service provider.
The following figure shows the MOSN call relationships:
Using the figure as an example, the MOSN call chain is as follows:
The service provider's MOSN writes its ingress protocol port to the registry.
The client's MOSN subscribes to the endpoint list from the registry.
The initial subscription returns the full list of endpoints. The port number is the one bound to the provider's ingress.
The registry pushes endpoint list changes to the client in real time. Each push contains the full list of endpoints.
The client application initiates a business RPC request. The SDK forwards the request to the egress port of the local MOSN instance.
The client's MOSN forwards the RPC request over the network. It uses load balancing to forward the traffic to the ingress port of a service provider's MOSN.
The request reaches the server-side ingress listener and is forwarded to the local application server. The response is sent back along the original TCP connection path.
Core forwarding flow of the mesh
As the data plane, MOSN is divided into four layers:
Network/IO: Responsible for network read and write operations.
Protocol: Responsible for protocol encoding and decoding.
Stream: Responsible for forwarding and coordinating models, such as request and response.
Proxy: Responsible for functions, such as routing.
For an introduction to the original MOSN architecture, see MOSN Architecture Introduction.
For easier understanding, this topic simplifies the MOSN flow into 12 steps, as shown in the figure:
During startup, MOSN exposes a local egress port to accept RPC requests from the client application. MOSN starts two coroutines that run in an infinite loop to handle TCP reads and writes. MOSN retrieves the request byte stream through the read coroutine and passes it to the protocol layer for processing.
MOSN uses a loop in the streamconn implementation to decode until a complete request message is parsed. Once the request message is parsed, an xstream associated with the request is created.
This `xstream` maintains the TCP association with the client application and is later used to send the response.
To forward the request to a machine in the service cluster, MOSN reaches the proxy layer and creates a downstream.
This step performs the following actions:
Executes the request/response filter chain.
Performs route matching.
Performs load balancing.
After selecting a machine in the service cluster, MOSN initializes the connection pool for the host that corresponds to the selected IP address.
At this point, MOSN acts as an intermediary. It acts as a server for the client application and as a client for the remote service.
The upstreamrequest object plays a key role:
Maintains a reference to the client application's TCP connection.
Maintains a reference to the forwarded server's TCP connection to forward client application requests and receive notifications when the server responds.
In the appendHeaders + appendData phase of the upstream, a sender `xstream` is created using the host selected in step 4.
This `xstream` is a client-side stream object that serves two main purposes:
It acts as a client to initialize the client request information and create a stream binding for the request to be forwarded.
Before forwarding, it replaces the request ID to prevent requests from overwriting each other due to connection I/O multiplexing.
NoteRequest overwriting occurs because the client application may have multiple TCP connections, but MOSN forwards requests to the server over a single TCP connection. For example, if two TCP connections from the client simultaneously send a request with `id=1`, MOSN forwards both requests over the same TCP connection to the server. When the response with `id=1` returns, MOSN cannot determine to which of the client application's TCP connections it belongs.
The solution is for MOSN to modify the request ID to be globally unique before forwarding it. This prevents confusion between requests and their corresponding responses.
Because MOSN modifies the request ID before forwarding, it must re-encode the request.
As an optimization, MOSN typically does not re-encode the entire message. Instead, it only modifies a few bytes in the protocol header.
Once the client `xstream` is ready to forward (endOfStream), it sends the request to the downstream host selected in step 4.
The request's coroutine is then blocked.
When MOSN forwards the request to the server host, it creates a new TCP connection. Each TCP connection also has two coroutines to handle reads and writes.
The MOSN client `xstream` receives the response byte stream through the read I/O and delivers it to the upper protocol layer for decoding.
Once the stream is fully decoded into a response object, it notifies the upstream request object.
The `upstream request` holds the downstream for the client request and wakes up the blocked downstream coroutine.
Corresponding to step 2, the MOSN server-side `xstream` is woken up. It replaces the request ID in the received response with the original request ID and calls the protocol layer to re-encode it into a byte stream.
The `xstream` holds the TCP connection from the client application's request. It writes the response back to the client and destroys all request-related resources in MOSN.