1. Introduction

In this chapter, we describe the overall architecture of mOS. We first briefly describe the threading model of the mTCP networking stack before we introduce the changes that were needed to develop the monitoring library.

1.1. mOS (or mTCP) end application threading model

mOS default threading model is based on mTCP: it is designed to maximize the performance in a general-purpose multicore machine by distributing ingress traffic workload (in a flow-wise symmetric fashion) across multiple cores. mOS assumes that its applications run in a single process and multiple threads model. In other words, a mOS application is composed of multiple identical threads (which we call ‘application threads’) performing the same tasks independently of each other. Each application thread is paired with a mOS thread, which handles the networking service on behalf of the application thread. A typical end host mOS(/mTCP)-based application e.g. epserver follows this model. Figure 2.1 graphically illustrates this architecture.

Workload is distributed over multiple cores through NIC hardware queues that use RSS (Receive-Side Scaling) algorithm to split traffic. In particular, we use Symmetric RSS (S-RSS), that maps bidirectional TCP flows (server->client and client->server) on to the same CPU core. Each mOS thread also has its own dedicated per-core listening queue. This design ensures that every single flow is handled by a mOS thread<—>application thread pair exclusively. By doing so, there is no inter-core communication and the CPU cache is utilized efficiently. The user is suggested to go through mTCP paper to see more optimizations that are implemented as part of this stack.

../_images/mtcp_threading_model.png

Figure 2.1. mOS(/mTCP) end application threading model architecture

A copy of this image first appeared in a powerpoint slide presented in NSDI 2014.

1.2. mOS middlebox application threading model

The initialization sequence of the mOS net library changes in two ways when the user wishes to use the underlying networking stack for monitoring purposes.

  1. It initializes a dual mTCP stack that simulates both client and server networking states.
  2. The runtime model is changed as mOS completely removes the application thread from the system. The middlebox application context (custom middlebox logic) is, in fact, shared within the mOS thread.

The mOS middlebox stack covers all the optimizations that are introduced by mTCP. A typical middlebox application communicates with the mOS stack via callback handlers that get triggered through registered monitoring events. We explain mOS events in mOS Event System subsection. mOS events are registered via monitoring sockets, which we describe in detail in mOS Socket Abstraction subsection.

../_images/mos_threading_model.png

Figure 2.2. mOS monitor application threading model architecture

1.2.1. Application Execution Model

There are two execution models for mOS applications.

  1. Running standalone applications with mOS net library
  2. Running embedded applications with mOS server

1.3. Running standalone middlebox applications with mOS net library

In standalone mode, mOS only allows users to run one middlebox application per mOS thread. A typical standalone mOS application begins by calling an mtcp_init() function which loads the mOS startup configuration file (with specific mOS parameters) and initializes the stack. The mOS configuration file contains a wide array of parameters ranging from selected I/O module (DPDK or PCAP), default receive TCP buffer size to concurrent connections threshold that it can monitor. A mOS developer can use mtcp_create_context() to spawn a mOS thread. We provide a library of monitoring functions through which a mOS monitor can (i) register events and, (ii) retrieve flow-specific meta-data of ingress traffic.

During termination, a mOS application can call mtcp_destroy() to clean up all the resources that are taken by the mOS thread.

We suggest the user to go through mOS Socket Abstraction and mOS Event System sections which describe the programming abstractions through which a mOS middlebox monitor can retrieve and manipulate flow activities. We then recommend going through Sample Applications User Guide to see how one can use our API to create monitoring applications.

1.4. Running embedded monitoring applications with mOS server

Embedded applications support for mOS server will be released in the near future.