Fork me on GitHub
mOS
A Specialized Network Programming Library for Stateful Middleboxes

mOS | Source code | User guides | API documentation | Publications & talks


1. What is mOS?

Stateful middleboxes such as intrusion detection systems and stateful firewalls rely on TCP flow management to keep track of on-going network connections. However, implementing complex TCP state management for high-performance middlebox applications is non-trivial. Unfortunately, there is no networking stack that can be reused to help implement flow-level features needed for custom flow-processing middleboxes. The status quo is that the developers often end up implementing customized flow management submodules for their systems from scratch. Such practice is highly unscalable and error-prone.

Figure 1. mOS overview

Our long term goal is to build a middlebox development framework called mOS (or middlebox Operating System) to assist middlebox developers with common, reusable functionalities needed for middlebox development. Our first release is its networking stack that offers a high-performance, user-space programming API specifically targeted for building software-based stateful monitoring middleboxes (Figure 1). Our library interface provides powerful event-based abstractions (see section 3) that can potentially simplify the codebase of complex middleboxes. We find that applications ported on mOS are more modularized than their counterparts as our stack can abstract intricate TCP state management tasks internally; allowing the applications to solely focus on custom middlebox logic.

mOS is an extension of our earlier work: mTCP, which is a high-performance user-level TCP/IPv4 stack. Our evaluation of mOS applications show that:

2. Benefits of mOS Networking Stack

Our API helps middlebox developers build a wide range of custom L4~L7 applications such as IDSes/IPSes, NATs and L7 protocol analyzers. Users can also build L2~L3 network tools that do not require flow state tracking such as bridges, routers and packet-level redundancy elimination systems. mOS offers three main benefits:

3. mOS Application Programming Interface

Existing (open-source) stateful middleboxes often have flow management modules tightly integrated with their core logic, which often create complex code that is hard to maintain. Our interface allows clean separation of the middlebox logic from the networking core. The key observation for the API is that most middlebox operations can be defined as a set of flow-related events; and their respective callback handlers can be used to express a customized logic.

The mOS networking API provides an elaborate event interface for building middlebox applications. We have identified a list of key flow-related network conditions known as built-in events that represent notable TCP flow state change in a connection. Developers can write many applications with just built-in events and their event handlers. In case built-in events are insufficient, developers can define their own custom events called user-defined events. A user-defined event (UDE) can be defined as an existing event (called base event) and a filter function that detects the concrete condition of the event. The UDE is generated if the base event is triggered and its filter function (invoked as part of base event triggering) returns true.

The mOS API defines a new socket type (MOS_SOCK_MONITOR_STREAM) used to monitor flow events in individual TCP flows.

The following code snippet explains how the events and socket abstractions can be used in a typical mOS application:

static bool
IsSYN(mctx_t m, int sock, int side, event_t event)
{
    struct pkt_info p;

    // Retrieve flow's last packet
    mtcp_getlastpkt(m, sock, side, &p);
    return (p.tcph->syn && !p.tcph->ack);
}

static void
OnFlowStart(mctx_t m, int sock, int side, event_t event)
{
    struct pkt_info p;

    // Retrieve flow's last packet
    mtcp_getlastpkt(m, sock, side, &p);
    // initialize per-flow user context
    struct HTTPContext *http_c = calloc(sizeof(struct HTTPContext));
    http_c->sport = p.tcph->source;
    ...
    mtcp_set_uctx(m, sock, http_c);
}

static void
OnFlowEnd(mctx_t m, int sock, int side, event_t event)
{
    // free up per-flow user context
    free(mtcp_get_uctx(m, sock));
}

static void
mOSAppInit(mctx_t m)
{
    monitor_filter_t ft = {0};
    int s;

    // create a passive monitoring socket & set up its traffic scope
    s = mtcp_socket(m, AF_INET, MOS_SOCK_MONITOR_STREAM, 0);
    ft.stream_syn_filter = "dst net 216.58 and dst port 80";
    mtcp_bind_monitor_filter(m, s, &ft);

    // set up user-defined event for OnFlowStart()
    event_t udeSYN = mtcp_define_event(MOS_ON_PKT_IN, IsSYN, NULL);
    // register event handlers
    mtcp_register_callback(m, s, udeSYN, MOS_HK_SND, OnFlowStart);
    mtcp_register_callback(m, s, MOS_ON_CONN_END, MOS_HK_SND, OnFlowEnd);
    mtcp_register_callback(m, s, MOS_ON_PKT_IN, MOS_HK_SND, processHTTP);
}

Figure 2. Initialization code of a typical mOS application (error handling is omitted).

In the code above, mOSAppInit() first creates a passive monitoring socket (we introduce a new socket type for this purpose called MOS_SOCK_MONITOR_STREAM). We can restrict the kind of traffic the monitor can scan by using the mtcp_bind_monitor_filter() function. We register three events we want to monitor: one at the start of the connection (on SYN request), another for packet arrival (the callback handler is not shown) and the last on connection termination. In the example, we use two built-in and one user-defined mOS events for this purpose. A developer can register for these events either on the sender or receiver end of the stack (see next section for details). The registered event triggers OnFlowStart() and OnFlowEnd() functions for connection initiation and termination respectively. After the initialization of the application, mOS calls the callback handlers for those flows that trigger the registered events. For OnFlowStart() example function, the mOS core passes the active monitoring socket that is unique for each connection, a side variable (to indicate whether it is a client or a server) and the registered event id.

4. mOS Networking Model

A typical mOS (per-core) runtime instance is illustrated in Figure 3. The mOS core can be seen as a coupling of two mTCP stacks that simulate the states of both TCP endpoints. An ingress packet first runs through the sender-side1 stack. The packet triggers an update to its TCP state and records all relevant events.

Figure 3. mOS Runtime flow

Next, the mOS core triggers callback handlers (MOS_HK_SND hook) if a mOS application has already registered for these events. It then checks whether the packet is retransmitted. Finally, this process is repeated with the receiver-side stack (MOS_HK_RCV hook).

5. Evaluation

Our experience with porting some middlebox applications with the mOS networking stack has been positive.

  1. Code reduction: We ported Abacus, a cellular data accounting flow monitoring system that detects "free-riding" attacks. The original Abacus version relies on a custom flow management module that spans 4,639 lines of code. When we port Abacus to mOS API, the size of the program is reduced to only 561 lines of code (88% code reduction).

  2. Performance: We also ported Snort3, a multi-threaded signature-based software NIDS. By using mOS event-based API, we can replace Snort's stream5 flow management and http-inspect (HTTP preprocessor) modules. Our results show that we achieve analyzing throughputs similar to the Snort3-DPDK setup.

6. Publications

7. Source code

Checkout the latest release of mOS at our github.

Our release contains the source code of mOS library, sample applications, and a detailed tutorial on how to program stateful middleboxes using mOS API. You can refer to our detailed documentation on how to build and install new mOS applications.

8. People

Students: Muhammad Asim Jamshed, YoungGyoun Moon, and Donghwi Kim
Faculty: Dongsu Han, and KyoungSoo Park
We are collectively reached by our mailing list: mtcp-user at list.ndsl.kaist.edu .

9. Press Coverage

10. Acknowledgment

This project is supported by ICT Research and Development Program of MSIP/IITP, Korea , under Grant B0101-16-1368, [Development of an NFV-inspired networked switch and an operating system for multi-middlebox services].

1. Please note that both client and server can act as receivers and senders (and vice versa) depending on the direction of the flow.
Last modified: Thur, March 30, 2017
Networked and Distributed Computing Systems Lab & Intelligent Network Architecture and Distributed Systems Research Lab