3. mOS Event System

The core part of mOS is its event module that facilitates event-driven middlebox processing for monitoring applications. The mOS API encourages modular middlebox programming by converting a complex application into a set of independent <event, event handler> pairs. Our experience in programming applications with mOS API led us to the conclusion that existing middlebox applications’ logic can easily be transformed into set of mOS defined events and their corresponding handlers.

mOS provides two classes of events: (i) built-in and (ii) user-defined events.

3.1. Built-in Events

Currently, mOS provides 8 built-in events that the developer can use.

Built-in event	Description
MOS_ON_PKT_IN	In-flow TCP packet arrival
MOS_ON_CONN_START	New connection initiation
MOS_ON_REXMIT	TCP retransmission
MOS_ON_TCP_STATE_CHANGE	TCP state change
MOS_ON_CONN_END	Connection termination
MOS_ON_CONN_NEW_DATA	New flow payload
MOS_ON_ORPHAN	non-TCP packet
MOS_ON_ERROR	Error report (e.g., receive buffer full)

Note

1. mOS raises MOS_ON_CONN_END event when TCP stream on each side is destroyed.
2. mOS raises MOS_ON_CONN_START event when it observes a TCP SYN packet.

A mOS application can register one callback function per built-in event on each socket. With a registered callback, a user can retrieve attributes of an ongoing connection from the mOS core. These attributes can be packet information (that triggers the callback function), the state information of the corresponding TCP flow (TCP state, received buffer contents/offset) etc. We suggest you to refer to the mOS Programming API for details.

3.2. User-defined Events (UDEs)

Some middlebox developers may find built-in events insufficient for implementing the applications’ logic. For example, an event that gets triggered after detecting an HTTP request does not exist in mOS by default. Such an event would require mOS to parse L7 (HTTP) headers of ingress packets. Although a developer can use built-in events (e.g., MOS_ON_PKT_IN, MOS_ON_CONN_NEW_DATA) to develop her own logic that covers all the checks for such complex events, there is always a tangible risk of the event code (for detecting conditions of middlebox processing) getting strongly coupled within the application logic. This design can lead to overly complex codebase that quickly becomes difficult to maintain over time.

User-Defined Events (UDEs) allow the developers to create their own events (that don’t already exist in the built-in events list). A UDE is defined using a base event and a boolean filter function that specifies the event condition. When the base event is raised, the mOS stack evaluates the filter function and raises the UDE only if the filter returns true.

UDEs bring three advantages to the event-driven middlebox development. First, new types of events can be created in a flexible manner because the filter function can evaluate arbitrary conditions of interest. A good filter function, however, should run fast without producing unnecessary side effects. Second, UDEs provide easy extensibility. One can create new events by extending any existing ones, including another UDE. For example, a developer can define a UDE that detects a Google search query by extending a generic HTTP request UDE. Third, it encourages code reuse. One can share a well-designed set of event definitions as a UDE library, and third party developers can implement their own event handlers. For example, an open-source NIDS can declare all corner cases in flow management as a UDE library while third party can provide custom actions to address each case.

3.3. Event Hook and Flow of Events

Figure 3.1. Flow of Events in mOS stack

Figure 3.1 described the sequence of events in an inline mOS monitor and shows how an ingress packet is processed in the mOS stack. On entry, the flow corresponding to the packet is first identified. In case the packet is TCP SYN, a new flow entry is created. The mOS core then updates the sender side’s TCP stack. Once the stack has been updated, it raises all flow events that are registered by the monitoring application pertaining to the sender side. It then checks whether the TCP packet is retransmitted. Finally it repeats the same process with receiver side stack before the packet is forwarded.

Attention

In the mOS stack, client and server can both be sender and receiver. For example, a SYN packet transmission from the client (to the server) would make client a sender and the server a receiver. On the flip side, a SYN-ACK packet would have the server acting as a sender while a client as the receiver.

We have placed two event hooks from where mOS events can be triggered. A developer can use MOS_HK_SND to capture events from sender’s side while MOS_HK_RCV can be used to receive events from the receiver side.

Important

Both event hooks are only triggered after the respective states have been updated.

Table 3.2 shows which events are available at given hook points in mOS. Please note that MOS_NULL hook point is used for those networking events that are raised without any flow context. MOS_ON_PKT_IN may still use MOS_NULL hook for those packets that don’t have a corresponding flow context (e.g., a TCP packet that arrives without a TCP handshake). MOS_ON_CONN_NEW_DATA is a special event that gets triggered when the mOS core detects that a flow has received reassembled payload in the receive buffer. This event can come out of context since the mOS core uses the DPDK I/O driver to read batch of packets from the NIC queue. The event is only raised once all the packets in the batch have been processed. This helps the mOS core to amortize the cost of calling MOS_ON_CONN_NEW_DATA event handler by triggering the event only once when multiple data packets arrive for the same flow.

Event	Available hooks
MOS_ON_PKT_IN	MOS_NULL, MOS_HK_SND, MOS_HK_RCV
MOS_ON_CONN_START	MOS_HK_SND, MOS_HK_RCV
MOS_ON_REXMIT	MOS_HK_SND, MOS_HK_RCV
MOS_ON_TCP_STATE_CHANGE	MOS_HK_SND, MOS_HK_RCV
MOS_ON_CONN_END	MOS_HK_SND, MOS_HK_RCV
MOS_ON_CONN_NEW_DATA	MOS_NULL
MOS_ON_ORPHAN	MOS_NULL
MOS_ON_ERROR	MOS_NULL

Table 3.2: List of events and their available hook points.

Note

Some events can be registered both on MOS_HK_SND and MOS_HK_RCV hooks. Examples include MOS_ON_PKT_IN, MOS_ON_CONN_START and MOS_ON_TCP_STATE_CHANGE. In order to distinguish these events from each other, all mOS event handlers pass side (client/server) variable as a function argument.

3.3.1. Example Initialization Sequence

A code snippet below shows all the concepts that we have introduced in this section.

enum {MOS_SIDE_CLI = 0, MOS_SIDE_SVR};

/*
 * mOSAppInit():
 * [In]: mctx_t m - Per-thread mOS context
 */
static void
mOSAppInit(mctx_t m)
{
   monitor_filter_t ft = {0};
   int s; event_t hev;

   // create a passive monitoring socket & set up its traffic scope
   s = mtcp_socket(m, AF_INET, MOS_SOCK_MONITOR_STREAM, 0);
   if (s == -1) {
        perror("mtcp_socket() failed: ");
        exit(EXIT_FAILURE);
   }
   ft.stream_syn_filter = "dst net 216.58 and dst port 80";
   if (mtcp_bind_monitor_filter(m, s, &ft) == -1) {
        perror("mtcp_bind_monitor_filter() failed: ");
        exit(EXIT_FAILURE);
   }

   // set up a built-in event handler for MOS_ON_REXMIT
   if (mtcp_register_callback(m, s, MOS_ON_REXMIT, MOS_HK_RCV, OnRexmitPkt) < 0)
        goto register_err;

   // set up a built-in event handler for MOS_ON_PKT_IN
   if (mtcp_register_callback(m, s, MOS_ON_PKT_IN, MOS_HK_RCV, OnFlowPkt) < 0)
        goto register_err;

   // set up a built-in event handler for MOS_ON_TCP_STATE_CHANGE
   if (mtcp_register_callback(m, s, MOS_ON_TCP_STATE_CHANGE,
                                MOS_HK_RCV, OnTcpStateChange) < 0)
        goto register_err;

   // set up a built-in event handler for MOS_ON_CONN_END
   if (mtcp_register_callback(m, s, MOS_ON_CONN_END, MOS_HK_RCV, OnFlowEnd) < 0)
        goto register_err;

   // define a user-defined event that detects an HTTP request
   hev = mtcp_define_event(MOS_ON_CONN_NEW_DATA, IsHTTPRequest);
   if (hev == 0) {
        perror("mtcp_define_event() failed: ");
        exit(EXIT_FAILURE);
   }

   // set up an event handler for hev
   if (mtcp_register_callback(m, s, hev, MOS_HK_RCV, OnHTTPRequest) < 0)
        goto register_err;

register_err:
        perror("mtcp_register_callback() failed: ");
        exit(EXIT_FAILURE);
}

/*
 * OnFlowPkt():
 * [In]: mctx_t m - Per-thread mOS context
 * [In]: int sock - active monitoring socket (represents individual connection)
 * [In]: int side - Server/Client?
 * [In]: event_t event - the event that was triggered
 */

static void // callback for MOS_ON_PKT_IN
OnFlowPkt(mctx_t m, int sock, int side, event_t event, filter_arg_t *arg)
{
   if (side == MOS_SIDE_CLI)
      g_pktcnt[sock]++;
}

static void // callback for MOS_ON_REXMIT
OnRexmitPkt(mctx_t m, int sock, int side, event_t event, filter_arg_t *arg)
{
   g_rexmit_cnt[sock]++;
}

static void // callback for MOS_ON_TCP_STATE_CHANGE
OnTCPStateChange(mctx_t m, int sock, int side, event_t event, filter_arg_t *arg)
{
   if (side == MOS_SIDE_CLI) {
      int state; socklent_t len = sizeof(state);
      if (mtcp_getsockopt(m, sock, SOL_MONSOCKET,
                      MOS_TCP_STATE_CLI, &state, &len) < 0) {
         perror("mtcp_getsockopt() failed: ");
         return;
      }
      if (state == TCP_FIN_WAIT_1)
         g_cli_term[sock]++;
   }
}

static void // callback for MOS_ON_TCP_CONN_END
OnFlowEnd(mctx_t m, int sock, int side, event_t event, filter_arg_t *arg)
{
   if (sock != MOS_SIDE_CLI) return;

   TRACE_LOG("TCP flow (sock=%d) had %d packets, rexmit: %d\n",
             sock, g_pktcnt[sock], g_rexmit_cnt[sock]);
   g_pktcnt[sock] = 0; g_rexmit_cnt[sock] = 0;
   g_total_flows++;
}