Middlebox Stats (midstat)


netstat (network statistics) is a command-line tool in UNIX-like operating systems which displays the network connection status for the Transmission Control Protocol. It is often used for debugging the network and to determine the number of ongoing TCP connections.

The original netstat focuses on the end-host networking stack; we develop a middlebox version of netstat on top of mOS which we call midstat. midstat shows the live statistics of both sides (client and server side) network connection. Network statistics include the IP addresses, port numbers, and the running TCP states of each side of the ongoing connection.

Code Walkthrough

The following sections provide an explanation of the main components of the mOS midstat code. All mOS library functions used in the sample code are prefixed with mtcp_ and are explained in detail in the Programmer’s Guide - mOS Programming API. Note that we omit the error handling logic in the example code for brevity.

(1) The main() Function

The main() function performs the initialization and calls the execution threads for each CPU core.

The first task is to initialize mOS thread based on the mOS configuration file. fname is the file path to the mos.conf file which will be provided to mtcp_init().

/* parse mos configuration file */
ret = mtcp_init(fname);

In case the mOS configuration needs an update after mtcp_init() call (e.g., for changing the number of cores required for running the mOS application), we can use mtcp_getconf() to retrieve the current config from the mOS core. mtcp_setconf() function can then be used to re-set some of the selected parameters (as shown in the example below).

/* set the core limit */
mcfg.num_cores = g_max_cores;

Afterwards, the main() function will create mtcp context for each core and launch the per-core thread starting from the function InitMonitor(). InitMonitor() function will be discussed in the following section.

for (i = 0; i < g_max_cores; i++) {
        /* Run mOS for each CPU core */
        if (!(g_mctx[i] = mtcp_create_context(i))) {
                fprintf(stderr, "Failed to craete mtcp context.\n");
                return -1;

        /* init monitor */
    InitMonitor(g_mctx[i], ev_new_syn);

(2) Per-thread Initialization Function

The InitMonitor() function is the core functional part of the mOS thread initialization. In order to allow midstat to track all the ongoing connections, it first initializes the memory for the global connection queue per each core.

/* Initialize internal memory structures */

Afterwards, this application creates a MOS_SOCK_MONITOR_STREAM socket to monitor TCP-related events of ongoing TCP flows.

/* create socket */
sock = mtcp_socket(ctx->mctx, AF_INET, MOS_SOCK_MONITOR_STREAM, 0);

Since this program does not perform any payload monitoring, it disables the socket buffer of each side as follows:

mtcp_setsockopt(mctx, sock, SOL_MONSOCKET, MOS_CLIBUF, &optval, sizeof(optval));
mtcp_setsockopt(mctx, sock, SOL_MONSOCKET, MOS_SVRBUF, &optval, sizeof(optval));

The last step of the per-thread initialization is to register callback functions for the TCP events that this program interested in. The RegisterCallbacks function will be described further in the later sections.

RegisterCallbacks(mctx, sock, ev_new_syn);

(3) The RegisterCallbacks() Function

The RegisterCallbacks() function is used for registering callback handler functions for the TCP events in order to monitor the TCP-level behavior.

First, cb_creation() function is called whenever a new connection is created.

mtcp_register_callback(mctx, sock, MOS_ON_CONN_START, MOS_HK_SND, cb_creation);

The role of the cb_creation() function is twofold: (1) to retrieve the address of each side, and (2) to insert the connection metadata into the global connection queue.

static void
cb_creation(mctx_t mctx, int sock, int side, event_t events, filter_arg_t *arg)
        mtcp_getpeername(mctx, c->sock, (void *)c->addrs, &addrslen, MOS_SIDE_CLI);

        /* Insert the structure to the queue */
        TAILQ_INSERT_TAIL(&g_sockq[mctx->cpu], c, link);

Second, cb_st_chg() function is called whenever there is any TCP state change. We note that cb_st_chg() function is registered at both sender- and receiver-side MOS_ON_TCP_STATE_CHANGE events, since the TCP states of each side can be changed independently.

mtcp_register_callback(mctx, sock, MOS_ON_TCP_STATE_CHANGE, MOS_HK_SND, cb_st_chg);
mtcp_register_callback(mctx, sock, MOS_ON_TCP_STATE_CHANGE, MOS_HK_RCV, cb_st_chg);

The role of the cb_st_chg() function is to retrieve the changed TCP state using mtcp_getsockopt() function. The side parameter given to the callback function indicates the side whose TCP state changed.

static void
cb_st_chg(mctx_t mctx, int sock, int side, event_t events, filter_arg_t *arg)
        if (side == MOS_SIDE_CLI) {
                mtcp_getsockopt(mctx, c->sock, SOL_MONSOCKET, MOS_TCP_STATE_CLI,
                (void *)&c->cli_state, &intlen);
        else {
                mtcp_getsockopt(mctx, c->sock, SOL_MONSOCKET, MOS_TCP_STATE_SVR,
                (void *)&c->svr_state, &intlen);

Third, cb_destroy() function is called whenever there is any connection which is about to be closed.

mtcp_register_callback(mctx, sock, MOS_ON_CONN_END, MOS_HK_SND, cb_destroy);

The role of the cb_destroy() function is to find and delete the connection from the global connection queue, and release the allocated memory.

/* Destroy connection structure */
static void
cb_destroy(mctx_t mctx, int sock, int side, event_t events, filter_arg_t *arg)
        struct connection *c;
        if (!(c = find_connection(mctx->cpu, sock)))
        TAILQ_REMOVE(&g_sockq[mctx->cpu], c, link);

The last step of the RegisterCallbacks() function is to register timer callback for printing the network statistics every second. Note that this function should be registered only for CPU core id 0, to prevent duplicate printing.

/* CPU 0 is in charge of printing stats */
if (mctx->cpu == 0 && mtcp_settimer(mctx, sock, &tv_1sec, cb_printstat)