Network Address Translation (NAT)

Introduction

The nat program is a sample Network Address Translation (NAT) program which maps a single IP address to multiple network addresses. This application can be used for sharing one global IP address of a NAT gateway for an entire private network.

The following code block shows an example run of a nat program. The user provides a global IP address to be shared using -i parameter. The -c parameter directs the mOS core to run the application in num_cores number of cores.

nat -i 200.200.200.200 -c <num_cores>

Code Walkthrough

The following sections provide an explanation of the main components of the nat code. All mOS net library functions used in the sample code are prefixed with mtcp_ and are explained in detail in the Programmer’s Guide - mOS Programming API. Note that we omit the error handling logic from the code snippets for brevity.

(1) The main() Function

The main() function performs the initialization and calls the execution threads for each CPU core.

The first task is to initialize mOS thread based on the mOS configuration file. mos_conf_file is the file path to the mos.conf file which will be provided to mtcp_init().

/* parse mos configuration file */
ret = mtcp_init(mos_conf_file);

In case additional changes are required in mOS configuration (e.g., updating the number of cores), we can use mtcp_getconf() to first retrieve the config. It then calls mtcp_setconf() function to update the configuration with new settings. In the case of nat, we update the num_cores variable that is passed by the user as a command-line argument.

/* set the core limit */
mtcp_getconf(&mcfg);
mcfg.num_cores = g_core_limit;
mtcp_setconf(&mcfg);

Next, it allocates the data structure for storing port numbers. g_free_addrs is a queue that consists of free addresses, which are not yet allocated. Since g_free_addrs is a global queue shared among multiple CPU core threads, it should be protected by the global mutex lock named g_addrlock. The nat program does not use the reserved port number space (from 0 to 1024).

/* Initialize global data structure */
pthread_mutex_init(&g_addrlock, NULL);
TAILQ_INIT(&g_free_addrs);
for (i = 1025; i < 65535; i++) {
struct port *p = malloc(sizeof(struct port));

p->port = htons(i);
TAILQ_INSERT_TAIL(&g_free_addrs, p, link);

Afterwards, the main function creates mtcp context for each core and launches the per-core thread starting from the function init_monitor(). init_monitor() function will be discussed in the following subsection.

for (i = 0; i < g_max_cores; i++) {
        /* Run mOS for each CPU core */
        if (!(g_mctx[i] = mtcp_create_context(i))) {
                ...
        }

        /* init monitor */
        init_monitor(mctx_list[i]);
}

(2) Per-thread Initialization Function

The init_monitor() function is the core functional part of the mOS thread initialization. In order to monitor the ongoing TCP flows, this application creates a MOS_SOCK_MONITOR_STREAM socket as follows.

/* create socket */
lsock = mtcp_socket(ctx->mctx, AF_INET, MOS_SOCK_MONITOR_STREAM, 0);

Using the MOS_SOCK_MONITOR_STREAM socket named lsock, nat registers the callback functions for those TCP events that it is interested in. First, translate_addr() function is called whenever a new packet arrives. Second, release_port() function is called whenever the connection finishes. The translate_addr() and release_port() functions will be described in further detail in the later subsections.

mtcp_register_callback(mctx, lsock, MOS_ON_PKT_IN, MOS_HK_SND, translate_addr);
mtcp_register_callback(mctx, lsock, MOS_ON_CONN_END, MOS_HK_SND, release_port);

(3) The translate_addr() Function

The translate_addr() function is used for translating the network addresses in a packet-by-packet manner.

First, it tries to call mtcp_get_uctx() to check whether the address of the flow has already been translated . If it is a first packet from the flow (meaning that the private user context uctx is NULL), it will try to assign a new port number.

if (!(w = mtcp_get_uctx(mctx, sock)))
        assign_port(mctx, sock);

When it assigns a port using the assign_port() function, it will first retrieve the network address of both sides, find the appropriate port number, and pull the port number from the free port number list (to use it). In order to determine the appropriate port number translation, it uses GetRSSCPUCore() function provided by mOS, so that the translated flow maps to the same CPU core as before the translation.

mtcp_getpeername(mctx, sock, (struct sockaddr *)&addr, &len, MOS_SIDE_CLI);

/* hold the mutex lock g_addrlock before accessing the g_free_addrs */
pthread_mutex_lock(&g_addrlock);

/* find for an appropriate port number to be mapped (to meet core affinity) */
TAILQ_FOREACH(w, &g_free_addrs, link)
if (GetRSSCPUCore(g_NATIP, addr[MOS_SIDE_SVR].sin_addr.s_addr,
        w->port, addr[MOS_SIDE_SVR].sin_port, g_core_limit) == mctx->cpu)
        break;
TAILQ_REMOVE(&g_free_addrs, w, link);

/* release the lock */
pthread_mutex_unlock(&g_addrlock);

(4) The release_port() Function

When the connection finishes, the release_port() callback function will be triggered. The role of release_port() function is simply to insert the port number back in to the free port number list (g_free_addrs), and set the port number allocation metadata to NULL (by passing NULL to mtcp_set_uctx()) to release the port.

if (!(w = mtcp_get_uctx(mctx, sock)))
        return;

/* assign a port number */
pthread_mutex_lock(&g_addrlock);
TAILQ_INSERT_TAIL(&g_free_addrs, w, link);
mtcp_set_uctx(mctx, sock, NULL);
pthread_mutex_unlock(&g_addrlock);

Important

Due to the scalability limitations of our nat application, please restrict the maximum concurrent connections limit to at most 2000 flows at any given time.