Running Monitor Applications in Inline Mode =============================================== This page describes how one can set up a mOS middlebox in inline mode and how a developer can control live traffic that goes through the node. Please note that the monitor application in this mode is capable of dropping (e.g., if used as a firewall or an IDS) or manipulating traffic (e.g., NAT) going through it. Environment ----------------------- This section assumes an environment configured as described below. .. figure:: images/midstat_inline.* :align: center :scale: 75 % `where:` * Data Network: Any Ethernet network including end hosts that sends and receives traffic via Host 1. * Host 1: A host which is placed in between two networks so that it can monitor, drop, or manipulate the traffic between them. This host runs :doc:`../samples/midstat`, which runs on mOS networking stack. It has two Ethernet interfaces named ``dpdk0`` and ``dpdk1``, which are connected to each Data Network. Configuration Steps ----------------------- Compile and build mOS library and applications ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Follow the directions in :doc:`../walkthrough/03_setup` to set up the mOS net library and the environment. * In this case, the IP addresses that are assigned to ``dpdk0`` and ``dpdk1`` do not matter. You are only required to configure the network (`i.e.`, just type ``ifconfig dpdk0 0.0.0.0 up``; ``ifconfig dpdk1 0.0.0.0 up``) so that Host 1 can monitor the traffic you want. The mOS core's dpdk driver always intializes the network interfaces in promiscuous mode. Refer to :ref:`sample_app`, and make sure that you successfully compile the mOS application. * In Host 1, please compile `midstat`. .. code-block:: console $ cd samples/midstat $ make Setup mOS application configurations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before running the ``midstat`` application, you need to the configuration the application. * In Host 1, please configure ``mos.conf`` as below. We assume that Host 1 uses CPU core 0~7 for running mOS ``midstat``. Please make sure that you set ``forward = 1`` to allow the ``midstat`` application to forward the traffic if the ruleset permits. Also set the ``nic_forward_table`` appropriately. .. code-block:: console ####################### # MOS-RELATED OPTIONS # ####################### mos { forward = 1 ####################### ##### I/O OPTIONS ##### ####################### # number of memory channels per socket [mandatory for DPDK] nb_mem_channels = 4 # devices used for MOS applications [mandatory] netdev { dpdk0 0x00FF dpdk1 0x00FF } ####################### ### LOGGING OPTIONS ### ####################### # NICs to print network statistics per second # if enabled, mTCP will print xx Gbps and xx pps for RX and TX stat_print = dpdk0 dpdk1 # A directory contains MOS system log files mos_log = logs/ # dpdk0 and dpdk1 will forward traffic in either direction nic_forward_table { dpdk0 dpdk1 } } Run midstat ------------------------- In Host 1, run ``midstat`` with sudo privileges. You would see TCP states of ongoing flows per second. .. code-block:: console $ sudo ./midstat Troubleshooting ----------------------- If your mOS program crashes, search through the printed log messages to check if you missed any of the following requirements: * You should load only the dpdk-registered interfaces (after loading the ``igb_uio`` driver) and configure hugepages as described in :doc:`../walkthrough/03_setup`. * You should create a log folder specified in ``mos_log`` parameter in ``mos.conf`` (``logs/`` in our example) within the same directory that contains the mOS application binary. * Please check if your system is running out of memory by looking at the memory footprint (OOM errors). If you have memory deficiency issues, please reduce the number of socket buffers in ``mos.conf``. If the traffic fails to pass through Host 1, check the following issues: * Check whether you achieve connectivity by reverting back to kernel network driver for the interfaces (see option 5 in :ref:`kernel_module`). You can try testing your setup as a network bridge with default Linux kernel network drivers. .. Inline Mode for Microbenchmarks/Debugging .. --------------------------------------------- .. In this section, we show how one can set up a mOS middlebox .. within a LAN in an inline mode but with a different configuration. .. figure: images/test_setup_white.* .. :align: center .. :scale: 50 % .. In this example, we place the middlebox between two endpoints so .. that it can monitor traffic between them using only **one network .. interface**. This setup requires at least three machines, two for .. traffic generation (A and B) and one for middlebox (M). End-host .. A and B do not have to run the networking applications on the .. mOS stack. .. * A, B: End-host machines .. * M: mOS middlebox between A and B .. * NIF(x): Network interface (e.g. dpdk0, dpdk1, ...) of a machine 'x' .. * MAC(x): MAC address of network interface 'x' .. * IP(x): IP address of network interface 'x' .. Static MAC address table setup .. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. We need to carefully set up static ARP table entries to make all traffic .. between A and B travel via M. You can use 'arp' command to set a .. static ARP table entry of A, B and M. mOS will read Linux kernel's ARP .. entry in its initialization phase. .. For A: .. code-block: console $ arp -s IP(NIF(B)) MAC(NIF(M)) .. For B: .. code-block: console .. $ arp -s IP(NIF(A)) MAC(NIF(M)) .. For M: .. code-block: console $ arp -s IP(NIF(A)) MAC(NIF(A)) $ arp -s IP(NIF(B)) MAC(NIF(B)) .. Switch's forwarding table setup .. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. ======= ================= .. Port MAC address .. 1 NIF(A) .. 2 NIF(M) .. 3 NIF(B) .. ======= ================= .. Compile and build mOS library and application (in Host 1) .. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. Please refer to :doc:`../walkthrough/03_setup` and :doc:`../walkthrough/04_run`. .. Run 'midstat' (in Host 1) .. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block: console .. $ sudo ./midstat .. warning:: Please note that the mOS stack currently can only run on a CPU processor with 16 CPU cores or less. This restriction will be removed in future versions of the stack. For now, we suggest that the user disables all CPU core IDs > 16 using the ``sysfs`` interface. .. code-block:: console $ cd /sys/devices/system/cpu $ for i in {16..$MAX_CPU} do cd $i echo 0 > online cd .. done Testing 'midstat' ------------------------------ You can pick two nodes on either side of Host 1 that can be used as TCP server and client applications. A typical server application can be a web server (see http://nginx.org or http://www.apache.org for further information) or simple evaluation tool such as nuttcp_ (server mode). A typical client application can be a web client (see https://www.gnu.org/software/wget/) or nuttcp_ (client mode). You can test basic connectivity of your setup by first testing whether ping messages are propogated across the network. .. _Sample Applications User Guide: ../samples/index.html .. _nuttcp: http://www.nuttcp.net/Welcome%20Page.html