3. Compiling the mOS net library

This chapter describes how to compile and build mOS net library from the mOS source release package: mos_release_${ver} (e.g., mos_release_0.3). We first explain the components of this distribution, and then describe the required steps for compiling it with each I/O library (DPDK or PCAP). Our I/O library of choice is DPDK. However, one can use PCAP I/O for testing/debugging purposes.

3.1. Components of mOS distribution

The mOS networking stack release package contains the following subfolders:

$ git clone https://github.com/ndsl-kaist/mOS-networking-stack.git
$ ls mos_release_0.3/
core/  docs/  drivers/  LICENSE  README.md  samples/  scripts/  setup.sh  util/
The main components of the distribution are:
  • core/ contains the core source code of the mOS networking stack.
  • docs/ contains user guides for developers (Programmer’s Guide and mOS API Specification).
  • drivers/ contains packet I/O libraries and drivers used by the stack.
  • LICENSE shows the license of our release.
  • samples/ contains sample mOS applications (midstat, simple_firewall, nat, epserver, and epwget). Please refer to Sample Applications User Guide for further explanation.
  • scripts/ contains the scripts for automating the installation process.
  • setup.sh is a user script for setting up mOS applications that will be discussed in later subsections.
  • util/ contains utility libraries used by the core networking library and the sample applications.

3.2. Compile and build mOS net library with DPDK

We first describe how to compile and build mOS net library with DPDK. Please make sure that your setup satisfies the conditions specified in System Requirements.

3.2.1. Clone the mOS release to local directory

$ git clone https://github.com/ndsl-kaist/mOS-networking-stack.git

3.2.2. Compile and build mOS net library with DPDK

When you run setup.sh with --compile-dpdk argument, the script first compiles Intel DPDK library, and then builds the mOS networking stack.

$ cd mos_release_0.3/
$ ./setup.sh --compile-dpdk

Once DPDK toolkit is successfully compiled, the script prints out the following message and prompts the user to comple the core stack.

Build complete [x86_64-native-linuxapp-gcc]
================== Installing ./
Installation in ./ complete

Done with DPDK setup. Press enter to start MOS setup ...

On pressing enter, the second half of the compilation process begins. A successful compilation run ends with the following message:

----------------------------------------------------------
Done with MOS library setup
----------------------------------------------------------

3.2.3. Configure runtime environment for mOS applications with DPDK

After your mOS net library compilation and build is complete, you need to set up the environment for running mOS applications with DPDK.

When you execute the setup.sh script with --run-dpdk argument, a menu for setting up the run-time environment is displayed.

$ ./setup.sh --run-dpdk

----------------------------------------------------------
Full setup (from start)
----------------------------------------------------------
[0] Full setup for running mOS with DPDK

----------------------------------------------------------
Step-by-step setup for running mOS with DPDK
----------------------------------------------------------
[1] Setup hugepage mappings
[2] Load and bind Ethernet devices to IGB_UIO module
[3] Bring the interfaces up (DPDK devices)
[4] Create new MOS configuration files for sample apps
[5] Unbind dpdk-registered NICs

[6] Exit script

Option:

Steps [1] ~ [4] are mandatory steps for configuring your network interface with DPDK driver. Alternatively, you can just enter [0] that automatically performs all the steps from [1] ~ [4].

In this section, we explain what each step does in more detail. A portion of the text below is borrowed from http://dpdk.readthedocs.org/.

Step 1. Setup hugepage mappings

The following selection demonstrates the creation of hugepages in a NUMA system. 1024 2 MByte pages are assigned to each node.

Option: 1

Removing currently reserved hugepages
mounting /mnt/huge and removing directory
Input the number of 2MB pages for each node
Example: to have 128MB of hugepages available per node,
enter '64' to reserve 64 * 2MB pages on each node
Number of pages for node0: 1024
Number of pages for node1: 1024
Reserving hugepages
Creating /mnt/huge and mounting as hugetlbfs

Step 2. Load and bind Ethernet devices to IGB_UIO module

The following selection first unloads any existing IGB_UIO module and then reloads and binds Ethernet devices for DPDK (& mOS) use. When an interface is bound to the IGB_UIO module, it also unbinds from its associated kernel driver module.

Option: 2

Unloading any existing DPDK UIO module
Loading DPDK UIO module

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:03:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em1 drv=tg3 unused=igb_uio *Active*
0000:03:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em2 drv=tg3 unused=igb_uio

Other network devices
=====================
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=igb_uio
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=igb_uio

Enter PCI address of device(s) to bind to IGB UIO driver (e.g., "04:00.0 04:00.1").
> 04:00.0
OK

Press enter to continue ...

Please note that you can unbind the interface from DPDK driver and go back to using kernel driver by selecting [5]. This option is useful when you want to test mOS applications with PCAP module.

Option: 5
em1
lo

Network devices using DPDK-compatible driver
============================================
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:03:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em1 drv=tg3 unused=igb_uio *Active*
0000:03:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em2 drv=tg3 unused=igb_uio

Other network devices
=====================
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=igb_uio

Enter PCI address of device to unbind: 04:00.0

Enter name of kernel driver to bind the device to: ixgbe
OK

Press enter to continue ...

Warning

As of release DPDK 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. Therefore, all ports that are to be used by a mOS application with DPDK must be bound to the igb_uio module before the application is run. Any network ports under Linux control will be ignored by the DPDK poll-mode drivers and cannot be used by any Linux application (including system stats like sysstat).

Hint

You cannot unbind those interfaces that are currently online in your system. In order to bind such interfaces, one first needs to bring it offline (e.g. ifconfig <intf> down)).

Step 3. Bring the interfaces up (DPDK devices)

The following selection brings the DPDK network interfaces up, and assign any network address (IP address) to each port.

Option: 3

igb_uio is loaded!

[dpdk0] enter IP address[/mask] (e.g., 10.0.0.9[/24])
> 10.0.0.7/24
sudo /sbin/ifconfig dpdk0 10.0.0.7/24 up

Press enter to continue ...

Note

For Running Monitor Applications in Inline Mode, you can leave the IP address unassigned.

Option: 3

igb_uio is loaded!

[dpdk0] enter IP address[/mask] (e.g., 10.0.0.9[/24])
> 0.0.0.0
sudo /sbin/ifconfig dpdk0 0.0.0.0 up

Press enter to continue ...

Step 4. Create new MOS configuration files for sample apps

This selection creates new mOS startup configuration files (mos.conf) for each application. It automatically reads the machine specs (e.g., number of total CPU cores, available DPDK network interfaces) and records the current configuration. The details on the configuration parameters are explained in detail in Configuration Parameter Tuning section.

Option: 4

...

------------------------------------------------
Done with configuration file setup.
Use the arp command to add static ARP entries
------------------------------------------------

Press enter to continue ...

Hint

You can subsequently run any of [1] ~ [5] afterwards to update the runtime environment.

Caution

The setup.sh script is prepared only for Linux x86_64 distributions. Please note that the user may have to manually compile DPDK library for other OS environments.

Step 5. Test that the installation is successful

You can test whether all 4 steps are successfully executed with four basic checks.

1. DPDK-compatible interfaces are up.

$ ifconfig
dpdk0     Link encap:Ethernet  HWaddr 90:e2:ba:7a:b0:00
          inet addr:10.0.0.7  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::92e2:baff:fe7a:b000/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0 B)  TX bytes:0 (0 B)

dpdk1     Link encap:Ethernet  HWaddr 90:e2:ba:7a:b0:01
          inet addr:10.0.1.7  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::92e2:baff:fe7a:b001/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0 B)  TX bytes:0 (0 B)

2. Makefile has been created in each sample application directory.

$ find samples/ -name 'Makefile'
samples/epserver/Makefile
samples/lighttpd-1.4.32/Makefile
samples/simple_firewall/Makefile
samples/epwget/Makefile
samples/nat/Makefile
samples/midstat/Makefile

3. mOS networking stack startup configuration files have been placed in all sample application directories.

$ find samples/ -name 'mos.conf'
samples/epserver/config/mos.conf
samples/lighttpd-1.4.32/config/mos.conf
samples/simple_firewall/config/mos.conf
samples/netstat/config/mos.conf
samples/epwget/config/mos.conf
samples/common/config/mos.conf
samples/nat/config/mos.conf
samples/midstat/config/mos.conf

4. mOS networking library is successfully created.

$ ls -lh core/lib/libmtcp.a
-rw-rw-r-- 1 wheel wheel 3.9M May 11 21:29 core/lib/libmtcp.a

3.3. Compile and build mOS net library with PCAP

We first describe how to compile and build mOS net library with PCAP. The first few steps are quite similar to how we prepare the DPDK version of the library. Once again we suggest you to make sure that your setup envrionment satisfies the conditions specified in System Requirements.

3.3.1. Clone the mOS release to local directory

Repeat this step if you have not already cloned the mOS net package on your machine.

$ git clone https://github.com/ndsl-kaist/mOS-networking-stack.git

Note

Please remember to delete outdated mOS configuration files (mos.conf) in case you previously set up the mOS networking stack with DPDK settings. The most convenient way to wipe out the files is by executing the following command:

$ ./setup.sh --cleanup

3.3.2. Compile and build mOS net library with PCAP

When you run the setup.sh script with --compile-pcap argument, it directly compiles the mOS net library.

$ cd mos_release_0.3/
$ ./setup.sh --compile-pcap

3.3.3. Configure runtime environment for mOS applications with PCAP

Once the mOS net library compiles, you can set up the runtime environment for executing mOS applications with PCAP.

$ sudo ./setup.sh --run-pcap

During the execution of the script, the setup.sh script scans all Ethernet ports and prompts the user to choose the network interface he/she wants to register with the mOS net library.

Warning

  • PCAP version of monitoring applications should always be executed with a single core. PCAP I/O does not provide a built-in support for splitting traffic flows symmetrically across cores.
  • Please remember to bring the interfaces up (ifconfig <iface> up).

Note

In case you want to detach dpdk-registered NICs for later PCAP use, we suggest using ./setup.sh --run-dpdk. Select option 5 to unbind NICs from DPDK’s igb_uio driver. Also, make sure that you have removed old mos.conf files from monitoring applications’ directories.