High speed I O

To achieve high throughput and low latency, one should use either Netmap or DPDK instead of the OS networking stacks. This also includes in-Kernel Click, which is far slower than userlevel Click when using one of those frameworks.

Netmap

Be sure to install Netmap on your system then configure with :

./configure --with-netmap --enable-netmap-pool --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose --enable-select=poll CFLAGS="-O3" CXXFLAGS="-std=gnu++11 -O3"  --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --enable-local --enable-zerocopy --enable-batch

to get the highest performance.

An example configuration is:

FromNetmapDevice(netmap:eth0) -> CheckIPHeader(OFFSET 14) -> ToNetmapDevice(netmap:eth1)

To run click, do:

sudo bin/click -j 4 -a /path/to/config/file.click

Where 4 is the number of threads to use. The FromNetmapDevice will share the assigned cores themselves, do not pin the thread.

We noted that Netmap performs better without MQ, or at least with a minimal amount of queues: ethtool -L eth% combined 1 will set the number of Netmap queues to 1. No need to pin the IRQ of the queues as our FastClick implementation will take care of it. Just kill irqbalance.

Also, be sure to read the sections of our paper about full push to make the faster configuration.

The --enable-netmap-pool option allows to use Netmap buffers instead of Click malloc'ed buffers. This enhance performance as there is only one kind of buffer floating into Click. However with this option you need to place at least one From/ToNetmapDevice in your configuration and allocate enough Netmap buffers using NetmapInfo.

DPDK

Partial DPDK support is now reverted into vanilla Click (without support for batching, auto-thread assignment, thread vector, ...).

Setup your DPDK environment. Version starting from 1.6 to the latest are supported. If you're using the very latest version and a problem happens, raise an issue. The following DPDK compilation and installation process will avoid installing DPDK globally, and install it instead in a subfolder. FastClick will be able to find DPDK by using pkg-config and the PKG_CONFIG_PATH environment variable. The dependencies are installed using apt, if you're on a non-debian system, the equivalent packages are easy to find.

sudo apt install libelf-dev build-essential pkg-config zlib1g-dev libnuma-dev python3-pyelftools
git clone git://dpdk.org/dpdk-stable
cd dpdk
git checkout v22.11.2
meson setup build -Dprefix=$(pwd)/install
export DPDK_PATH=$(pwd)/install/
cd build
ninja && ninja install
export PKG_CONFIG_PATH=${DPDK_PATH}/lib/x86_64-linux-gnu/pkgconfig/:$PKG_CONFIG_PATH

If instead you want to install DPDK globally, do not pass -Dprefix to meson, and PKG_CONFIG_PATH won't be needed.

Then configure FastClick with:

./configure --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose CFLAGS="-g -O3" CXXFLAGS="-g -std=gnu++11 -O3" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --enable-dpdk --enable-batch --with-netmap=no --enable-zerocopy --disable-dpdk-pool --disable-dpdk-packet

to get the highest performance.

An example configuration is:

FromDPDKDevice(0) -> CheckIPHeader(OFFSET 14) -> ToDPDKDevice(1)

To run click with DPDK, you can add the usual EAL parameters:

sudo bin/click --dpdk -c 0xf -n 4 -- /path/to/config/file.click

where 4 is the number of memory channels and 0xf the core mask.

DPDK only supports "full push" (run-to-completion) mode.

As for Netmap --enable-dpdk-pool option forces FastClick to use only DPDK buffers instead of Click malloc'ed packet buffers. The --enable-dpdk-packet option uses DPDK packet handling mechanism instead of Click's Packet object. All Packet function will be changed by wrappers around DPDK's rte_pktmbuf functions. However this feature, while reducing memory footprint, do not enhance the performances as Packets objects are recycled in LIFO and stays in cache while every new access to metadata inside the rte_mbuf produce a cache miss. See the PacketMill paper for more details.

For best performance, DPDK needs to have special support for NICs as it needs a userlevel driver to avoid going through the Kernel. If your NIC is not supported by DPDK, one solution is to use a virtual driver that uses an AF_PACKET socket, much like PCAP does:

sudo click --dpdk --vdev=eth_af_packet0,iface=eth0,blocksz=4096,framesz=2048,framecnt=512,qpairs=1,qdisc_bypass=0 -- config.click 

You may also try Click with a virtual TAP interface instead:

sudo click --dpdk -c '0x1' -n 4 --vdev 'net_tap,iface=tap0' -- conf/dpdk/dpdk-bounce.click

AF_XDP

AF_XDP leverages the eXpress DataPath in the Linux Kernel and eBPF to "steal" packets from the drivers like Netmap does. There is no need for a native XDP element in FastClick, as DPDK as an "XDP" driver included.

sudo click --dpdk -c '0x1' -n 4 --vdev 'net_af_xdp,iface=eth0' -- conf/dpdk/dpdk-bounce.click

Check the page AF_XDP PMD driver for more information about the options. AF_XDP will be much faster than the PCAP and AF_PACKET option proposed above if your NIC is not natively compatible with DPDK.