High speed I O

To achieve high throughput and low latency, one should use either Netmap or DPDK instead of the OS networking stacks. This also includes in-Kernel Click, which is far slower than userlevel Click when using one of those frameworks.

DPDK

Note : partial DPDK support is now reverted into vanilla Click. The version in FastClick is more maintained and much more full-featured, as the version in vanilla Click does not support batching, auto-thread assignment, thread vector, ...

Setup your DPDK environment. Version starting from 1.6 to the latest are supported. If you're using the very latest version and a problem happens, raise an issue. The following DPDK compilation and installation process will avoid installing DPDK globally, and install it instead in a subfolder. FastClick will be able to find DPDK by using pkg-config and the PKG_CONFIG_PATH environment variable. The dependencies are installed using apt, if you're on a non-debian system, the equivalent packages are easy to find.

sudo apt install libelf-dev build-essential pkg-config zlib1g-dev libnuma-dev python3-pyelftools ninja-build meson
git clone git://dpdk.org/dpdk-stable
cd dpdk-stable
git checkout v22.11.1
meson setup build -Dprefix=$(pwd)/install
export DPDK_PATH=$(pwd)/install
cd build
ninja && ninja install
export PKG_CONFIG_PATH=${DPDK_PATH}/lib/x86_64-linux-gnu/pkgconfig/
export LD_LIBRARY_PATH=${DPDK_PATH}/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH

If instead you want to install DPDK globally, do not pass -Dprefix to meson, and PKG_CONFIG_PATH won't be needed.

Then configure FastClick with:

./configure --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose CFLAGS="-g -O3" CXXFLAGS="-g -std=gnu++11 -O3" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --enable-dpdk --enable-batch --with-netmap=no --enable-zerocopy --disable-dpdk-pool --disable-dpdk-packet

to get the highest performance.

An example configuration is:

FromDPDKDevice(0) -> CheckIPHeader(OFFSET 14) -> ToDPDKDevice(1)

DPDK uses hugepages that must be configured to launch a DPDK application. See the DPDK documentation if hugepages are not enabled. Alternatively, below after --dpdk you might add "--no-huge -m 1024" to avoid using hugepages. That will however decrease performance.

To run click with DPDK, you can add the usual DPDK EAL parameters:

sudo bin/click --dpdk -c 0xf -n 4 -- /path/to/config/file.click

where 4 is the number of memory channels and 0xf the core mask.

DPDK only supports "full push" (run-to-completion) mode.

As for Netmap --enable-dpdk-pool option forces FastClick to use only DPDK buffers instead of Click malloc'ed packet buffers. The --enable-dpdk-packet option uses DPDK packet handling mechanism instead of Click's Packet object. All Packet function will be changed by wrappers around DPDK's rte_pktmbuf functions. However this feature, while reducing memory footprint, do not enhance the performances as Packets objects are recycled in LIFO and stays in cache while every new access to metadata inside the rte_mbuf produce a cache miss. See the PacketMill paper for more details.

For best performance, DPDK needs to have special support for NICs as it needs a userlevel driver to avoid going through the Kernel. If your NIC is not supported by DPDK, one solution is to use a virtual driver that uses an AF_PACKET socket, much like PCAP does:

sudo click --dpdk --vdev=eth_af_packet0,iface=eth0,blocksz=4096,framesz=2048,framecnt=512,qpairs=1,qdisc_bypass=0 -- config.click 

You may also try Click with a virtual TAP interface instead:

sudo click --dpdk -c '0x1' -n 4 --vdev 'net_tap,iface=tap0' -- conf/dpdk/dpdk-bounce.click

AF_XDP

AF_XDP leverages the eXpress DataPath in the Linux Kernel and eBPF to "steal" packets from the drivers like Netmap does. There is no need for a native XDP element in FastClick, as DPDK as an "XDP" driver included.

sudo click --dpdk -c '0x1' -n 4 --vdev 'net_af_xdp,iface=eth0' -- conf/dpdk/dpdk-bounce.click

Check the page AF_XDP PMD driver for more information about the options. AF_XDP will be much faster than the PCAP and AF_PACKET option proposed above if your NIC is not natively compatible with DPDK.

Netmap

Note: Arguably, DPDK has won against Netmap over time. We therefore recommend using DPDK instead of Netmap as subsequently, we also use DPDK more often and the integration is more featured. Netmap abstracts the interface with the driver, which prevents using advanced features of the NIC. DPDK, and FastClick, have support for NIC-assisted flow-steering for instance, marking packets in hardware, ... which Netmap can't provide.

Be sure to install Netmap on your system then configure with :

./configure --with-netmap --enable-netmap-pool --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose --enable-select=poll CFLAGS="-O3" CXXFLAGS="-std=gnu++11 -O3"  --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --enable-local --enable-zerocopy --enable-batch

to get the highest performance.

An example configuration is:

FromNetmapDevice(netmap:eth0) -> CheckIPHeader(OFFSET 14) -> ToNetmapDevice(netmap:eth1)

To run click, do:

sudo bin/click -j 4 -a /path/to/config/file.click

Where 4 is the number of threads to use. The FromNetmapDevice will share the assigned cores themselves, do not pin the thread.

We noted that Netmap performs better without MQ, or at least with a minimal amount of queues: ethtool -L eth% combined 1 will set the number of Netmap queues to 1. No need to pin the IRQ of the queues as our FastClick implementation will take care of it. Just kill irqbalance.

Also, be sure to read the sections of our paper about full push to make the faster configuration.

The --enable-netmap-pool option allows to use Netmap buffers instead of Click malloc'ed buffers. This enhance performance as there is only one kind of buffer floating into Click. However with this option you need to place at least one From/ToNetmapDevice in your configuration and allocate enough Netmap buffers using NetmapInfo.