Myricom and OISF just announced that Myricom joined to OISF consortium to support the development of Suricata. The good folks at Myricom already sent me one of their cards earlier. In this post I’ll describe how you can use these cards already, even though Suricata doesn’t have native Myricom support yet. So in this guide I’ll describe using the Myricom libpcap support.

Getting started

I’m going to assume you installed the card properly, installed the Sniffer driver and made sure that all works. Make sure that in your dmesg you see that the card is in sniffer mode:

[ 2102.860241] myri_snf INFO: eth4: Link0 is UP
[ 2101.341965] myri_snf INFO: eth5: Link0 is UP

I have installed the Myricom runtime and libraries in /opt/snf

Compile Suricata against Myricom’s libpcap:

./configure --with-libpcap-includes=/opt/snf/include/ --with-libpcap-libraries=/opt/snf/lib/ --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make
sudo make install

Next, configure the amount of ringbuffers. I’m going to work with 8 here, as my quad core + hyper threading has 8 logical CPU’s.

pcap:
  - interface: eth5
    threads: 8
    buffer-size: 512kb
    checksum-checks: no

The 8 threads setting makes Suricata create 8 reader threads for eth5. The Myricom driver makes sure each of those is attached to it’s own ringbuffer.

Then start Suricata as follows:
SNF_NUM_RINGS=8 SNF_FLAGS=0x1 suricata -c suricata.yaml -i eth5 --runmode=workers

If you want 16 ringbuffers, update the “threads” variable in your yaml to 16 and start Suricata:
SNF_NUM_RINGS=16 SNF_FLAGS=0x1 suricata -c suricata.yaml -i eth5 --runmode=workers

It looks like you can use any number of ringbuffers, so not limited to a power of 2 for example.

Example with CPU affinity

You can also use Suricata’s built in CPU affinity settings to assign a worker to a cpu/core. In this example I’ll create 7 worker threads that will each run on their own logical CPU. The remaining CPU can then be used by the management threads, most importantly the flow manager.

max-pending-packets: 8192

detect-engine:
  - sgh-mpm-context: full

mpm-algo: ac-bs

threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
      cpu: [ "0" ]
    - detect-cpu-set:
      cpu: [ "1-7" ]
      mode: "exclusive"
      prio:
        default: "high"

pcap:
  - interface: eth5
    buffer-size: 512kb
    threads: 7
    checksum-checks: no

Then start Suricata with:
SNF_NUM_RINGS=7 SNF_FLAGS=0x1 suricata -c suricata.yaml -i eth5 --runmode=workers

This configuration will reserve cpu0 for the management threads and will assign a worker thread (and thus a ringbuffer) to cpu1 to cpu7. Note that I added a few more performance tricks to it. This config with 7 cpu pinned threads appears to be a little faster than the case where CPU affinity is not used.

Myricom has a nice traffic replay tool as well. This replays a pcap at 1Gbps:
snf_replay -r 1.0 -i 10 -p1 /path/to/pcap

Final remarks

The Myricom card already works nicely with Suricata. Because of the way their libpcap code works, we can already use the ringbuffers feature of the card. Myricom does also offer a native API. Later this year, together with Myricom, we’ll be looking into adding support for it.