AF_Packet Mode on Bonded Interfaces

Help
1

Hey Team,

I currently have a Centos 7 box running kernel 3.10.0-1127.el7.x86_64. I have the box inline underneath a firewall and before a switch so traffic flows internet–>firewall–>suricata–>switch. And I am trying to take advantage of the AF_Packet mode.

Unfortunately the firewall sitting above Suricata only has 1GbE interfaces. To increase throughput, these interfaces are bonded together via a LACP port channel. One port channel serves the inside (internal hosts) vlan and the other serves a dmz vlan. On the Centos7 box that is running Suricata I have bonded the proper interfaces together and setup the appropriate port channels. The Centos7 box is able to successfully bond with the firewall inside and dmz port channel and the switch inside and dmz port channel. So in total I have four port channels, 2 going from Centos7 to firewall, and 2 going from Centos7 to switch. Each port channel has multiple interfaces that are a part of it. This all works well.

My thought is to run Suricata in AF_Packet mode to bridge the bonds together. I will detail out the bond names below:

bond_firewall (serves inside vlan with 2 1GbE interfaces)

bond_firewall2 (serves dmz vlan with 2 1GbE interfaces)

bond_switch (serves inside vlan with 4 10GbE interfaces)

bond_switch2 (serves dmz vlan with 4 10GbE interfaces)

My suricata config is below:

max-pending-packets: 1024

runmode: workers

af-packet:

- interface: bond_firewall

threads: auto

defrag: yes

cluster-type: cluster_flow

cluster-id: 99

ring-size: 2000

copy-mode: ips

copy-iface: bond_switch

#buffer-size: 6453555

use-mmap: yes

tpacket-v3: no

#rollover: yes

- interface: bond_switch

    threads: auto

    defrag: yes

    cluster-type: cluster_flow

    cluster-id: 98

    ring-size: 2000

    copy-mode: ips

    copy-iface: bond_firewall

    #buffer-size: 6453555

    use-mmap: yes

    tpacket-v3: no

    #rollover: yes

- interface: bond_firewall2

    threads: auto

    defrag: yes

    cluster-type: cluster_flow

    cluster-id: 97

    ring-size: 2000

    copy-mode: ips

    copy-iface: bond_switch2

    #buffer-size: 6453555

    use-mmap: yes

    tpacket-v3: no

    #rollover: yes

- interface: bond_switch2

    threads: auto

    defrag: yes

    cluster-type: cluster_flow

    cluster-id: 96

    ring-size: 2000

    copy-mode: ips

    copy-iface: bond_firewall2

    #buffer-size: 6453555

    use-mmap: yes

    tpacket-v3: no

    #rollover: yes

I then start suricata and it looks to start up ok (see images).

image
image941×183 6.67 KB

image
image945×185 7.48 KB

image
image938×194 6.64 KB

image
image972×201 7.13 KB

image
image1039×450 11.8 KB

However, performance is brutally slow. When downloading a 2.0GB file from the internet on a host sitting below suricata, transfer rate is an average of 12 KB/s.

Just to make sure it wasn’t a layer 1 issue, or OS issue I was having, I removed Suricata and used the Linux bridging kernel module to bridge together the port channels and that worked as expected, up to 10 MB/s for the same file download.

Is Suricata able to bind to and then bridge these port channels? My guess is that Suricata is getting confused by the multiple interfaces that are apart of the bonds. The LACP bond is set to a transmit hash of Layer2+3 so is this a hash that is difficult for suricata to understand when it does its own internal hashing to match a packet to a given flow?

Is there any way for me to accomplish what I am trying to do? I’ve also been reading online that the interfaces need to have the promiscuous mode set to on. Would that serve any benefit?

I really appreciate any insight any of you guys may have as I have been left really scratching my head on this. Have you seen other users achieve this in the past but maybe through different options?

Thanks so much!

Taylor

2

384 threads? How many cores does this system have?

I would suggest to start with a smaller setup first, instead of 4 bond interfaces just 2. Also try no rules for this test as well.

It might be even worth to just test IDS mode on one bond interface first if that part is already working or if there is any issue.

3

Hey Andreas,

Thanks for getting back to me.

The box has 96 cores and prior to this full deployment I had a lab setup where I was able to push up to 14 gbps. However, I did have just 1 interface belonging to each port channel, so that is different than the actuall environment I am trying to run in. Also the latency is only seen on TCP traffic, UDP traffic flows through fine. Maybe a checksum issue? I will try testing with all rules disabled and see if that helps and test the TAP mode.

I had the NICs set to 1 RSS queue but still no luck. I will also try increasing the RSS queue but enabling a symmetric hash. I enabled promiscuous mode but that served me no luck. I think that Suricata is having a difficult time calculating a flow’s hash and tracking packets belonging to that flow. Do you know how Suricata sets this flow? Is it a combo of:

layer 2 + layer 3: ((( source_IP XOR dest_IP ) AND 0xffff) XOR ( source_MAC XOR destination_MAC ))

only layer 2: (source_MAC_address XOR destination_MAC)

layer 3 +4: (( source_port XOR dest_port ) XOR (( source_IP XOR dest_IP ) AND 0xffff)

Or is this set by another algorithm? Im thinking that maybe this flow algorithm could contribute to my problem?

Do you know of anyone else running af_packet mode on bonded interfaces? I wouldn’t think I’m the first to try this type of deployment but I dont see anything similar online.

Thanks for the help!

Taylor

4

So that’s the reason why you end up with such a huge number. Each interface section tries to use all 96 cores and mutiply it by 4 and you end up with 384. I would recommend setting the threads value to 24 or maybe a bit lower like 22 to keep some cores for the OS.

I tried it once but didn’t succeed with setting the same performance settings as with the direct interface access.

And some traffic can be an issue as well. But first proceed with the test without rules and maybe run perf top on Suricata while it’s running and you see drops (make sure debug symbols are installed). That might give a hint for the bottleneck.

5

Hey Andreas,

What is the best way to test without rules? Should I just comment out:

default-rule-path: /var/lib/suricata/rules
rule-files:

  • suricata.rules

And this will load suricata without any rules and not throw any errors?

Thanks,
Taylor

6

If you’re testing from the command line you could add -S /dev/null… -S tells Suricata what rule file to load instead of loading the rule files specified in the configuration, and /dev/null is just empty.