When running Suricata in AF Packet IPS mode (mirroring between two interfaces), if one of the interfaces is a linux soft bridge interface (br0), I am seeing pretty low performance on only one direction, bridge to physical WAN port, around 1-2Mbit/sec.
If I replace the suricata.yaml config to address a physical interface instead of the bridge, everything goes back to normal.
In other words:
eth0 → br0 is always fast
br0 → eth0 is abysmally slow with 30-50% packet loss at times even with slow streams like SSH traffic.
Replacing br0 with eth1 (one of the members of br0), makes the problem go away, but then I cannot address the other members in the bridge.
Is this a known issue with AF Packet mode or is there some specific setting to make it behave with the same performance curve when using a bridge interface?
I don’t see this issue in NFQ mode, but obviously that works through IPTables whereas AF Packet is mirroring interfaces itself.
Below is the related section in my config:
- interface: eth0
- interface: br0
Probably unrelated, but the rollover option is considered bad. It breaks the flow balancing over the threads.
Can you post a htop screenshot while suricata is dropping packets?
@syoc Htop/CPU utilization is close to 0%
@vjulien Disabling rollover did not change anything unfortunately.
I did notice that tcp.pkt_on_wrong_thread counters are pretty high compared to total packets, although changing thread counts, switching to different workers mode (workes vs autofp), or changing cluster_config did not help either. Full stats.log below:
Counter | TM Name | Value
capture.kernel_packets | Total | 41349
decoder.pkts | Total | 41349
decoder.bytes | Total | 38518042
decoder.invalid | Total | 3625
decoder.ipv4 | Total | 41299
decoder.ipv6 | Total | 5
decoder.ethernet | Total | 41349
decoder.tcp | Total | 37645
decoder.udp | Total | 32
decoder.avg_pkt_size | Total | 931
decoder.max_pkt_size | Total | 1514
flow.tcp | Total | 3
flow.udp | Total | 9
decoder.ipv4.trunc_pkt | Total | 3625
decoder.ipv4.opt_pad_required | Total | 1
tcp.pkt_on_wrong_thread | Total | 15923
app_layer.flow.failed_udp | Total | 9
flow_mgr.new_pruned | Total | 8
flow.spare | Total | 10000
flow_mgr.rows_checked | Total | 65536
flow_mgr.rows_skipped | Total | 65536
tcp.memuse | Total | 4587520
tcp.reassembly_memuse | Total | 786432
flow.memuse | Total | 7235520