AF_Packet upload performance problems when running in IPS mode and using a linux bridge interface

Help
1

Hi,

When running Suricata in AF Packet IPS mode (mirroring between two interfaces), if one of the interfaces is a linux soft bridge interface (br0), I am seeing pretty low performance on only one direction, bridge to physical WAN port, around 1-2Mbit/sec.
If I replace the suricata.yaml config to address a physical interface instead of the bridge, everything goes back to normal.
In other words:
eth0 → br0 is always fast
br0 → eth0 is abysmally slow with 30-50% packet loss at times even with slow streams like SSH traffic.

Replacing br0 with eth1 (one of the members of br0), makes the problem go away, but then I cannot address the other members in the bridge.
Is this a known issue with AF Packet mode or is there some specific setting to make it behave with the same performance curve when using a bridge interface?
I don’t see this issue in NFQ mode, but obviously that works through IPTables whereas AF Packet is mirroring interfaces itself.

Below is the related section in my config:

af-packet:
  - interface: eth0
    threads: auto
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes
    rollover: yes
    use-mmap: yes
    copy-mode: ips
    copy-iface: br0

  - interface: br0
    threads: auto
    cluster-id: 98
    cluster-type: cluster_flow
    defrag: yes
    rollover: yes
    use-mmap: yes
    copy-mode: ips
    copy-iface: eth0
2

Probably unrelated, but the rollover option is considered bad. It breaks the flow balancing over the threads.

3

Can you post a htop screenshot while suricata is dropping packets?

4

@syoc Htop/CPU utilization is close to 0%
@vjulien Disabling rollover did not change anything unfortunately.

I did notice that tcp.pkt_on_wrong_thread counters are pretty high compared to total packets, although changing thread counts, switching to different workers mode (workes vs autofp), or changing cluster_config did not help either. Full stats.log below:

Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 41349
decoder.pkts                               | Total                     | 41349
decoder.bytes                              | Total                     | 38518042
decoder.invalid                            | Total                     | 3625
decoder.ipv4                               | Total                     | 41299
decoder.ipv6                               | Total                     | 5
decoder.ethernet                           | Total                     | 41349
decoder.tcp                                | Total                     | 37645
decoder.udp                                | Total                     | 32
decoder.avg_pkt_size                       | Total                     | 931
decoder.max_pkt_size                       | Total                     | 1514
flow.tcp                                   | Total                     | 3
flow.udp                                   | Total                     | 9
decoder.ipv4.trunc_pkt                     | Total                     | 3625
decoder.ipv4.opt_pad_required              | Total                     | 1
tcp.pkt_on_wrong_thread                    | Total                     | 15923
app_layer.flow.failed_udp                  | Total                     | 9
flow_mgr.new_pruned                        | Total                     | 8
flow.spare                                 | Total                     | 10000
flow_mgr.rows_checked                      | Total                     | 65536
flow_mgr.rows_skipped                      | Total                     | 65536
tcp.memuse                                 | Total                     | 4587520
tcp.reassembly_memuse                      | Total                     | 786432
flow.memuse                                | Total                     | 7235520