How to find out reason that cause stats_tcp_pkt_on_wrong_thread count?

Hi,

My IDS has high stats_tcp_pkt_on_wrong_thread(up to 8M).
How to find out reason?

===Environment===
IDS : SELKS 6 (with Suricata 6 rc1)
VM : Proxmox 6.2-11
CPU: 2 sockets, 4 cores (with NUMA)
two NICs, connect to Linux bridge as VirtIO mode
Server : Dell PowerEdge R640 (with Intel Ethernet 10G 4P X520/I350 rNDC)

===Configure===
threading:
set-cpu-affinity: yes
cpu-affinity:
- management-cpu-set:
cpu: [ “0”,“4” ]
mode: “exclusive”
- worker-cpu-set:
cpu: [ “1-3”,“5-7” ]
mode: “exclusive”
prio:
default: “high”

af-packet:

  • interface: ens19
    threads: 3
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    ring-size: 800000
    block-size: 32768
    bpf-filter: (not host 10.0.110.66) or (not (host 10.0.70.77 and icmp))
  • interface: ens20
    threads: 3
    cluster-id: 100
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    ring-size: 800000
    block-size: 32768
    bpf-filter: (not host 10.0.110.66) or (not (host 10.0.70.77 and icmp))

===ethtool of ens19 and ens20===
rx-checksumming: on [fixed]
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

We have been tracking the issue here for a bit now - https://redmine.openinfosecfoundation.org/issues/2725
It seems it is largely related to type of traffic. NIC and Kernel version settings are rather important though.
Please feel free to update the ticket with extra info of our set up if you can.

Just to confirm - what NIC/Kernel are you running ?

Hi Peter

OS: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.19.0-10-amd64
Ethernet controller: Red Hat, Inc Virtio network device

What is the output of

ethtool -l interface