Hi,
What is the meaning of the tcp_pkt_wrong_thread?
I am getting a considerable amount of entries in stats.log. How can i lower this value, assuming that is not very good to have a high count.
I am using cluster_flow and af_packet
There could be multiple reasons some related to driver and kernel versions even. Can you share a bit more detail of the setup please? (HW/VM/NIC/OS etc)
Some more info - Support #2725: stream/packet on wrong thread - Suricata - Open Information Security Foundation
The counter basically means that load balancing is not working as expected - aka a thread sees packets from a stream lets say that are supposed to be seen on different thread.
Here goes some info.
Date: 4/16/2021 -- 10:42:52 (uptime: 0d, 11h 03m 24s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
capture.kernel_packets | Total | 645319899
capture.kernel_drops | Total | 1060401
capture.errors | Total | 0
decoder.pkts | Total | 644282079
decoder.bytes | Total | 382864175179
decoder.invalid | Total | 146
decoder.ipv4 | Total | 642078395
decoder.ipv6 | Total | 429620
decoder.ethernet | Total | 644282082
decoder.chdlc | Total | 0
decoder.raw | Total | 0
decoder.null | Total | 0
decoder.sll | Total | 0
decoder.tcp | Total | 416570098
decoder.udp | Total | 215521446
decoder.sctp | Total | 0
decoder.icmpv4 | Total | 9135082
decoder.icmpv6 | Total | 134520
decoder.ppp | Total | 0
decoder.pppoe | Total | 0
decoder.geneve | Total | 0
decoder.gre | Total | 25
decoder.vlan | Total | 403883762
decoder.vlan_qinq | Total | 0
decoder.vxlan | Total | 7
decoder.ieee8021ah | Total | 0
decoder.teredo | Total | 0
decoder.ipv4_in_ipv6 | Total | 0
decoder.ipv6_in_ipv6 | Total | 0
decoder.mpls | Total | 0
decoder.avg_pkt_size | Total | 594
decoder.max_pkt_size | Total | 4133
decoder.max_mac_addrs_src | Total | 0
decoder.max_mac_addrs_dst | Total | 0
decoder.erspan | Total | 0
flow.memcap | Total | 0
flow.tcp | Total | 13118079
flow.udp | Total | 15472232
flow.icmpv4 | Total | 440999
flow.icmpv6 | Total | 7889
flow.tcp_reuse | Total | 263540
flow.get_used | Total | 0
flow.get_used_eval | Total | 0
flow.get_used_eval_reject | Total | 0
flow.get_used_eval_busy | Total | 0
flow.get_used_failed | Total | 0
flow.wrk.spare_sync_avg | Total | 100
flow.wrk.spare_sync | Total | 143750
flow.wrk.spare_sync_incomplete | Total | 0
flow.wrk.spare_sync_empty | Total | 0
defrag.ipv4.fragments | Total | 44988
defrag.ipv4.reassembled | Total | 22272
defrag.ipv4.timeouts | Total | 0
defrag.ipv6.fragments | Total | 0
defrag.ipv6.reassembled | Total | 0
defrag.ipv6.timeouts | Total | 0
defrag.max_frag_hits | Total | 0
decoder.event.ipv4.pkt_too_small | Total | 0
decoder.event.ipv4.hlen_too_small | Total | 0
decoder.event.ipv4.iplen_smaller_than_hlen | Total | 0
decoder.event.ipv4.trunc_pkt | Total | 46
decoder.event.ipv4.opt_invalid | Total | 0
decoder.event.ipv4.opt_invalid_len | Total | 0
decoder.event.ipv4.opt_malformed | Total | 0
decoder.event.ipv4.opt_pad_required | Total | 3748
decoder.event.ipv4.opt_eol_required | Total | 0
decoder.event.ipv4.opt_duplicate | Total | 0
decoder.event.ipv4.opt_unknown | Total | 0
decoder.event.ipv4.wrong_ip_version | Total | 0
decoder.event.ipv4.icmpv6 | Total | 0
decoder.event.icmpv4.pkt_too_small | Total | 0
decoder.event.icmpv4.unknown_type | Total | 0
decoder.event.icmpv4.unknown_code | Total | 30
decoder.event.icmpv4.ipv4_trunc_pkt | Total | 0
decoder.event.icmpv4.ipv4_unknown_ver | Total | 0
decoder.event.icmpv6.unknown_type | Total | 0
decoder.event.icmpv6.unknown_code | Total | 0
decoder.event.icmpv6.pkt_too_small | Total | 0
decoder.event.icmpv6.ipv6_unknown_version | Total | 0
decoder.event.icmpv6.ipv6_trunc_pkt | Total | 0
decoder.event.icmpv6.mld_message_with_invalid_hl | Total | 0
decoder.event.icmpv6.unassigned_type | Total | 0
decoder.event.icmpv6.experimentation_type | Total | 0
decoder.event.ipv6.pkt_too_small | Total | 0
decoder.event.ipv6.trunc_pkt | Total | 0
decoder.event.ipv6.trunc_exthdr | Total | 0
decoder.event.ipv6.exthdr_dupl_fh | Total | 0
decoder.event.ipv6.exthdr_useless_fh | Total | 0
decoder.event.ipv6.exthdr_dupl_rh | Total | 0
decoder.event.ipv6.exthdr_dupl_hh | Total | 0
decoder.event.ipv6.exthdr_dupl_dh | Total | 0
decoder.event.ipv6.exthdr_dupl_ah | Total | 0
decoder.event.ipv6.exthdr_dupl_eh | Total | 0
decoder.event.ipv6.exthdr_invalid_optlen | Total | 0
decoder.event.ipv6.wrong_ip_version | Total | 0
decoder.event.ipv6.exthdr_ah_res_not_null | Total | 0
decoder.event.ipv6.hopopts_unknown_opt | Total | 0
decoder.event.ipv6.hopopts_only_padding | Total | 0
decoder.event.ipv6.dstopts_unknown_opt | Total | 0
decoder.event.ipv6.dstopts_only_padding | Total | 0
decoder.event.ipv6.rh_type_0 | Total | 0
decoder.event.ipv6.zero_len_padn | Total | 4611
decoder.event.ipv6.fh_non_zero_reserved_field | Total | 0
decoder.event.ipv6.data_after_none_header | Total | 0
decoder.event.ipv6.unknown_next_header | Total | 3706
decoder.event.ipv6.icmpv4 | Total | 0
decoder.event.tcp.pkt_too_small | Total | 0
decoder.event.tcp.hlen_too_small | Total | 30
decoder.event.tcp.invalid_optlen | Total | 27
decoder.event.tcp.opt_invalid_len | Total | 42730
decoder.event.tcp.opt_duplicate | Total | 0
decoder.event.udp.pkt_too_small | Total | 0
decoder.event.udp.hlen_too_small | Total | 0
decoder.event.udp.hlen_invalid | Total | 0
decoder.event.sll.pkt_too_small | Total | 0
decoder.event.ethernet.pkt_too_small | Total | 0
decoder.event.ppp.pkt_too_small | Total | 0
decoder.event.ppp.vju_pkt_too_small | Total | 0
decoder.event.ppp.ip4_pkt_too_small | Total | 0
decoder.event.ppp.ip6_pkt_too_small | Total | 0
decoder.event.ppp.wrong_type | Total | 0
decoder.event.ppp.unsup_proto | Total | 0
decoder.event.pppoe.pkt_too_small | Total | 0
decoder.event.pppoe.wrong_code | Total | 0
decoder.event.pppoe.malformed_tags | Total | 0
decoder.event.gre.pkt_too_small | Total | 0
decoder.event.gre.wrong_version | Total | 0
decoder.event.gre.version0_recur | Total | 0
decoder.event.gre.version0_flags | Total | 0
decoder.event.gre.version0_hdr_too_big | Total | 0
decoder.event.gre.version0_malformed_sre_hdr | Total | 0
decoder.event.gre.version1_chksum | Total | 0
decoder.event.gre.version1_route | Total | 0
decoder.event.gre.version1_ssr | Total | 0
decoder.event.gre.version1_recur | Total | 0
decoder.event.gre.version1_flags | Total | 0
decoder.event.gre.version1_no_key | Total | 0
decoder.event.gre.version1_wrong_protocol | Total | 0
decoder.event.gre.version1_malformed_sre_hdr | Total | 0
decoder.event.gre.version1_hdr_too_big | Total | 0
decoder.event.vlan.header_too_small | Total | 0
decoder.event.vlan.unknown_type | Total | 0
decoder.event.vlan.too_many_layers | Total | 0
decoder.event.ieee8021ah.header_too_small | Total | 0
decoder.event.ipraw.invalid_ip_version | Total | 0
decoder.event.ltnull.pkt_too_small | Total | 0
decoder.event.ltnull.unsupported_type | Total | 0
decoder.event.sctp.pkt_too_small | Total | 0
decoder.event.ipv4.frag_pkt_too_large | Total | 0
decoder.event.ipv6.frag_pkt_too_large | Total | 0
decoder.event.ipv4.frag_overlap | Total | 24
decoder.event.ipv6.frag_overlap | Total | 0
decoder.event.ipv4.frag_ignored | Total | 0
decoder.event.ipv6.frag_ignored | Total | 0
decoder.event.ipv6.ipv4_in_ipv6_too_small | Total | 0
decoder.event.ipv6.ipv4_in_ipv6_wrong_version | Total | 0
decoder.event.ipv6.ipv6_in_ipv6_too_small | Total | 0
decoder.event.ipv6.ipv6_in_ipv6_wrong_version | Total | 0
decoder.event.mpls.header_too_small | Total | 0
decoder.event.mpls.pkt_too_small | Total | 0
decoder.event.mpls.bad_label_router_alert | Total | 0
decoder.event.mpls.bad_label_implicit_null | Total | 0
decoder.event.mpls.bad_label_reserved | Total | 0
decoder.event.mpls.unknown_payload_type | Total | 0
decoder.event.vxlan.unknown_payload_type | Total | 7
decoder.event.geneve.unknown_payload_type | Total | 0
decoder.event.erspan.header_too_small | Total | 0
decoder.event.erspan.unsupported_version | Total | 0
decoder.event.erspan.too_many_vlan_layers | Total | 0
decoder.event.dce.pkt_too_small | Total | 0
decoder.event.chdlc.pkt_too_small | Total | 0
decoder.too_many_layers | Total | 0
flow_bypassed.local_pkts | Total | 31139055
flow_bypassed.local_bytes | Total | 30762112625
flow_bypassed.local_capture_pkts | Total | 0
flow_bypassed.local_capture_bytes | Total | 0
flow.wrk.flows_evicted_needs_work | Total | 577083
flow.wrk.flows_evicted_pkt_inject | Total | 1054412
flow.wrk.flows_evicted | Total | 14116171
flow.wrk.flows_injected | Total | 548667
tcp.sessions | Total | 9610759
tcp.ssn_memcap_drop | Total | 0
tcp.pseudo | Total | 8010
tcp.pseudo_failed | Total | 0
tcp.invalid_checksum | Total | 23848
tcp.no_flow | Total | 0
tcp.syn | Total | 10571117
tcp.synack | Total | 3268041
tcp.rst | Total | 2334933
tcp.midstream_pickups | Total | 0
tcp.pkt_on_wrong_thread | Total | 3726861
tcp.segment_memcap_drop | Total | 0
tcp.stream_depth_reached | Total | 1084
tcp.reassembly_gap | Total | 582554
tcp.overlap | Total | 46078
tcp.overlap_diff_data | Total | 0
tcp.insert_data_normal_fail | Total | 0
tcp.insert_data_overlap_fail | Total | 0
tcp.insert_list_fail | Total | 0
detect.alert | Total | 228304
app_layer.flow.http | Total | 28712
app_layer.tx.http | Total | 92269
app_layer.flow.ftp | Total | 2
app_layer.tx.ftp | Total | 4
app_layer.flow.smtp | Total | 6577
app_layer.tx.smtp | Total | 10924
app_layer.flow.tls | Total | 87052
app_layer.tx.tls | Total | 0
app_layer.flow.ssh | Total | 9532
app_layer.tx.ssh | Total | 0
app_layer.flow.imap | Total | 0
app_layer.tx.imap | Total | 0
app_layer.flow.smb | Total | 3105
app_layer.tx.smb | Total | 20517
app_layer.flow.dcerpc_tcp | Total | 6909
app_layer.tx.dcerpc_tcp | Total | 14927
app_layer.flow.dns_tcp | Total | 2043
app_layer.tx.dns_tcp | Total | 7528
app_layer.flow.nfs_tcp | Total | 18
app_layer.tx.nfs_tcp | Total | 2318
app_layer.flow.ntp | Total | 49435
app_layer.tx.ntp | Total | 55354
app_layer.flow.ftp-data | Total | 0
app_layer.tx.ftp-data | Total | 0
app_layer.flow.tftp | Total | 7769
app_layer.tx.tftp | Total | 7576
app_layer.flow.ikev2 | Total | 310
app_layer.tx.ikev2 | Total | 319
app_layer.flow.krb5_tcp | Total | 20162
app_layer.tx.krb5_tcp | Total | 20173
app_layer.flow.dhcp | Total | 16941
app_layer.tx.dhcp | Total | 59049
app_layer.flow.snmp | Total | 1551479
app_layer.tx.snmp | Total | 1935560
app_layer.flow.sip | Total | 256506
app_layer.tx.sip | Total | 259951
app_layer.flow.rfb | Total | 0
app_layer.tx.rfb | Total | 0
app_layer.flow.mqtt | Total | 3
app_layer.tx.mqtt | Total | 22
app_layer.flow.rdp | Total | 2
app_layer.tx.rdp | Total | 9
app_layer.flow.failed_tcp | Total | 144666
app_layer.flow.dcerpc_udp | Total | 22
app_layer.tx.dcerpc_udp | Total | 0
app_layer.flow.dns_udp | Total | 6211609
app_layer.tx.dns_udp | Total | 10403779
app_layer.flow.nfs_udp | Total | 0
app_layer.tx.nfs_udp | Total | 0
app_layer.flow.krb5_udp | Total | 25
app_layer.tx.krb5_udp | Total | 21
app_layer.flow.failed_udp | Total | 7378136
flow.mgr.full_hash_pass | Total | 166
flow.mgr.closed_pruned | Total | 0
flow.mgr.new_pruned | Total | 0
flow.mgr.est_pruned | Total | 0
flow.mgr.bypassed_pruned | Total | 0
flow.spare | Total | 11700
flow.emerg_mode_entered | Total | 0
flow.emerg_mode_over | Total | 0
flow.mgr.rows_maxlen | Total | 30
flow.mgr.flows_checked | Total | 9774132
flow.mgr.flows_notimeout | Total | 4903605
flow.mgr.flows_timeout | Total | 4870527
flow.mgr.flows_timeout_inuse | Total | 3710
flow.mgr.flows_evicted | Total | 14736117
flow.mgr.flows_evicted_needs_work | Total | 548667
flow_bypassed.closed | Total | 0
flow_bypassed.pkts | Total | 0
flow_bypassed.bytes | Total | 0
tcp.memuse | Total | 23209688
tcp.reassembly_memuse | Total | 205404320
http.memuse | Total | 11001334
http.memcap | Total | 0
ftp.memuse | Total | 0
ftp.memcap | Total | 0
app_layer.expectations | Total | 0
file_store.open_files | Total | 0
flow.memuse | Total | 68031424
root@suricata:~# suricata --dump-config | grep af-packet
af-packet = (null)
af-packet.0 = interface
af-packet.0.interface = enp2s0f1
af-packet.0.threads = 8
af-packet.0.cluster-id = 99
af-packet.0.cluster-type = cluster_flow
af-packet.0.defrag = yes
af-packet.0.use-mmap = yes
af-packet.0.mmap-locked = yes
af-packet.0.tpacket-v3 = yes
af-packet.0.ring-size = 50000
af-packet.0.rollover = no
af-packet.1 = interface
af-packet.1.interface = enp3s0f0
af-packet.1.threads = 8
af-packet.1.cluster-type = cluster_flow
af-packet.1.cluster-id = 98
af-packet.1.use-mmap = yes
af-packet.1.mmap-locked = yes
af-packet.1.rollover = no
af-packet.1.tpacket-v3 = yes
af-packet.1.ring-size = 50000
root@suricata:~# ethtool -l enp2s0f1
Channel parameters for enp2s0f1:
Pre-set maximums:
RX: 8
TX: 8
Other: 0
Combined: 0
Current hardware settings:
RX: 8
TX: 8
Other: 0
Combined: 0
root@suricata:~# ethtool -l enp3s0f0
Channel parameters for enp3s0f0:
Pre-set maximums:
RX: 8
TX: 8
Other: 0
Combined: 0
Current hardware settings:
RX: 8
TX: 8
Other: 0
Combined: 0
root@suricata:~# ethtool -i enp2s0f1
driver: bnx2
version: 2.2.6
firmware-version: bc 4.6.4 NCSI 1.0.3
expansion-rom-version:
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
description: Rack Mount Chassis
product: ProLiant DL380 G6 (470065-125)
vendor: HP
serial: CZC9410PQM
width: 64 bits
capabilities: smbios-2.6 dmi-2.6 smp vsyscall32
configuration: boot=hardware-failure-fw chassis=rackmount family=ProLiant sku=470065-125 uuid=34373030-3635-435A-4339-34313050514D
*-core
description: Motherboard
physical id: 0
*-firmware
description: BIOS
vendor: HP
physical id: 0
version: P62
date: 07/24/2009
size: 64KiB
capacity: 8128KiB
capabilities: pci pnp upgrade shadowing escd cdboot bootselect edd int13floppy360 int13floppy1200 int13floppy720 int5printscreen int9keyboard int14serial int17printer int10video acpi usb biosbootspecification netboot
root@suricata:~# uname -r
4.15.0-136-genericOn the latest analysis, the tcp.pkt_on_wrong_thread value is about 0.8% of the total kernel_packets. It´s below 1%
What is the output of ethtool -x interface_name?
This seems a bit off (3 times more ack than syn seen)
tcp.syn | Total | 10571117
tcp.synack | Total | 3268041
tcp.rst | Total | 2334933
root@suricata:~# ethtool -x enp2s0f1
Cannot get RX ring count: Operation not supported
root@suricata:~# ethtool -x enp3s0f0
Cannot get RX ring count: Operation not supported
Hi,
The CPU utilization for the worker threads will correlate with the traffic ingest rate.
You can view the CPU affinity assignments for the Suricata worker threads – htop will display this information or you can use this (on a Linux system):
ps -p $(pidof suricata) -L|cut -f4 -d' '|while read tid; do taskset -c -p $tid; done
For htop, choose Setup then
- Display options:
- Select
Show custom thread names - Clear
Hide userland threads
- Select
Select Columns and add Processor to the Active Columns list
I suspect the fact that the ethtool command does not return a supported results is because Brodcoam might have it’s own version of the tool that should be used to tune and configure the NIC. The documentation should usually mention something about it.
It is very important that the correct tool/version is used for the NIC tuning. Otherwise that could explain some different results.
Some sort of result should be returned. Something like -
/usr/sbin/ethtool -x ens1np0
RX flow hash indirection table for ens1np0 with 32 RX ring(s):
0: 0 1 2 3 4 5 6 7
8: 8 9 10 11 12 13 14 15
16: 16 17 18 19 20 21 22 23
24: 24 25 26 27 28 29 30 31
32: 0 1 2 3 4 5 6 7
40: 8 9 10 11 12 13 14 15
48: 16 17 18 19 20 21 22 23
56: 24 25 26 27 28 29 30 31
64: 0 1 2 3 4 5 6 7
72: 8 9 10 11 12 13 14 15
80: 16 17 18 19 20 21 22 23
88: 24 25 26 27 28 29 30 31
96: 0 1 2 3 4 5 6 7
104: 8 9 10 11 12 13 14 15
112: 16 17 18 19 20 21 22 23
120: 24 25 26 27 28 29 30 31
RSS hash key:
23:2b:7e:f2
RSS hash function:
toeplitz: off
xor: off
crc32: on
I caught the pattern here. I disabled the inside interface (I am monitoring LAN and WAN) and now I have 0 drops and 0 wrong_thread. Maybe something related to that interface only?
enp2s0f1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1514
inet6 fe80::226:55ff:fe4a:7654 prefixlen 64 scopeid 0x20
inet6 2001:690:810:72:226:55ff:fe4a:7654 prefixlen 64 scopeid 0x0
ether 00:26:55:4a:76:54 txqueuelen 1000 (Ethernet)
RX packets 44281932070 bytes 25777168600414 (25.7 TB)
RX errors 54700193 dropped 54400 overruns 0 frame 54700193
TX packets 515172 bytes 54370677 (54.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 fe80::226:55ff:fe4a:7656 prefixlen 64 scopeid 0x20
inet6 2001:690:810:72:226:55ff:fe4a:7656 prefixlen 64 scopeid 0x0
ether 00:26:55:4a:76:56 txqueuelen 1000 (Ethernet)
RX packets 38426128594 bytes 21749982633841 (21.7 TB)
RX errors 166321218 dropped 51800987 overruns 0 frame 166321218
TX packets 13202 bytes 1506260 (1.5 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0 is the LAN interface. I think it has a abnormal drop count