I have what I think is an issue with either flow hashing or stream reassembly – I’m fairly new to suricata so I’m still learning. I’m seeing a flood of stream events complaining about invalid acks being seen:
04/12/2021-16:08:19.399940 [**] [1:2210010:2] SURICATA STREAM 3way handshake wrong seq wrong ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:443 -> xxxx:51746
04/12/2021-16:08:19.411492 [**] [1:2210045:2] SURICATA STREAM Packet with invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:42760 -> xxxx:443
04/12/2021-16:08:19.411492 [**] [1:2210046:2] SURICATA STREAM SHUTDOWN RST invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:42760 -> xxxx:443
04/12/2021-16:08:19.428855 [**] [1:2210045:2] SURICATA STREAM Packet with invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:64704 -> xxxx:443
04/12/2021-16:08:19.428855 [**] [1:2210046:2] SURICATA STREAM SHUTDOWN RST invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:64704 -> xxxx:443
04/12/2021-16:08:19.431376 [**] [1:2210045:2] SURICATA STREAM Packet with invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:52444 -> xxxx:443
04/12/2021-16:08:19.431376 [**] [1:2210046:2] SURICATA STREAM SHUTDOWN RST invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:52444 -> xxxx:443
04/12/2021-16:08:19.434586 [**] [1:2210045:2] SURICATA STREAM Packet with invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:46400 -> xxxx:443
04/12/2021-16:08:19.434586 [**] [1:2210046:2] SURICATA STREAM SHUTDOWN RST invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:46400 -> xxxx:443
04/12/2021-16:08:19.437184 [**] [1:2210045:2] SURICATA STREAM Packet with invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:19249 -> xxxx:443
04/12/2021-16:08:19.437184 [**] [1:2210046:2] SURICATA STREAM SHUTDOWN RST invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:19249 -> xxxx:443
04/12/2021-16:08:19.437685 [**] [1:2210045:2] SURICATA STREAM Packet with invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:51729 -> xxxx:443
04/12/2021-16:08:19.437685 [**] [1:2210046:2] SURICATA STREAM SHUTDOWN RST invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:51729 -> xxxx:443
04/12/2021-16:08:19.449960 [**] [1:2210045:2] SURICATA STREAM Packet with invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:55874 -> xxxx:443
04/12/2021-16:08:19.449960 [**] [1:2210046:2] SURICATA STREAM SHUTDOWN RST invalid ack [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:55874 -> xxxx:443
04/12/2021-16:08:19.450675 [**] [1:2210042:2] SURICATA STREAM TIMEWAIT ACK with wrong seq [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} xxxx:16862 -> xxxx:443
Yes, I know I could disable the relevant rules in the config but I believe this is indicative of a problem in my setup somehow which is fairly complicated. We have a box running in IDS mode capturing all the inbound/outbound traffic via a passive fiber tap that is connected to our two ISP uplinks. What this means is that there are 4 separate 10G interfaces each only seeing one direction of traffic. I’ve solved for this by putting all the interface in a single bond and suricata is configured to use that bond interface. I’ve ensured that there is a single RSS queue on each physical interface and a single queue on the bond interface (by default the bond driver creates 16). I’m fairly certain I’m not running out of memory and losing flows due to hitting memcap. I’m a little bit worried its a bug or issue with the linux bond driver though I’m not sure where to start looking.
Our traffic is mostly https, but there is going to be a mix of everything: ftp, dns, ssh, etc. We are normally seeing around 6-700 Mbps (150 kpps) but it does periodically burst up to 1.5Gbps (240kpps) for backups or other large file transfers. The invalid ack alerts fire constantly though – even at the lower traffic rates.
I am running suricata 6.0.2 on Ubuntu 20.04 (kernel 5.4.0-65-generic) on a box with 24 cores. The interfaces are all X710s with the latest i40g drivers from intel. Here are relevant suricata.yaml configs and output from stats.log:
- interface: bond1
threads: 14
cluster-id: 99
cluster-type: cluster_flow
defrag: yes
use-mmap: yes
#mmap-locked: yes
tpacket-v3: yes
ring-size: 400000
block-size: 1048576
buffer-size: 262144
checksum-checks: kernel
[...]
defrag:
memcap: 512mb
hash-size: 65536
trackers: 65535 # number of defragmented flows to follow
max-frags: 262140 # number of fragments to keep (higher than trackers)
prealloc: yes
timeout: 60
flow:
memcap: 1gb
hash-size: 65536
prealloc: 50000
emergency-recovery: 30
vlan:
use-for-tracking: false
flow-timeouts:
default:
new: 30
established: 300
closed: 0
bypassed: 100
emergency-new: 10
emergency-established: 100
emergency-closed: 0
emergency-bypassed: 50
tcp:
new: 60
established: 600
closed: 60
bypassed: 100
emergency-new: 5
emergency-established: 100
emergency-closed: 10
emergency-bypassed: 50
udp:
new: 30
established: 300
bypassed: 100
emergency-new: 10
emergency-established: 100
emergency-bypassed: 50
icmp:
new: 30
established: 300
bypassed: 100
emergency-new: 10
emergency-established: 100
emergency-bypassed: 50
stream:
memcap: 12gb
checksum-validation: no # reject incorrect csums
prealloc-sessions: 200000
inline: no # auto will use inline mode in IPS mode, yes or no set it statically
#bypass: yes
reassembly:
memcap: 20gb
depth: 8mb # reassemble 1mb into a stream
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: yes
stats.log:
Date: 4/12/2021 -- 16:32:32 (uptime: 0d, 00h 43m 35s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
capture.kernel_packets | Total | 365348207
capture.kernel_drops | Total | 0
capture.errors | Total | 0
decoder.pkts | Total | 365467575
decoder.bytes | Total | 200943489720
decoder.invalid | Total | 15
decoder.ipv4 | Total | 341345244
decoder.ipv6 | Total | 24126252
decoder.ethernet | Total | 365467575
decoder.chdlc | Total | 0
decoder.raw | Total | 0
decoder.null | Total | 0
decoder.sll | Total | 0
decoder.tcp | Total | 306751723
decoder.udp | Total | 54583742
decoder.sctp | Total | 0
decoder.icmpv4 | Total | 150189
decoder.icmpv6 | Total | 27084
decoder.ppp | Total | 0
decoder.pppoe | Total | 0
decoder.geneve | Total | 0
decoder.gre | Total | 0
decoder.vlan | Total | 0
decoder.vlan_qinq | Total | 0
decoder.vxlan | Total | 1
decoder.ieee8021ah | Total | 0
decoder.teredo | Total | 0
decoder.ipv4_in_ipv6 | Total | 0
decoder.ipv6_in_ipv6 | Total | 0
decoder.mpls | Total | 0
decoder.avg_pkt_size | Total | 549
decoder.max_pkt_size | Total | 3222
decoder.max_mac_addrs_src | Total | 0
decoder.max_mac_addrs_dst | Total | 0
decoder.erspan | Total | 0
flow.memcap | Total | 0
flow.tcp | Total | 3889757
flow.udp | Total | 268824
flow.icmpv4 | Total | 6939
flow.icmpv6 | Total | 522
flow.tcp_reuse | Total | 813488
flow.get_used | Total | 0
flow.get_used_eval | Total | 0
flow.get_used_eval_reject | Total | 0
flow.get_used_eval_busy | Total | 0
flow.get_used_failed | Total | 0
flow.wrk.spare_sync_avg | Total | 100
flow.wrk.spare_sync | Total | 22913
flow.wrk.spare_sync_incomplete | Total | 0
flow.wrk.spare_sync_empty | Total | 0
defrag.ipv4.fragments | Total | 0
defrag.ipv4.reassembled | Total | 0
defrag.ipv4.timeouts | Total | 0
defrag.ipv6.fragments | Total | 13288
defrag.ipv6.reassembled | Total | 6636
defrag.ipv6.timeouts | Total | 0
defrag.max_frag_hits | Total | 0
decoder.event.ipv4.pkt_too_small | Total | 0
decoder.event.ipv4.hlen_too_small | Total | 0
decoder.event.ipv4.iplen_smaller_than_hlen | Total | 0
decoder.event.ipv4.trunc_pkt | Total | 0
decoder.event.ipv4.opt_invalid | Total | 0
decoder.event.ipv4.opt_invalid_len | Total | 0
decoder.event.ipv4.opt_malformed | Total | 0
decoder.event.ipv4.opt_pad_required | Total | 0
decoder.event.ipv4.opt_eol_required | Total | 0
decoder.event.ipv4.opt_duplicate | Total | 0
decoder.event.ipv4.opt_unknown | Total | 0
decoder.event.ipv4.wrong_ip_version | Total | 0
decoder.event.ipv4.icmpv6 | Total | 0
decoder.event.icmpv4.pkt_too_small | Total | 0
decoder.event.icmpv4.unknown_type | Total | 0
decoder.event.icmpv4.unknown_code | Total | 6
decoder.event.icmpv4.ipv4_trunc_pkt | Total | 0
decoder.event.icmpv4.ipv4_unknown_ver | Total | 0
decoder.event.icmpv6.unknown_type | Total | 0
decoder.event.icmpv6.unknown_code | Total | 0
decoder.event.icmpv6.pkt_too_small | Total | 0
decoder.event.icmpv6.ipv6_unknown_version | Total | 0
decoder.event.icmpv6.ipv6_trunc_pkt | Total | 0
decoder.event.icmpv6.mld_message_with_invalid_hl | Total | 0
decoder.event.icmpv6.unassigned_type | Total | 0
decoder.event.icmpv6.experimentation_type | Total | 0
decoder.event.ipv6.pkt_too_small | Total | 0
decoder.event.ipv6.trunc_pkt | Total | 0
decoder.event.ipv6.trunc_exthdr | Total | 0
decoder.event.ipv6.exthdr_dupl_fh | Total | 0
decoder.event.ipv6.exthdr_useless_fh | Total | 1
decoder.event.ipv6.exthdr_dupl_rh | Total | 0
decoder.event.ipv6.exthdr_dupl_hh | Total | 0
decoder.event.ipv6.exthdr_dupl_dh | Total | 0
decoder.event.ipv6.exthdr_dupl_ah | Total | 0
decoder.event.ipv6.exthdr_dupl_eh | Total | 0
decoder.event.ipv6.exthdr_invalid_optlen | Total | 0
decoder.event.ipv6.wrong_ip_version | Total | 0
decoder.event.ipv6.exthdr_ah_res_not_null | Total | 0
decoder.event.ipv6.hopopts_unknown_opt | Total | 0
decoder.event.ipv6.hopopts_only_padding | Total | 0
decoder.event.ipv6.dstopts_unknown_opt | Total | 0
decoder.event.ipv6.dstopts_only_padding | Total | 0
decoder.event.ipv6.rh_type_0 | Total | 0
decoder.event.ipv6.zero_len_padn | Total | 0
decoder.event.ipv6.fh_non_zero_reserved_field | Total | 0
decoder.event.ipv6.data_after_none_header | Total | 0
decoder.event.ipv6.unknown_next_header | Total | 0
decoder.event.ipv6.icmpv4 | Total | 0
decoder.event.tcp.pkt_too_small | Total | 0
decoder.event.tcp.hlen_too_small | Total | 10
decoder.event.tcp.invalid_optlen | Total | 0
decoder.event.tcp.opt_invalid_len | Total | 51
decoder.event.tcp.opt_duplicate | Total | 0
decoder.event.udp.pkt_too_small | Total | 2
decoder.event.udp.hlen_too_small | Total | 0
decoder.event.udp.hlen_invalid | Total | 0
decoder.event.sll.pkt_too_small | Total | 0
decoder.event.ethernet.pkt_too_small | Total | 0
decoder.event.ppp.pkt_too_small | Total | 0
decoder.event.ppp.vju_pkt_too_small | Total | 0
decoder.event.ppp.ip4_pkt_too_small | Total | 0
decoder.event.ppp.ip6_pkt_too_small | Total | 0
decoder.event.ppp.wrong_type | Total | 0
decoder.event.ppp.unsup_proto | Total | 0
decoder.event.pppoe.pkt_too_small | Total | 0
decoder.event.pppoe.wrong_code | Total | 0
decoder.event.pppoe.malformed_tags | Total | 0
decoder.event.gre.pkt_too_small | Total | 0
decoder.event.gre.wrong_version | Total | 0
decoder.event.gre.version0_recur | Total | 0
decoder.event.gre.version0_flags | Total | 0
decoder.event.gre.version0_hdr_too_big | Total | 0
decoder.event.gre.version0_malformed_sre_hdr | Total | 0
decoder.event.gre.version1_chksum | Total | 0
decoder.event.gre.version1_route | Total | 0
decoder.event.gre.version1_ssr | Total | 0
decoder.event.gre.version1_recur | Total | 0
decoder.event.gre.version1_flags | Total | 0
decoder.event.gre.version1_no_key | Total | 0
decoder.event.gre.version1_wrong_protocol | Total | 0
decoder.event.gre.version1_malformed_sre_hdr | Total | 0
decoder.event.gre.version1_hdr_too_big | Total | 0
decoder.event.vlan.header_too_small | Total | 0
decoder.event.vlan.unknown_type | Total | 0
decoder.event.vlan.too_many_layers | Total | 0
decoder.event.ieee8021ah.header_too_small | Total | 0
decoder.event.ipraw.invalid_ip_version | Total | 0
decoder.event.ltnull.pkt_too_small | Total | 0
decoder.event.ltnull.unsupported_type | Total | 0
decoder.event.sctp.pkt_too_small | Total | 0
decoder.event.ipv4.frag_pkt_too_large | Total | 0
decoder.event.ipv6.frag_pkt_too_large | Total | 0
decoder.event.ipv4.frag_overlap | Total | 0
decoder.event.ipv6.frag_overlap | Total | 0
decoder.event.ipv4.frag_ignored | Total | 0
decoder.event.ipv6.frag_ignored | Total | 0
decoder.event.ipv6.ipv4_in_ipv6_too_small | Total | 0
decoder.event.ipv6.ipv4_in_ipv6_wrong_version | Total | 0
decoder.event.ipv6.ipv6_in_ipv6_too_small | Total | 0
decoder.event.ipv6.ipv6_in_ipv6_wrong_version | Total | 0
decoder.event.mpls.header_too_small | Total | 0
decoder.event.mpls.pkt_too_small | Total | 0
decoder.event.mpls.bad_label_router_alert | Total | 0
decoder.event.mpls.bad_label_implicit_null | Total | 0
decoder.event.mpls.bad_label_reserved | Total | 0
decoder.event.mpls.unknown_payload_type | Total | 0
decoder.event.vxlan.unknown_payload_type | Total | 1
decoder.event.geneve.unknown_payload_type | Total | 0
decoder.event.erspan.header_too_small | Total | 0
decoder.event.erspan.unsupported_version | Total | 0
decoder.event.erspan.too_many_vlan_layers | Total | 0
decoder.event.dce.pkt_too_small | Total | 0
decoder.event.chdlc.pkt_too_small | Total | 0
decoder.too_many_layers | Total | 0
flow_bypassed.local_pkts | Total | 0
flow_bypassed.local_bytes | Total | 0
flow_bypassed.local_capture_pkts | Total | 0
flow_bypassed.local_capture_bytes | Total | 0
flow.wrk.flows_evicted_needs_work | Total | 1489009
flow.wrk.flows_evicted_pkt_inject | Total | 2936533
flow.wrk.flows_evicted | Total | 499457
flow.wrk.flows_injected | Total | 1375969
tcp.sessions | Total | 3027328
tcp.ssn_memcap_drop | Total | 0
tcp.pseudo | Total | 27522
tcp.pseudo_failed | Total | 0
tcp.invalid_checksum | Total | 0
tcp.no_flow | Total | 0
tcp.syn | Total | 3220343
tcp.synack | Total | 3541752
tcp.rst | Total | 2585587
tcp.midstream_pickups | Total | 0
tcp.pkt_on_wrong_thread | Total | 1196
tcp.segment_memcap_drop | Total | 0
tcp.stream_depth_reached | Total | 9
tcp.reassembly_gap | Total | 2752
tcp.overlap | Total | 126347
tcp.overlap_diff_data | Total | 2
tcp.insert_data_normal_fail | Total | 0
tcp.insert_data_overlap_fail | Total | 0
tcp.insert_list_fail | Total | 0
detect.alert | Total | 854551
app_layer.flow.http | Total | 358311
app_layer.tx.http | Total | 610175
app_layer.flow.ftp | Total | 133
app_layer.tx.ftp | Total | 6571
app_layer.flow.smtp | Total | 7985
app_layer.tx.smtp | Total | 8167
app_layer.flow.tls | Total | 1626422
app_layer.tx.tls | Total | 0
app_layer.flow.ssh | Total | 261
app_layer.tx.ssh | Total | 0
app_layer.flow.imap | Total | 0
app_layer.tx.imap | Total | 0
app_layer.flow.dns_tcp | Total | 1358
app_layer.tx.dns_tcp | Total | 2869
app_layer.flow.nfs_tcp | Total | 0
app_layer.tx.nfs_tcp | Total | 0
app_layer.flow.ntp | Total | 374
app_layer.tx.ntp | Total | 547
app_layer.flow.ftp-data | Total | 1035
app_layer.tx.ftp-data | Total | 0
app_layer.flow.tftp | Total | 65
app_layer.tx.tftp | Total | 85
app_layer.flow.ikev2 | Total | 16
app_layer.tx.ikev2 | Total | 10
app_layer.flow.krb5_tcp | Total | 0
app_layer.tx.krb5_tcp | Total | 0
app_layer.flow.dhcp | Total | 0
app_layer.tx.dhcp | Total | 0
app_layer.flow.snmp | Total | 482
app_layer.tx.snmp | Total | 572
app_layer.flow.sip | Total | 2383
app_layer.tx.sip | Total | 2383
app_layer.flow.rdp | Total | 0
app_layer.tx.rdp | Total | 0
app_layer.flow.failed_tcp | Total | 4215
app_layer.flow.dns_udp | Total | 193012
app_layer.tx.dns_udp | Total | 375239
app_layer.flow.nfs_udp | Total | 0
app_layer.tx.nfs_udp | Total | 0
app_layer.flow.krb5_udp | Total | 0
app_layer.tx.krb5_udp | Total | 0
app_layer.flow.failed_udp | Total | 72492
flow.mgr.full_hash_pass | Total | 11
flow.mgr.closed_pruned | Total | 0
flow.mgr.new_pruned | Total | 0
flow.mgr.est_pruned | Total | 0
flow.mgr.bypassed_pruned | Total | 0
flow.spare | Total | 47299
flow.emerg_mode_entered | Total | 0
flow.emerg_mode_over | Total | 0
flow.mgr.rows_maxlen | Total | 10
flow.mgr.flows_checked | Total | 88300
flow.mgr.flows_notimeout | Total | 59198
flow.mgr.flows_timeout | Total | 29102
flow.mgr.flows_timeout_inuse | Total | 0
flow.mgr.flows_evicted | Total | 3295817
flow.mgr.flows_evicted_needs_work | Total | 1375969
flow_bypassed.closed | Total | 0
flow_bypassed.pkts | Total | 0
flow_bypassed.bytes | Total | 0
tcp.memuse | Total | 784171568
tcp.reassembly_memuse | Total | 234451976
http.memuse | Total | 19369907
http.memcap | Total | 0
ftp.memuse | Total | 14195
ftp.memcap | Total | 0
app_layer.expectations | Total | 10
file_store.open_files | Total | 0
flow.memuse | Total | 138416704
