I use traffic mirroring on AWS platform to send data to Suricata Server via VXLAN. Suricata received HTTP traffic for multiple Nginx data.
Nginx uses TCP keeplive, which encapsulates HTTP requests from different users in the same TCP session.When using TCP keeplive, Suricata cannot fully parse out the HTTP request in the TCP stream.
Usually only parses a part, the rest of the HTTP data is not restored normally.
Can anyone help me? What should I do?
Are you able to share a (small) pcap of this traffic?
Another thing that could be useful is to share the stats.log so traffic anomalies can be observed.
Due to privacy issues, I cannot share my pcap.But I can share my stats.log
This problem bothers me.The HTTP traffic data parsed by Suricata is only 20% of the Nginx Web Log
How do I upload a stats.log ? It seems not allowed.
------------------------------------------------------------------------------------
Date: 4/2/2020 -- 08:53:26 (uptime: 0d, 23h 18m 24s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
capture.kernel_packets | Total | 3229742576
decoder.pkts | Total | 3229758632
decoder.bytes | Total | 3980496400028
decoder.ipv4 | Total | 6458925617
decoder.ipv6 | Total | 23
decoder.ethernet | Total | 3229758632
decoder.tcp | Total | 3141175141
decoder.udp | Total | 3229170172
decoder.icmpv6 | Total | 23
decoder.vxlan | Total | 3229170078
decoder.avg_pkt_size | Total | 1232
decoder.max_pkt_size | Total | 9015
flow.tcp | Total | 14883182
flow.udp | Total | 921
flow.icmpv6 | Total | 23
decoder.event.tcp.opt_invalid_len | Total | 1
stream.3whs_synack_resend_with_diff_ack | Total | 1
stream.3whs_syn_resend_diff_seq_on_syn_recv | Total | 1
stream.3whs_wrong_seq_wrong_ack | Total | 1690
stream.closewait_fin_out_of_window | Total | 17
stream.closewait_pkt_before_last_ack | Total | 3
stream.est_packet_out_of_window | Total | 3436
stream.est_pkt_before_last_ack | Total | 26396
stream.est_syn_resend_diff_seq | Total | 4477
stream.est_invalid_ack | Total | 43515
stream.fin_invalid_ack | Total | 604
stream.fin1_fin_wrong_seq | Total | 5
stream.fin1_invalid_ack | Total | 6
stream.fin_but_no_session | Total | 320534
stream.fin_out_of_window | Total | 4
stream.lastack_ack_wrong_seq | Total | 2
stream.lastack_invalid_ack | Total | 1
stream.rst_but_no_session | Total | 442099
stream.timewait_ack_wrong_seq | Total | 12
stream.pkt_invalid_timestamp | Total | 9693
stream.pkt_invalid_ack | Total | 44219
stream.pkt_broken_ack | Total | 1
stream.rst_invalid_ack | Total | 93
stream.pkt_retransmission | Total | 7585
stream.reassembly_seq_gap | Total | 1894998
tcp.sessions | Total | 13391042
tcp.syn | Total | 13843576
tcp.synack | Total | 13259391
tcp.rst | Total | 5760661
tcp.stream_depth_reached | Total | 34
tcp.reassembly_gap | Total | 1894998
tcp.overlap | Total | 135393
detect.alert | Total | 440
app_layer.flow.http | Total | 5090456
app_layer.tx.http | Total | 24276475
app_layer.flow.tls | Total | 5342428
app_layer.flow.ssh | Total | 5
app_layer.flow.failed_tcp | Total | 379201
app_layer.flow.failed_udp | Total | 921
flow_mgr.closed_pruned | Total | 12981363
flow_mgr.new_pruned | Total | 1641957
flow_mgr.est_pruned | Total | 237533
flow.spare | Total | 1048575
flow.tcp_reuse | Total | 4717
flow_mgr.flows_checked | Total | 1274
flow_mgr.flows_notimeout | Total | 1069
flow_mgr.flows_timeout | Total | 205
flow_mgr.flows_timeout_inuse | Total | 52
flow_mgr.flows_removed | Total | 153
flow_mgr.rows_checked | Total | 1048576
flow_mgr.rows_skipped | Total | 1047186
flow_mgr.rows_empty | Total | 132
flow_mgr.rows_maxlen | Total | 2
tcp.memuse | Total | 12930120
tcp.reassembly_memuse | Total | 437959024
http.memuse | Total | 6354735
flow.memuse | Total | 418656968
------------------------------------------------------------------------------------
Date: 4/2/2020 -- 08:54:26 (uptime: 0d, 23h 19m 24s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
capture.kernel_packets | Total | 3230922994
decoder.pkts | Total | 3230941656
decoder.bytes | Total | 3981891809620
decoder.ipv4 | Total | 6461291245
decoder.ipv6 | Total | 23
decoder.ethernet | Total | 3230941656
decoder.tcp | Total | 3142328429
decoder.udp | Total | 3230352778
decoder.icmpv6 | Total | 23
decoder.vxlan | Total | 3230352684
decoder.avg_pkt_size | Total | 1232
decoder.max_pkt_size | Total | 9015
flow.tcp | Total | 14890731
flow.udp | Total | 921
flow.icmpv6 | Total | 23
decoder.event.tcp.opt_invalid_len | Total | 1
stream.3whs_synack_resend_with_diff_ack | Total | 1
stream.3whs_syn_resend_diff_seq_on_syn_recv | Total | 1
stream.3whs_wrong_seq_wrong_ack | Total | 1690
stream.closewait_fin_out_of_window | Total | 17
stream.closewait_pkt_before_last_ack | Total | 3
stream.est_packet_out_of_window | Total | 3436
stream.est_pkt_before_last_ack | Total | 26397
stream.est_syn_resend_diff_seq | Total | 4477
stream.est_invalid_ack | Total | 43515
stream.fin_invalid_ack | Total | 604
stream.fin1_fin_wrong_seq | Total | 5
stream.fin1_invalid_ack | Total | 6
stream.fin_but_no_session | Total | 320686
stream.fin_out_of_window | Total | 4
stream.lastack_ack_wrong_seq | Total | 2
stream.lastack_invalid_ack | Total | 1
stream.rst_but_no_session | Total | 442340
stream.timewait_ack_wrong_seq | Total | 12
stream.pkt_invalid_timestamp | Total | 9693
stream.pkt_invalid_ack | Total | 44219
stream.pkt_broken_ack | Total | 1
stream.rst_invalid_ack | Total | 93
stream.pkt_retransmission | Total | 7591
stream.reassembly_seq_gap | Total | 1895669
tcp.sessions | Total | 13398103
tcp.syn | Total | 13850949
tcp.synack | Total | 13266348
tcp.rst | Total | 5763173
tcp.stream_depth_reached | Total | 34
tcp.reassembly_gap | Total | 1895669
tcp.overlap | Total | 135423
detect.alert | Total | 440
app_layer.flow.http | Total | 5093471
app_layer.tx.http | Total | 24287871
app_layer.flow.tls | Total | 5344448
app_layer.flow.ssh | Total | 5
app_layer.flow.failed_tcp | Total | 379473
app_layer.flow.failed_udp | Total | 921
flow_mgr.closed_pruned | Total | 12988375
flow_mgr.new_pruned | Total | 1642575
flow_mgr.est_pruned | Total | 237652
flow.spare | Total | 1048575
flow.tcp_reuse | Total | 4717
flow_mgr.flows_checked | Total | 1186
flow_mgr.flows_notimeout | Total | 989
flow_mgr.flows_timeout | Total | 197
flow_mgr.flows_timeout_inuse | Total | 73
flow_mgr.flows_removed | Total | 124
flow_mgr.rows_checked | Total | 1048576
flow_mgr.rows_skipped | Total | 1047294
flow_mgr.rows_empty | Total | 109
flow_mgr.rows_maxlen | Total | 2
tcp.memuse | Total | 12930120
tcp.reassembly_memuse | Total | 434865412
http.memuse | Total | 7240014
flow.memuse | Total | 418602520
------------------------------------------------------------------------------------
Date: 4/2/2020 -- 08:55:26 (uptime: 0d, 23h 20m 24s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
capture.kernel_packets | Total | 3232082773
decoder.pkts | Total | 3232101112
decoder.bytes | Total | 3983261229517
decoder.ipv4 | Total | 6463609730
decoder.ipv6 | Total | 23
decoder.ethernet | Total | 3232101112
decoder.tcp | Total | 3143458833
decoder.udp | Total | 3231511809
decoder.icmpv6 | Total | 23
decoder.vxlan | Total | 3231511715
decoder.avg_pkt_size | Total | 1232
decoder.max_pkt_size | Total | 9015
flow.tcp | Total | 14898262
flow.udp | Total | 923
flow.icmpv6 | Total | 23
decoder.event.tcp.opt_invalid_len | Total | 1
stream.3whs_synack_resend_with_diff_ack | Total | 1
stream.3whs_syn_resend_diff_seq_on_syn_recv | Total | 1
stream.3whs_wrong_seq_wrong_ack | Total | 1690
stream.closewait_fin_out_of_window | Total | 17
stream.closewait_pkt_before_last_ack | Total | 3
stream.est_packet_out_of_window | Total | 3436
stream.est_pkt_before_last_ack | Total | 26397
stream.est_syn_resend_diff_seq | Total | 4478
stream.est_invalid_ack | Total | 43515
stream.fin_invalid_ack | Total | 604
stream.fin1_fin_wrong_seq | Total | 5
stream.fin1_invalid_ack | Total | 6
stream.fin_but_no_session | Total | 320871
stream.fin_out_of_window | Total | 4
stream.lastack_ack_wrong_seq | Total | 2
stream.lastack_invalid_ack | Total | 1
stream.rst_but_no_session | Total | 442637
stream.timewait_ack_wrong_seq | Total | 12
stream.pkt_invalid_timestamp | Total | 9693
stream.pkt_invalid_ack | Total | 44219
stream.pkt_broken_ack | Total | 1
stream.rst_invalid_ack | Total | 93
stream.pkt_retransmission | Total | 7591
stream.reassembly_seq_gap | Total | 1896315
tcp.sessions | Total | 13405049
tcp.syn | Total | 13858214
tcp.synack | Total | 13273188
tcp.rst | Total | 5765739
tcp.stream_depth_reached | Total | 34
tcp.reassembly_gap | Total | 1896315
tcp.overlap | Total | 135424
detect.alert | Total | 440
app_layer.flow.http | Total | 5096436
app_layer.tx.http | Total | 24298152
app_layer.flow.tls | Total | 5346397
app_layer.flow.ssh | Total | 5
app_layer.flow.failed_tcp | Total | 379746
app_layer.flow.failed_udp | Total | 923
flow_mgr.closed_pruned | Total | 12995446
flow_mgr.new_pruned | Total | 1643221
flow_mgr.est_pruned | Total | 237762
flow.spare | Total | 1048576
flow.tcp_reuse | Total | 4718
flow_mgr.flows_checked | Total | 1165
flow_mgr.flows_notimeout | Total | 985
flow_mgr.flows_timeout | Total | 180
flow_mgr.flows_timeout_inuse | Total | 41
flow_mgr.flows_removed | Total | 139
flow_mgr.rows_checked | Total | 1048576
flow_mgr.rows_skipped | Total | 1047300
flow_mgr.rows_empty | Total | 127
flow_mgr.rows_maxlen | Total | 3
tcp.memuse | Total | 12930120
tcp.reassembly_memuse | Total | 429272996
http.memuse | Total | 6432877
flow.memuse | Total | 418516256
------------------------------------------------------------------------------------
Date: 4/2/2020 -- 08:56:26 (uptime: 0d, 23h 21m 24s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
capture.kernel_packets | Total | 3233256314
decoder.pkts | Total | 3233274116
decoder.bytes | Total | 3984644497192
decoder.ipv4 | Total | 6465955318
decoder.ipv6 | Total | 23
decoder.ethernet | Total | 3233274116
decoder.tcp | Total | 3144602469
decoder.udp | Total | 3232684395
decoder.icmpv6 | Total | 23
decoder.vxlan | Total | 3232684301
decoder.avg_pkt_size | Total | 1232
decoder.max_pkt_size | Total | 9015
flow.tcp | Total | 14905958
flow.udp | Total | 924
flow.icmpv6 | Total | 23
decoder.event.tcp.opt_invalid_len | Total | 1
stream.3whs_synack_resend_with_diff_ack | Total | 1
stream.3whs_syn_resend_diff_seq_on_syn_recv | Total | 1
stream.3whs_wrong_seq_wrong_ack | Total | 1690
stream.closewait_fin_out_of_window | Total | 17
stream.closewait_pkt_before_last_ack | Total | 3
stream.est_packet_out_of_window | Total | 3436
stream.est_pkt_before_last_ack | Total | 26397
stream.est_syn_resend_diff_seq | Total | 4479
stream.est_invalid_ack | Total | 43515
stream.fin_invalid_ack | Total | 604
stream.fin1_fin_wrong_seq | Total | 5
stream.fin1_invalid_ack | Total | 6
stream.fin_but_no_session | Total | 320969
stream.fin_out_of_window | Total | 4
stream.lastack_ack_wrong_seq | Total | 2
stream.lastack_invalid_ack | Total | 1
stream.rst_but_no_session | Total | 442790
stream.timewait_ack_wrong_seq | Total | 12
stream.pkt_invalid_timestamp | Total | 9693
stream.pkt_invalid_ack | Total | 44219
stream.pkt_broken_ack | Total | 1
stream.rst_invalid_ack | Total | 93
stream.pkt_retransmission | Total | 7591
stream.reassembly_seq_gap | Total | 1897001
tcp.sessions | Total | 13412185
tcp.syn | Total | 13865663
tcp.synack | Total | 13280220
tcp.rst | Total | 5768143
tcp.stream_depth_reached | Total | 34
tcp.reassembly_gap | Total | 1897001
tcp.overlap | Total | 135424
detect.alert | Total | 440
app_layer.flow.http | Total | 5099439
app_layer.tx.http | Total | 24308780
app_layer.flow.tls | Total | 5348501
app_layer.flow.ssh | Total | 5
app_layer.flow.failed_tcp | Total | 380015
app_layer.flow.failed_udp | Total | 924
flow_mgr.closed_pruned | Total | 13002366
flow_mgr.new_pruned | Total | 1643880
flow_mgr.est_pruned | Total | 237895
flow.spare | Total | 1048576
flow.tcp_reuse | Total | 4720
flow_mgr.flows_checked | Total | 1106
flow_mgr.flows_notimeout | Total | 939
flow_mgr.flows_timeout | Total | 167
flow_mgr.flows_timeout_inuse | Total | 43
flow_mgr.flows_removed | Total | 124
flow_mgr.rows_checked | Total | 1048576
flow_mgr.rows_skipped | Total | 1047378
flow_mgr.rows_empty | Total | 109
flow_mgr.rows_maxlen | Total | 2
tcp.memuse | Total | 12930120
tcp.reassembly_memuse | Total | 431618532
http.memuse | Total | 6560912
flow.memuse | Total | 418496576
Upload should work now, but looks like embedding works as well.
stats.tgz share link:
hi this is stats.tgz share link:
I hope to solve this problem, I can provide my suricata.yaml if needed
This stat suggests there is packet loss somewhere. What it means is that data is missing in the TCP data streams. This will certainly cause HTTP logs volume to be much reduced.
@vjulien
I try to capture data using tcpdump and analyze http data using wirehark.wireshark can restore much more http data than suricata.
Why is this? The same pcap, suricata -r restores very little http data
@vjulien
This is a data update every minute. Why does tcp.reassembly_gap keep increasing?
$ tail -f stats.log | grep 'tcp.reassembly_gap'
tcp.reassembly_gap | Total | 4042054
tcp.reassembly_gap | Total | 4042840
tcp.reassembly_gap | Total | 4043583
tcp.reassembly_gap | Total | 4044380
Our HTTP parser currently doesn’t support catching up after a GAP. We’re tracking that work here Task #3559: http: support GAP recovery - Suricata - Open Information Security Foundation
I don’t see any clues in the stats.log. If you record a pcap and replay that in Suricata, do you then see the same behavior?
@vjulien hi, again:
Compared with wireshark, suricata only restores a small amount of http information.
$ suricata --runmode autofp -r /tmp/test3.pcap -l /tmp/suricata
------------------------------------------------------------------------------------
Date: 4/4/2020 -- 08:55:23 (uptime: 0d, 00h 01m 22s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
decoder.pkts | Total | 77289
decoder.bytes | Total | 104335835
decoder.ipv4 | Total | 154578
decoder.ethernet | Total | 77289
decoder.tcp | Total | 74577
decoder.udp | Total | 77289
decoder.vxlan | Total | 77289
decoder.avg_pkt_size | Total | 1349
decoder.max_pkt_size | Total | 9015
flow.tcp | Total | 2268
flow.udp | Total | 128
stream.fin_but_no_session | Total | 213
stream.rst_but_no_session | Total | 128
stream.pkt_invalid_timestamp | Total | 1
stream.reassembly_seq_gap | Total | 22
tcp.sessions | Total | 372
tcp.syn | Total | 372
tcp.synack | Total | 372
tcp.rst | Total | 134
tcp.reassembly_gap | Total | 22
app_layer.flow.http | Total | 170
app_layer.tx.http | Total | 378
app_layer.flow.tls | Total | 98
app_layer.flow.failed_tcp | Total | 14
app_layer.flow.failed_udp | Total | 128
flow_mgr.closed_pruned | Total | 46
flow_mgr.new_pruned | Total | 356
flow.spare | Total | 1048576
flow_mgr.flows_checked | Total | 134
flow_mgr.flows_notimeout | Total | 134
flow_mgr.rows_checked | Total | 1048576
flow_mgr.rows_skipped | Total | 1048442
flow_mgr.rows_maxlen | Total | 1
tcp.memuse | Total | 9175040
tcp.reassembly_memuse | Total | 1572864
flow.memuse | Total | 411827680Suricata tracks TCP sessions by inspecting the sequence and ack numbers. When we see an ACK for data at sequence numbers for which we didn’t see the data, we know that we missed some packets. This condition we call a “GAP”. Whether a parser can recover from that is currently implementation dependent. Some parsers can (nfs, smb, dns) others cannot (http).
The GAPs can have many causes: packet loss, other capture anomalies, memory limits reached, etc. In this case the stats give us no clue.
I would suggest inspecting the (virtual) switch for clues: is the traffic mirror capacity reached? Is only a part of the traffic mirrored? Things like this.
I understand sharing pcaps is hard, but it would be useful. Perhaps sharing it privately could work?
@vjulien
The traffic mirrors the TCP protocol, and the traffic at the mirror port is 1~2G during off-peak hours and 2~3G during peak hours.
You can leave your email, I will send you pcap later, because it involves sensitive information, do not send to anyone
Great, thanks. Just a single TCP session with a GAP is enough to get an idea. My email is vjulien@oisf.net.
Got it, thanks.
So when looking at it in Wireshark, I see lots of:
- [TCP ACKed unseen segment]
- [TCP Previous segment not captured]
- [XX bytes missing in capture file] (in follow TCP stream view)
So the issue is packet loss. Since Suricata doesn’t report any loss, my guess is that it is loss happening before Suricata captures the traffic.
What capture method are you using? (AF_PACKET, NETMAP, etc?)
I use AWS traffic mirroring, and i used AF_PACKET
Is there a way to make Suricata, like Wireshark, restore such packets?