Suricata http traffic parse exception in mirror traffic

Help
1

Hi，
I’m using suricata as my ids to monitor network traffic. I have installed suricata 6.0.4 on centos 7 from the copr repo.
however, I found that suricata can’t parse http traffic correctly on my machine.
I tried to capture data using tcpdump and save it as pcap files, then analyze http data using wireshark, both wireshark and tcpdump can restore the http data correctly but suricata did not.

image
image1112×116 6.95 KB

i can’t see it in the http.log(using suricata -r test.pcap -c /etc/suricata -l ./, i have enabled the http.log and other settings is default.
futher more， all the extranet http traffic can’t be seen in the http.log.
can anyone help me? what should i do?

Date: 12/13/2021 – 10:57:06 (uptime: 0d, 00h 00m 00s)

Counter | TM Name | Value

decoder.pkts | Total | 84
decoder.bytes | Total | 67779
decoder.ipv4 | Total | 84
decoder.ethernet | Total | 84
decoder.tcp | Total | 84
decoder.vlan | Total | 52
decoder.avg_pkt_size | Total | 806
decoder.max_pkt_size | Total | 1438
flow.tcp | Total | 10
flow.wrk.spare_sync_avg | Total | 100
flow.wrk.spare_sync | Total | 7
tcp.sessions | Total | 2
tcp.syn | Total | 2
tcp.synack | Total | 2
flow.mgr.full_hash_pass | Total | 1
flow.spare | Total | 9300
flow.mgr.rows_maxlen | Total | 1
flow.mgr.flows_checked | Total | 10
flow.mgr.flows_notimeout | Total | 10
tcp.memuse | Total | 14548992
tcp.reassembly_memuse | Total | 2359296
flow.memuse | Total | 7394304

2

Can you share the config?
I guess the pcap is not to be shared?
Hard to tell why a rather default setting does not work

3

hi，Andreas
thank you for your reply :grinning:.
i have already solved that problem not long ago.
it was caused by the default setting:

vlan:
  use-for-tracking: true

i turned it off, then i found that suricata can parse the http traffic correctly.
It’s a small problem, but it has troubled me for a long time :rofl:
This issue can be closed. Sorry for wasting your time.

4

hi， Andreas
after a few days， i found that the problem has not been completely solved.
the http traffic parse exception problem still happens sometimes.
i caputre the traffic using tcpdump, here is the pcap file.
a.pcap (1.2 KB)
wireshark can restore the http data correctly
but i can’t restore it using suricata.
using

suricata -r a.pcap -c suricata.yaml

here is the config file
suricata.yaml (70.9 KB)

in addition, i reinstall it from the source in order to use pf_ring. And the suricata build info:

This is Suricata version 6.0.4 RELEASE
Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LIBJANSSON PROFILING TLS TLS_GNU MAGIC RUST 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-44), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.39, linked against LibHTP v0.5.39

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         yes
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no 
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes
  libluajit:                               no
  GeoIP2 support:                          yes
  Non-bundled htp:                         no
  Hyperscan support:                       no
  Libnet support:                          yes
  liblz4 support:                          yes
  HTTP2 decompression:                     no

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /usr/bin/rustc
  Rust compiler version:                   rustc 1.56.1 (Red Hat 1.56.1-1.el7)
  Cargo path:                              /usr/bin/cargo
  Cargo version:                           cargo 1.56.0
  Cargo vendor:                            yes

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Python distutils                         yes
  Python yaml                              yes
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       yes
  Profiling locks enabled:                 no

  Plugin support (experimental):           yes

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-unknown-linux-gnu
  Compiler:                                gcc (exec name) / g++ (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -std=gnu99 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist
  PCAP_CFLAGS                               
  SECCFLAGS
5

Hi, I also meet your problem. My Suricata can’t parse http pcaps.
did you solve this problem?

6

haven’t yet. i guess it might be libhtp does not parse correctly :pensive:.
In addition, i found that if you close the checksum-validation might show you more http logs, but still not all. Set the stream.memcap and stream.reassembly.memcap also useful，But none of the above can completely solve the problem.

If you have solved this problem，please let me know. thx