Suricata 7.0 conditional pcap does not record entire flow packets to pcap as expected

Hello, everyone:
Recently I am using conditional pcap feature of Suricata version 7.0.0-rc2, and my doubt or problem is that, it seems that suricata just logged the flow packet which triggers one specific rule to pcap file, not the entire flow packets, which starts from something like tcp three-way handshake;

Here is my config:

      - pcap-log:
          enabled: yes
          filename: log.pcap
          limit: 1024mb
          max-files: 2000
          compression: none
          #lz4-checksum: no
          #lz4-level: 0
          mode: normal # normal, multi or sguil.
          #dir: /nsm_data/
          #ts-format: usec 
          use-stream-depth: no 
          honor-pass-rules: no 
          conditional: alerts
# stream:
#   memcap: 64mb                # Can be specified in kb, mb, gb.  Just a
#                               # number indicates it's in bytes.
#   memcap-policy: ignore       # Can be "drop-flow", "pass-flow", "bypass",
#                               # "drop-packet", "pass-packet", "reject" or
#                               # "ignore" default is "ignore"
#   checksum-validation: yes    # To validate the checksum of received
#                               # packet. If csum validation is specified as
#                               # "yes", then packets with invalid csum values will not
#                               # be processed by the engine stream/app layer.
#                               # Warning: locally generated traffic can be
#                               # generated without checksum due to hardware offload
#                               # of checksum. You can control the handling of checksum
#                               # on a per-interface basis via the 'checksum-checks'
#                               # option
#   prealloc-sessions: 2048     # 2k sessions prealloc'd per stream thread
#   midstream: false            # don't allow midstream session pickups
#   midstream-policy: ignore    # Can be "drop-flow", "pass-flow", "bypass",
#                               # "drop-packet", "pass-packet", "reject" or
#                               # "ignore" default is "ignore"
#   async-oneside: false        # don't enable async stream handling
#   inline: no                  # stream inline mode
#   drop-invalid: yes           # in inline mode, drop packets that are invalid with regards to streaming engine
#   max-syn-queued: 10          # Max different SYNs to queue
#   max-synack-queued: 5        # Max different SYN/ACKs to queue
#   bypass: no                  # Bypass packets when stream.reassembly.depth is reached.
#                               # Warning: first side to reach this triggers
#                               # the bypass.
#   liberal-timestamps: false   # Treat all timestamps as if the Linux policy applies. This
#                               # means it's slightly more permissive. Enabled by default.
#
#   reassembly:
#     memcap: 256mb             # Can be specified in kb, mb, gb.  Just a number
#                               # indicates it's in bytes.
#     memcap-policy: ignore     # Can be "drop-flow", "pass-flow", "bypass",
#                               # "drop-packet", "pass-packet", "reject" or
#                               # "ignore" default is "ignore"
#     depth: 1mb                # Can be specified in kb, mb, gb.  Just a number
#                               # indicates it's in bytes.
#     toserver-chunk-size: 2560 # inspect raw stream in chunks of at least
#                               # this size.  Can be specified in kb, mb,
#                               # gb.  Just a number indicates it's in bytes.
#     toclient-chunk-size: 2560 # inspect raw stream in chunks of at least
#                               # this size.  Can be specified in kb, mb,
#                               # gb.  Just a number indicates it's in bytes.
#     randomize-chunk-size: yes # Take a random value for chunk size around the specified value.
#                               # This lowers the risk of some evasion techniques but could lead
#                               # to detection change between runs. It is set to 'yes' by default.
#     randomize-chunk-range: 10 # If randomize-chunk-size is active, the value of chunk-size is
#                               # a random value between (1 - randomize-chunk-range/100)*toserver-chunk-size
#                               # and (1 + randomize-chunk-range/100)*toserver-chunk-size and the same
#                               # calculation for toclient-chunk-size.
#                               # Default value of randomize-chunk-range is 10.
#
#     raw: yes                  # 'Raw' reassembly enabled or disabled.
#                               # raw is for content inspection by detection
#                               # engine.
#
#     segment-prealloc: 2048    # number of segments preallocated per thread
#
#     check-overlap-different-data: true|false
#                               # check if a segment contains different data
#                               # than what we've already seen for that
#                               # position in the stream.
#                               # This is enabled automatically if inline mode
#                               # is used or when stream-event:reassembly_overlap_different_data;
#                               # is used in a rule.
stream:
  memcap: 64mb
  #memcap-policy: ignore
  checksum-validation: yes      # reject incorrect csums
  midstream: no
  #midstream-policy: ignore
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 256mb
    #memcap-policy: ignore
    depth: 32mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
    #raw: yes
    #segment-prealloc: 2048
    #check-overlap-different-data: true

and here is my rules used during the experiment:

alert http $HOME_NET any -> any any (msg:"ET POLICY Http  url contains passwd= in cleartext"; flow:established,to_server; content:"passwd="; http_uri;nocase; classtype:policy-violation; sid:4000001; rev:4; metadata:created_at 2011_05_30, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY HTTP url contains pass= in cleartext"; flow:established,to_server; content:"pass=";http_uri; nocase; classtype:policy-violation; sid:4000002; rev:4; metadata:created_at 2011_05_30, former_category POLICY, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY HTTP url  contains passphrase= in cleartext"; flow:established,to_server; content:"passphrase=";http_uri; nocase; classtype:policy-violation; sid:4000003; rev:4; metadata:created_at 2011_05_30, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY Http url  contains pword= in cleartext"; flow:established,to_server;  content:"pword="; http_uri;nocase; classtype:policy-violation; sid:4000004; rev:4; metadata:created_at 2011_05_30, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY Http url  contains password in cleartext"; flow:established,to_server; content:"password=";http_uri; nocase; classtype:policy-violation; sid:4000005; rev:4; metadata:created_at 2011_05_30, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY Http Client Body json contains passwd in cleartext"; flow:established,to_server;  content:"passwd"; http_client_body;nocase; classtype:policy-violation; sid:4000010; rev:4; metadata:created_at 2011_05_30, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY HTTP POST  json contains passWord in cleartext"; flow:established,to_server;  content:"password"; http_client_body;nocase; classtype:policy-violation; sid:4000011; rev:4; metadata:created_at 2011_05_30, former_category POLICY, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY Http Client Body  json contains pword in cleartext"; flow:established,to_server;  content:"pword";http_client_body; nocase; classtype:policy-violation; sid:4000012; rev:4; metadata:created_at 2011_05_30, updated_at 2020_04_21;)
alert http $HOME_NET any -> any any (msg:"ET POLICY Http Client Body  json contains passphrase in cleartext"; flow:established,to_server;  content:"passphrase"; http_client_body;nocase; classtype:policy-violation; sid:4000013; rev:4; metadata:created_at 2011_05_30, updated_at 2020_04_21;)

and this is the curl command that triggerred alert:

curl --interface eno7 http://192.168.30.73:8080/llogin  -H "Content-Type: application/json" -X POST -d '{"username":"admin", "password":"jiangming"}'

and here is my the pcap file that suricata logged:
log.pcap (6.6 KB)
and screen shot:


it seems that suricata just logs the Flow packet which triggers the alert, and the following packets that after the triggerring packet, but not including the packets before the triggering packet, like tcp three-way handshake. My question is, is this correct that suricata just logs the triggering packet but not including something like tcp three-way handshake.
Anyone can explain ? and thanks a lot !

1 Like

Yes your analysis is correct. The algorithm is currently streaming based and it will not log the packets without any data that are seen before the alert.

What if any data seen before the alert? Will conditional pcap feature log those packets? Say like within the TCP session before triggering the alert we do some other activities, will the streaming based conditional pcap log those activities? Thanks in advance!

Thanks, Eric, your answer helped me a lot, and If I want to achieve a complete flow packets, do you have any suggestions ?

Capture goes back in time by just using what has currently not being inspected (so discarded). Due to the fact there is a minimum stream size to reach before inspection some may be captured but there is strictly no guarantee. And it could not really work in a full extend, for example take a SMB session that stays up for days and then trigger an alert.

Without coding, you could activate FPC in Suricata and extract content when there is an alert to move it to another PCAP file (for example at flow end for all alerted flows). With coding, it should be possible by attaching the initial packets to the flow entry and then dumping then when alert is seen but this would not be correct if alert occurs a few transactions after the start of the flow.

Thanks for prompt reply! So from code perspective the mechanism is not session based but stream based and it depends on the buffer size for the capture before the alert? Say like a login web page refresh using tcp long connection on the login page followed by an clear text login trigger the alert( alert for clear text password), so it is highly possible that the action of refreshing pages packets being captured via conditional pcap before clear text login(alert start to trigger)?