Unique use case for Suricata File Carving - Need expert advice

, ,
1

Hello,

I have an interesting scenario that requires carving a file off of the wire being sent via HTTP to a destination that doesn’t actually exist.

My situation is such that I am pushing a file over the network (via scapy) which looks like an HTTP PUT. The destination IP address isn’t really on the network, but the router default routes the traffic past my tap where Suricata is listening in IDS mode.

The host that is doing the PUT has iptables setup to reject all incoming RST and RST, ACK flags to keep the HTTP PUT going (even though scapy can ignore them anyway).

The conditions:

  • A file is being sent over the network via an HTTP PUT to a non-existent destination
  • The file is being sent using random starting sequence numbers so each PUT of the same file looks different to Suricata (the file still has the same SHA256 hash, but the file time updates showing that it’s a new file)
  • Suricata is configured with file-store enabled and stream-depth set to 0
  • libhttp default-config request-body-limit and response-body-limit is 0
  • stream memcap is 0
  • midstream is set to true
  • I have a local.rules file that is configured to carve files via http and to store all files (I will tightening this later for a specific file extension)
  • HOME_NET and EXTERNAL_NET is any and any

I’m able to carve the file when I fake a three way handshake (that is generate and push the three way handshake with scapy) then push the file with HTTP PUT. If I don’t send the three way handshake, the file never gets carved.

I have a few questions:

  1. Should Suricata be able to carve the file from the stream even if no three way handshake is seen by using the “midstream true” setting? If yes, do I need to set any other parameters for this to work?

  2. Why does Suricata sometimes carve the file as soon as the HTTP PUT finishes, but other times it just sits and waits? But if I stop the Suricata service after my PUT is finished, a partial file is written to the file-store directory.

  3. If I send FIN and FIN,ACK packets after the HTTP PUT, would that help Suricata know that the stream is complete?

  4. Besides the suricata.yaml settings I listed above, what else do I need to set to be sure I can send a file of any size and it will get carved without being truncated? Small files seem to be fine, but larger files (20MB+) sometimes don’t get carved fully (they are smaller than the original file).

Thank you for your help with this!

2

I found the async setting that seems to help with the one way data push. My recent results provide consistent file carving some times, and inconsistent other times, even with no variable changes. Thoughts on this and the scenario overall?

3

You would have to use the option if parts of the flow are missing. In general Suricata works much better if it sees the full flow, thus bidirectional traffic.