Hi, has anyone noticed a problem with filestore I can’t explain where a file is written to disk but it’s empty? I’m seeing zero byte files get created for extracted files but the log says the file was both saved and CLOSED, not TRUNCATED. The size of the file is correctly logged and it’s well below both the global and file-store stream depth. I’ve noticed this problem with both filestore v1 and v2. I’m running Suricata version 4.1.8 RELEASE.
Can you give us more details about your setup, configuration and how you run Suricata?
And ideally a reproduceable example.
test.pcap (3.0 MB)
Hi Andreas. I did some testing over the weekend. I think it may be related to use of flowbits in the rules? I use flowbits in my file-store rules so they worked with thresholding. When I tried thresholding a normal file-store rule the alerts stopped but files were still extracted and stored to disk from the suppressed IP. In order to get both alerts and extraction to stop as a workaround, I used two rules, one that just sets a flowbit and that I use for the suppression and the other that checks the flowbit and alerts and stores the file.
Here is my current file-store config in suricata-live.yaml.
- file-store:
enabled: yes
log-dir: files
force-magic: yes
force-md5: no
waldo: file.waldo
stream-depth: 3mb
TEST #1
Here is a test using two rules and a flowbit. You can see file.2 is written but it’s empty.
~$ suricata -r /tmp/test.pcap -c suricata-live.yaml --runmode single -k none -S file-extraction-two-rules-flowbits.rules -l .
…
29/6/2020 – 03:15:30 - - Pcap-file module read 1 files, 2415 packets, 3066567 bytes
29/6/2020 – 03:15:30 - - (W#01) Files extracted 1
29/6/2020 – 03:15:30 - - Alerts: 1
$ ls -al files/*
-rw-r----- 1 user primarygroup 0 Jun 29 03:15 files/file.2
-rw-r----- 1 user primarygroup 794 Jun 29 03:15 files/file.2.meta
Here are the two rules in file-extraction-two-rules-flowbits.rules:
This one looks for Windows PE file magic and sets a flowbit if seen.
alert http !$HOME_NET any -> $HOME_NET any (msg:“FILEMAGIC Windows executable flowbit set”; filemagic:“for MS Windows”; flowbits:noalert; flowbits:set,file_extraction_store; sid:7000002; rev:1;)
This is a PE file magic store and alert rule. It checks for the flowbit to be set.
alert http !$HOME_NET any -> $HOME_NET any (msg:“FILEMAGIC Windows executable”; filemagic:“for MS Windows”; filestore; **flowbits:isset,**file_extraction_store; sid:7000001; rev:1;)
TEST #2
When I use a normal file-store rule (one rule, no flowbits). Extraction works and the file is the right size.
~$ suricata -r /tmp/test.pcap -c suricata-live.yaml --runmode single -k none -S file-extraction-one-rule-no-flowbits.rules -l .
…
29/6/2020 – 03:17:30 - - Pcap-file module read 1 files, 2415 packets, 3066567 bytes
29/6/2020 – 03:17:30 - - (W#01) Files extracted 1
29/6/2020 – 03:17:30 - - Alerts: 1
~$ ls -al files/file.3*
-rw-r----- 1 user primarygroup 1048576 Jun 29 03:17 files/file.3
-rw-r----- 1 user primarygroup 794 Jun 29 03:17 files/file.3.meta
alert http !$HOME_NET any -> [$HOME_NET,!$GUEST_NET] any (msg:“FILEMAGIC Windows executable”; filemagic:“for MS Windows”; filestore; sid:7000001; rev:1;)
In both cases the saved .meta file contents are exactly the same:
$ cat file.2.meta
TIME: 06/27/2020-02:29:46.254324
PCAP PKT NUM: 727
SRC IP: 72.21.81.240
DST IP: 172.22.137.67
PROTO: 6
SRC PORT: 80
DST PORT: 64793
APP PROTO: http
HTTP URI: /d/msdownload/update/software/defu/2020/06/am_delta_57b2ef591610c4fb36d241b1c7fa48866541c834.exe
HTTP HOST: au.download.windowsupdate.com
HTTP REFERER:
HTTP USER AGENT: Microsoft-Delivery-Optimization/10.0
FILENAME: /d/msdownload/update/software/defu/2020/06/am_delta_57b2ef591610c4fb36d241b1c7fa48866541c834.exe
MAGIC: PE32+ executable (GUI) x86-64, for MS Windows
STATE: CLOSED
SHA256: 8b5bd2520d8a4dfbececc17de400de09aa566e4177fedb7c2e63fcd0a8aca034
SIZE: 1048576
Note: With a standard file-store v2 config, I also have this problem of 0-byte files being created. I noticed that it goes away but setting force-filestore:yes. But enabling that doesn’t work for me because it stores way too many files that I’m not interested in.
I can reproduce that and IMHO I would expect that it works with flowbits as well, could you submit this as a bugreport to our redmine? (https://redmine.openinfosecfoundation.org)
And keep in mind that we deprecate filestore v1, see https://suricata-ids.org/about/deprecation-policy/ so I only tested it with filestore v2.
Hi Andreas. I created the following bug. https://redmine.openinfosecfoundation.org/issues/3809
Thank you.