Suricata 5.0.2 File Extraction

Help
1

Hi, I seem to have an issue with the file extraction functionality. It first started when I saw that Suricata generates a different hash then it shoud. When I looked into it, the fileinfo showed the size of the extracted file is less than the original file and the state says it the extraction is TRUNCATED. I was thinking the problem lies in the stream depth but I’ve had the same issue, even when I set the filestore.steam-depth value to 0 (unlimited).

my suricata.yaml filestore block:
*- file-store:
version: 2
enabled: yes
dir: filestore
write-fileinfo: yes
force-filestore: no
stream-depth: 0
#max-open-files: 1000
#force-hash: [sha1, md5]
xff:
enabled: no
mode: extra-data
deployment: reverse
header: X-Forwarded-For
*
the rule I’m using: alert http any any -> any any (msg:“EXE file claimed”; fileext:“exe”; filestore; sid:3300003; rev:1;)

The main problem is it happens occasionally because sometimes the file is extracted correctly with the right hash, size and a CLOSED state but most of the time not. The version I’ve installed is 5.0.2.

2

Can you post a pcap that demonstrates the issue?

3

Hi, I managed to capture the issue. I downloaded the same file three times total of which the hash is 0f5d4dbbe5e55b7aa31b91e5925ed901fdf46a367491d81381846f05ad54c45e. Note that I had downloaded this file already a couple of times before I managed to simulate the issue and capture it.

link to pcap:
https://hwcloud.yordi.nl/nextcloud/index.php/s/obDGiHsXzxqa8ob

4

Hi,

I’m using 5.0.3; these are the results I get in eve.json. Can you compare these to what you’re seeing?

$ cat /tmp/ll/eve.json|jq -c 'select(.event_type=="fileinfo")'
{"timestamp":"2020-07-06T10:58:16.754239-0400","flow_id":1421729671462264,"pcap_cnt":1888,"event_type":"fileinfo","vlan":[70],"src_ip":"192.168.70.40","src_port":80,"dest_ip":"192.168.80.5","dest_port":60810,"proto":"TCP","http":{"hostname":"192.168.70.40","url":"/executables/7z1900-x64.exe","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","http_content_type":"application/x-msdos-program","http_refer":"http://192.168.70.40/","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":326162},"app_proto":"http","fileinfo":{"filename":"/executables/7z1900-x64.exe","sid":[3300003],"gaps":false,"state":"TRUNCATED","sha256":"ab30795dfc06e3cf98d29cb0fc8a50a10f96862608c48a0e0eb3e0fec29a8685","stored":true,"file_id":2,"size":326162,"tx_id":0}}
{"timestamp":"2020-07-06T10:58:17.112728-0400","flow_id":1455589046140246,"pcap_cnt":5532,"event_type":"fileinfo","vlan":[80],"src_ip":"192.168.70.40","src_port":80,"dest_ip":"192.168.80.5","dest_port":60810,"proto":"TCP","http":{"hostname":"192.168.70.40","url":"/executables/7z1900-x64.exe","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","http_content_type":"application/x-msdos-program","http_refer":"http://192.168.70.40/","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":1447178},"app_proto":"http","fileinfo":{"filename":"/executables/7z1900-x64.exe","sid":[3300003],"gaps":false,"state":"CLOSED","sha256":"0f5d4dbbe5e55b7aa31b91e5925ed901fdf46a367491d81381846f05ad54c45e","stored":true,"file_id":1,"size":1447178,"tx_id":0}}
{"timestamp":"2020-07-06T10:58:23.504821-0400","flow_id":873889413408097,"pcap_cnt":7187,"event_type":"fileinfo","vlan":[80],"src_ip":"192.168.70.40","src_port":80,"dest_ip":"192.168.80.5","dest_port":60813,"proto":"TCP","http":{"hostname":"192.168.70.40","url":"/executables/7z1900-x64.exe","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","http_content_type":"application/x-msdos-program","http_refer":"http://192.168.70.40/","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":128138},"app_proto":"http","fileinfo":{"filename":"/executables/7z1900-x64.exe","sid":[3300003],"gaps":false,"state":"TRUNCATED","sha256":"67117698895d8fa3f0a2e35b3afcf3b642058ffa41b9cb29889683de880a8d97","stored":true,"file_id":3,"size":128138,"tx_id":0}}
{"timestamp":"2020-07-06T10:58:23.805499-0400","flow_id":1369434150086098,"pcap_cnt":11315,"event_type":"fileinfo","vlan":[70],"src_ip":"192.168.70.40","src_port":80,"dest_ip":"192.168.80.5","dest_port":60813,"proto":"TCP","http":{"hostname":"192.168.70.40","url":"/executables/7z1900-x64.exe","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","http_content_type":"application/x-msdos-program","http_refer":"http://192.168.70.40/","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":1447178},"app_proto":"http","fileinfo":{"filename":"/executables/7z1900-x64.exe","sid":[3300003],"gaps":false,"state":"CLOSED","sha256":"0f5d4dbbe5e55b7aa31b91e5925ed901fdf46a367491d81381846f05ad54c45e","stored":true,"file_id":4,"size":1447178,"tx_id":0}}
{"timestamp":"2020-07-06T10:58:31.158284-0400","flow_id":1751469344045415,"pcap_cnt":17757,"event_type":"fileinfo","vlan":[70],"src_ip":"192.168.70.40","src_port":80,"dest_ip":"192.168.80.5","dest_port":60814,"proto":"TCP","http":{"hostname":"192.168.70.40","url":"/executables/7z1900-x64.exe","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","http_content_type":"application/x-msdos-program","http_refer":"http://192.168.70.40/","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":1447178},"app_proto":"http","fileinfo":{"filename":"/executables/7z1900-x64.exe","sid":[3300003],"gaps":false,"state":"CLOSED","sha256":"0f5d4dbbe5e55b7aa31b91e5925ed901fdf46a367491d81381846f05ad54c45e","stored":true,"file_id":5,"size":1447178,"tx_id":0}}
{"timestamp":"2020-07-06T10:58:31.167141-0400","flow_id":1406383754197258,"pcap_cnt":17773,"event_type":"fileinfo","vlan":[80],"src_ip":"192.168.70.40","src_port":80,"dest_ip":"192.168.80.5","dest_port":60814,"proto":"TCP","http":{"hostname":"192.168.70.40","url":"/executables/7z1900-x64.exe","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","http_content_type":"application/x-msdos-program","http_refer":"http://192.168.70.40/","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":1447178},"app_proto":"http","fileinfo":{"filename":"/executables/7z1900-x64.exe","sid":[3300003],"gaps":false,"state":"CLOSED","sha256":"0f5d4dbbe5e55b7aa31b91e5925ed901fdf46a367491d81381846f05ad54c45e","stored":true,"file_id":6,"size":1447178,"tx_id":0}}
5

Hi, actually I’ve been using the ‘write-fileinfo’ option from the file-store module instead of the ‘eve.files’ module but I’m getting the same fileinfo records as you did. The problem here is that the same file is downloaded (the 7zip.exe) but two times the state got TRUNCATED and the sha256 is therefore different. However I dowloaded the file those times exactly the same way as all other times so I do not get why Suricata truncates the file extraction…

6

So with this pcap, 2 out of 6 fileinfo records show “truncated” … that’s the concern?

7

Exactly. The size of those two files are less that the orginal size thus generates a different hash. Now I am wondering why Suricata stops the extraction mid stream and in how I can get the filestore to consistently extract complete files. I was thinking the problem had to lie in the filestore.steam-depth but I configured it to 0 in the suricata.yaml.

8

Can you paste the output of suricata --dump-config|grep stream.reassembly.depth

9

stream.reassembly.depth = 0

10

Hi,

Our runs with your pcap file show these stats

 "stream_depth_reached": 4,
 "reassembly_gap": 2,

This correlates strongly with the observation I made earlier – 2 of the 6 file saves were truncated.

11

@DI_Ralph can you please share how you solved this issue?