File extraction based on http/smtp/smb keywords

1

Hello team,

is it possible to combine http/smtp/smb keywords with the filestore ones?

I tried rules like the following without success:
alert http any any → any any (msg:“ignore based on uri”; http.uri; pcre:!“/.*msftncsi.com/”; fileext:“txt”; filestore; classtype:bad-unknown; sid:1; rev:1;)
alert http any any → any any (msg:“grab based on uri”; http.uri; pcre:“/.*msftncsi.com/”; fileext:“txt”; filestore; classtype:bad-unknown; sid:2; rev:1;)

alert http any any → any any (msg:“ignore based on hostname”; http.host; content:!“www.msftncsi.com”; fileext:“txt”; filestore; classtype:bad-unknown; sid:3; rev:1;)
alert http any any → any any (msg:“grab based on hostname”; http.host; content:“www.msftncsi.com”; fileext:“txt”; filestore; classtype:bad-unknown; sid:4; rev:1;)

Without the http. part the following rule works fine:
alert http any any → any any (msg:“grab all txt”; fileext:“txt”; filestore; classtype:bad-unknown; sid:5; rev:1;)

Thanks in advance,
Jiivas

2

What version are you using, how do you run it and how does the config look like?

3

Hello Andreas,

I’m running suricata 6.0.11 in pcap mode. The suricata config is attached.

suricata -c suricata_test.yaml -l log/ -r 2015-02-15-traffic-analysis-exercise.pcap

The pcap contains the windows msftncsi.com/ncsi.txt. I just picked this one for testing.
The pcap contains malware so I don’t want to upload it here but you download it from: hxxps://www.malware-traffic-analysis.net/2015/02/15/index.html

The test rules are the following:

alert http any any → any any (msg:“ignore based on uri”; http.uri; pcre:!“/.*msftncsi.com/”; fileext:“txt”; filestore; classtype:bad-unknown; sid:1; rev:1;)
alert http any any → any any (msg:“grab based on uri”; http.uri; pcre:“/.*msftncsi.com/”; fileext:“txt”; filestore; classtype:bad-unknown; sid:2; rev:1;)
alert http any any → any any (msg:“ignore based on uri”; http.host; content:!“www.msftncsi.com”; fileext:“txt”; filestore; classtype:bad-unknown; sid:3; rev:1;)
alert http any any → any any (msg:“grab based on uri”; http.host; content:“www.msftncsi.com”; fileext:“txt”; filestore; classtype:bad-unknown; sid:4; rev:1;)
alert http any any → any any (msg:“grab all txt”; fileext:“txt”; filestore; classtype:bad-unknown; sid:5; rev:1;)

I expect rules number 2,4 and 5 to hit but only 5 gets a trigger.
Is the combination of http/smtp/smb + filestore invalid or am I missing something else?

Thanks in advance,
Jiivas

suricata_test.yaml (55.1 KB)
test.rules (753 Bytes)

4

Can you try passing -k none to your suricata run command and run again see if any difference ?

5

Hello Peter,

suricata -c suricata_test.yaml -l log/ -r 2015-02-15-traffic-analysis-exercise.pcap -k none

rules:

alert http any any → any any (msg:“detect uri hostname”; http.host; content:“www.msftncsi.com”; classtype:bad-unknown; sid:1; rev:1;)
alert http any any → any any (msg:“grab based on uri”; http.host; content:“www.msftncsi.com”; fileext:“txt”; filestore; classtype:bad-unknown; sid:2; rev:1;)
alert http any any → any any (msg:“grab all txt”; fileext:“txt”; filestore; classtype:bad-unknown; sid:3; rev:1;)

I changed the rules file to test the http parameter without the file keywords and expect 1,2 and 3 to hit but only 1 and 3 are getting a trigger.

Attached is the log
fast.log (327 Bytes)

6

This is just a guess, but I wonder if this has to do with the the direction of flow as it relates to the http.host keyword being a “request” keyword, where as in this case the filestore will be grabbing the file from the response traffic.

@jiivas can you try this:

alert http any any → any any (msg:"msftnsci - hostname (flowbit set)"; flowbits:set,msftnsci; http.host; content:"www.msftncsi.com"; classtype:bad-unknown; sid:1; rev:1;)
alert http any any → any any (msg:"txt from msftnsci (flowbit isset)"; flowbits:isset:msftnsci; fileext:"txt"; filestore; classtype:bad-unknown; sid:2; rev:1;)
alert http any any → any any (msg:"grab all txt"; fileext:"txt"; filestore; classtype:bad-unknown; sid:3; rev:1;)
7

Hi bmurphy,

thanks a lot for this tip. There was a little typo in the second rule “flowbits:isset:msftnsci;” → “flowbits:isset,msftnsci;” but now it works.

I thought that the combination of http + file keywords automatically implies request/response tracking but this doesn’t seem to bee the case.

Since the following rule cannot work, suricata should throw an error message which currently isn’t the case.

alert http any any → any any (msg:“grab based on uri”; http.host; content:“www.msftncsi.com”; fileext:“txt”; filestore; classtype:bad-unknown; sid:2; rev:1;)

8

nice!

I suspect it would have thrown a warning/error if the “flow” option was included in the rule. Keep in mind, that the filestore could also work on POST data which is being uploaded to a server. Without the “flow” keyword, suricata doesn’t know if it should be looking at response or request traffic.

9

Yes, with the last rule here you should have a warning like so in the rules analysis log :

/opt/suritest/bin/suricata -S test.rules  --engine-analysis -l logs/ -T

cat logs/rules_analysis.txt 

-------------------------------------------------------------------
Date: 9/5/2023 -- 22:09:17
-------------------------------------------------------------------
== Sid: 2 ==
alert http any any -> any any (msg:"grab based on uri"; http.host; content:"www.msftncsi.com"; fileext:"txt"; filestore; classtype:bad-unknown; sid:2; rev:1;)
    Rule matches on http host buffer.
    App layer protocol is http_any.
    Rule contains 0 content options, 1 http content options, 0 pcre options, and 0 pcre options with http modifiers.
    Fast Pattern "www.msftncsi.com" on "http host (http_host)" buffer.
    Warning: TCP rule without a flow or flags option.
             -Consider adding flow or flags to improve performance of this rule.
10

Wonder if (just for test) it would filestore files with txt extension if you remove the fileext:txt match from the rule ?
(prob best to try out on pcap first :slight_smile: )

11

try edit use-for-tracking: false in suricata.yaml,