Help please! File Hash detection issues

Help
1

Ish had been helping me(Komahal) with this in IRC. The first attempt was trying to do this with Windows. Evidently, there’s an issue with hash generation under Windows server. So I rebuilt Suricata under Ubuntu and now CentOS; and I’m seeing a similar issue.
What I am trying to accomplish is for Suricata to generate MD5/SHA1/SHA256 hashes and alert if it finds a matching hash in provided lists. I get threat intel that includes file hashes of known bad files.

So. CentOS v8 on VMWare ESXi 6.7, Suricata 5.0.4 installed from RPM.
From the Suricata.yaml:

- fast:
      enabled: yes
      filename: fast.log
      append: yes
 - eve-log:
      enabled: yes
- files
   force-hash: [md5, sha1, sha256]
- file-store:
      version: 2
      enabled: no
      stream-depth: 0
outputs:
  - console:
      enabled: yes
      # type: json
  - file:
      enabled: yes
      level: info
      filename: suricata.log
af-packet
 - interface: ens224
 - cluster-Id: 99

app-layer
 protocols: 
   libhtp:
   default config:
    personality: IDS
   # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 100mb
           response-body-limit: 100mb

           # inspection limits
           request-body-minimal-inspect-size: 4kb
           request-body-inspect-window: 4kb
           response-body-minimal-inspect-size: 4kb
           response-body-inspect-window: 4kb
Stream:
 memcap: 256mb
 depth: 100mb

the rest is pretty much default.
the three rules I have added for capturing this kind of alert are as follows:
alert http any any -> any any (msg:"Black list checksum match MD5"; filemd5: /var/lib/suricata/hash/hash.md5; classtype: bad-unknown; sid: 2800; rev: 2;)
alert http any any -> any any (msg:"Black list checksum match SHA1"; filesha1: /var/lib/suricata/hash/hash.sha1; classtype: bad-unknown; sid: 2801; rev: 2;)
alert http any any -> any any (msg:"Black list checksum match SHA256"; filesha256: /var/lib/suricata/hash/hash.sha256; classtype: bad-unknown; sid: 2802; rev: 2;)
There are no startup errors related to the hash.*** files, if it couldn’t open them, it would tell me.

I added a specific MD5 hash to the blacklist to attempt to force an alert, so it is in there. all lower case. And when I download the file, in the eve.json, I see:

{**"timestamp":"2020-11-06T16:40:00.917299-0800"**,"flow_id":1493020505547753,"in_iface":"ens224","event_type":"fileinfo","src_ip":"137.82.116.42","src_port":80,"dest_ip":"xx.xx.xx.x","dest_port":50944,"proto":"TCP","http":{"hostname":"mirror.it.ubc.ca","url":"\/centos\/8.2.2004\/AppStream\/x86_64\/kickstart\/Packages\/binutils-devel-2.30-73.el8.i686.rpm","http_user_agent":"Mozilla\/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/86.0.4240.183 Safari\/537.36 Edg\/86.0.622.63","http_content_type":"application\/x-rpm","http_refer":"http:\/\/mirror.it.ubc.ca\/centos\/8.2.2004\/AppStream\/x86_64\/kickstart\/Packages\/","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":1127224},"app_proto":"http","fileinfo":{"filename":"\/centos\/8.2.2004\/AppStream\/x86_64\/kickstart\/Packages\/binutils-devel-2.30-73.el8.i686.rpm","sid":[],"gaps":false,"state":"CLOSED","**md5":"af18dbf9930e4af5a582ce67faeb5248"**,"sha1":"459b763b394e22839711dc62dbed2bb0c5e17a91","sha256":"ebd171abacede5f59e69ee9e6d4afd16fd3b857d8f817d00833c85a289e38fbd","stored":false,"size":1127224,"tx_id":0}}

The hash is there. So that is working.
If I look at the fast.log corresponding to that timestamp, it should be between these two entries:

`11/06/2020-16:39:55.774388  [**] [1:2260002:1] SURICATA Applayer Detect protocol only one direction [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 10.xx.xx.xx:88 -> 10.11.0.128:56756`

11/06/2020-16:40:05.768270 [**] [1:2260002:1] SURICATA Applayer Detect protocol only one direction [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 10.xx.xx.xx:88 -> 10.11.0.128:56776

I’m doing something wrong, obviously, any ideas?

2

From the example here -

"gaps":false,"state":"CLOSED","**md5":"af18dbf9930e4af5a582ce67faeb5248"**,"sha1":"459b763b394e22839711dc62dbed2bb0c5e17a91","sha256":"ebd171abacede5f59e69ee9e6d4afd16fd3b857d8f817d00833c85a289e38fbd","stored":false,"size":1127224,"tx_id":0}

Do the hashes in the log correspond correctly to the file hash ? (just to confirm)

Can you try with just one rule, md5 hash rule match only ? Maybe also enable filestore - to narrow it down.

3

Hi Peter,

Yeah, so I would download the file onto my laptop (off-network) and use PowerShell to get the file hash, add that to the md5.hash file on the Suricata box, and then download the file on a server on a network that Suricata was capturing packets for. Since it’s Linux, I figure it is probably case sensitive, so I make sure the hash that’s put into the blacklist file is lower case.

The md5 hash matches what’s in the eve.json, it’s just not dropping an alert into the fast.log.

I’ll try with filestore and let you know how it goes.
Thanks!

4

Alright. Initial results indicate that enabling the filestore worked.
I guess the only other thing is (this may be by design) if I download file A, generate a hash, delete the file and download it again, Suricata doesn’t seem to record the 2nd download.
We’re not using a proxy or anything. Browser cache?

Thank you very much.

5

Could be BC yes, try it with wget/curl