File extraction on FTP, NFS protocol only works sometimes

Hi, I need to extract files from different protocols. The document says that suricata is able to deal with 5 kinds of protocols: http, smb, ftp, nfs, smtp. I tried all of them, but only http and smb work. Suricata 6.0.3 was used in my experiment, and the configuration yaml is as follows:

outputs:
- http
- files
- ftp

file-store:
version: 2
enabled: yes # set to yes to enable

Summary

This text will be hidden

  dir: filestore    # directory to store the files
  force-hash: [md5]
  force-filestore: yes # force storing of all files
  stream-depth: 0
  write-fileinfo: yes
  max-open-files: 2000

app-layers:
ftp:
enabled: yes
nfs:
enabled: yes

rules:
alert ftp-data any any → any any (msg:“FTP files”; filestore; sid:3; rev:1;)
alert nfs any any → any any (msg:“File found within NFS and stored”; filestore; sid:34; rev:1;)

I have tried afpacket, pcap and pfring module, but none of them completely extract files successfully. Suricata couldn’t even extract any files sometimes. It seems that pfring and afpacket loss some packets. Please help me with the issue.
My pcap files:
ftp_nta_file_new_zip.pcap (7.4 KB)
nfs-little.pcap (3.0 MB)

Hi,

Could it be that ftpdata_command is missing ?

I tried alert ftp-data any any → any any (msg:“FTP store password”; filestore; ftpdata_command:stor; sid:3; rev:1;) also failed. :rofl:

Hí,

:man_facepalming:

Well, something is missing. I think you have to specify something else:
fileext
filemagic

Or you must have RUST activated for nfs.

And if not … force-filestore: yes :joy: :joy:

I enabled rust with “force-filestore:yes” open, but still not work… I am wondering if there were some traffic loss, but why this works on http and smb.

Hí,

With your pcap, meerkat does not extract any files from me. Also, I get pcap error. With Zeek if you extract the file.

Info Suricata:

15/10/2021 -- 13:32:31 - <Notice> - all 3 packet processing threads, 4 management threads initialized, engine started.
15/10/2021 -- 13:32:31 - <Info> - Starting file run for ../pcap/4kbDlXiHid9q7AsQlVx0ZOrckQP.pcap
15/10/2021 -- 13:32:31 - <Info> - pcap file ../pcap/4kbDlXiHid9q7AsQlVx0ZOrckQP.pcap end of file reached (**pcap err code 0**)

:Zeek_IDS:

/extract_files$ file extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

binwalk extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Zip archive data, at least v1.0 to extract, name: nta_file/
146           0x92            End of Zip archive

ftp.log Zeek_IDS:

  "ts": "2021-10-13T10:12:42.676111Z",
  "uid": "CM9FwZ2WbgTo3M3o88",
  "id.orig_h": "192.168.179.1",
  "id.orig_p": 60061,
  "id.resp_h": "192.168.179.132",
  "id.resp_p": 21,
  "user": "<unknown>",
  "command": "RETR",
  "arg": "ftp://192.168.179.132/nta_file_new.zip",
  "file_size": 168,
  "reply_code": 150,
  "reply_msg": "Opening BINARY mode data connection for /nta_file_new.zip (168 bytes)."

files.log Zeek_IDS:

"ts": "2021-10-13T10:12:42.677021Z",
  "fuid": "F24qAU19BovcyWnjik",
  "tx_hosts": [
    "192.168.179.132"
  ],
  "rx_hosts": [
    "192.168.179.1"
  ],
  "conn_uids": [
    "Czq0fS2m7wtdatI2T1"
  ],
  "source": "FTP_DATA",
  "depth": 0,
  "analyzers": [
    "MD5",
    "EXTRACT",
    "SHA256",
    "SHA1"
  ],
  "mime_type": "application/zip",
  "duration": 0,
  "local_orig": true,
  "is_orig": false,
  "seen_bytes": 168,
  "missing_bytes": 0,
  "overflow_bytes": 0,
  "timedout": false,
  "md5": "2d6dfbed41e85db491a5e50ada4def28",
  "sha1": "689531c2cab5202bc23588e7796ab01241a00b60",
  "sha256": "eea81758058f63141a520b58d18abe7e21976e0b21f6c9fcaccf278a29ca0fe7",
  "extracted": "extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik",
  "extracted_cutoff": false

.

Zeek didn’t get the file, either, if I understand correctly. Do you mean the ftp pcap file has some problem? Could you please try with the nfs pcap?

Hí,

The pcap should not be fine, but Zeek does extract the file.

“arg”: “ftp://192.168.179.132/nta_file_new.zip”,

  "mime_type": "application/zip",
  "duration": 0,
  "local_orig": true,
  "is_orig": false,
  "seen_bytes": 168,
  "missing_bytes": 0,
  "overflow_bytes": 0,
  "timedout": false,
  "md5": "2d6dfbed41e85db491a5e50ada4def28",
  "sha1": "689531c2cab5202bc23588e7796ab01241a00b60",
  "sha256": "eea81758058f63141a520b58d18abe7e21976e0b21f6c9fcaccf278a29ca0fe7",
  "extracted": "extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik",

extracted": "extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

binwalk extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Zip archive data, at least v1.0 to extract, name: nta_file/
146           0x92            End of Zip archive

Thanks, I will try other ways to extract ftp file. BTY, do you know why Zeek works but meerkat doesn’t work?

It’s not that it doesn’t work, it’s that it works differently. I believe that it extracts if it is part of an alert.

I tried another pcap, adding alert rules. It generated a different number of alerts every time, that also because pcap file’s error? “pcap err code 0” means error?
Here is the rule:alert ftp-data any any → any any (msg:“FTP store password”; filestore; ftpdata_command:stor; sid:3; rev:1;)


With which tool to generate the pcap file?
Suricata version ?
Config file share. (suricata.yaml)

Use vsftpd to send ftp traffic and Wireshake 3.4.8 to get the pcap. Suricata 6.0.3 is used to detect. suricata.yaml (68.3 KB)

files.rules by default everything is commented. Check it.

Although Suricata says:

" - pcap file …/pcap/nfs-little.pcap end of file reached (pcap err code 0)"

It does not appear to be true. The pcap is correct:

pcapfix ../pcap/../pcap/nfs-little.pcap -s -v

[*] Your pcap file looks proper. Nothing to fix!

suricata:


{
  "timestamp": "2021-10-15T10:07:26.894961+0200",
  "flow_id": 1455606137289499,
  "pcap_cnt": 45,
  "event_type": "fileinfo",
  "src_ip": "192.168.179.140",
  "src_port": 80,
  "dest_ip": "192.168.179.1",
  "dest_port": 51344,
  "proto": "TCP",
  "http": {
    "hostname": "192.168.179.140",
    "url": "/var/www/html/nfs",
    "http_user_agent": "Microsoft-WebDAV-MiniRedir/10.0.19042",
    "http_content_type": "text/html",
    "http_method": "PROPFIND",
    "protocol": "HTTP/1.1",
    "status": 405,
    "length": 242
  },
  "app_proto": "http",
  "fileinfo": {
    "filename": "/var/www/html/nfs",
    "sid": [],
    "magic": "HTML document, ASCII text",
    "gaps": false,
    "state": "CLOSED",
    "md5": "e6d28b075d068c95666118478c07fe75",
    "sha1": "c7d3bc24244077b5e999bc5105da1c7162ec9e6f",
    "sha256": "74f2739bc72a3b339d55f24ada476de399190a06b4e527a5dd8c5d308aed6cec",
    "stored": true,
    "file_id": 1,
    "size": 242,
    "tx_id": 0
  }
}
{
  "timestamp": "2021-10-15T10:07:27.441853+0200",
  "flow_id": 1514208818566712,
  "pcap_cnt": 90,
  "event_type": "fileinfo",
  "src_ip": "192.168.179.140",
  "src_port": 80,
  "dest_ip": "192.168.179.1",
  "dest_port": 63611,
  "proto": "TCP",
  "http": {
    "hostname": "192.168.179.140",
    "url": "/var/www",
    "http_user_agent": "Microsoft-WebDAV-MiniRedir/10.0.19042",
    "http_content_type": "text/html",
    "http_method": "PROPFIND",
    "protocol": "HTTP/1.1",
    "status": 405,
    "length": 233

But the pcap is fine…

I use the same yaml run twice, the number of alerts is also different.