File extraction on FTP, NFS protocol only works sometimes

ellenzhu · October 15, 2021, 8:25am

Hi, I need to extract files from different protocols. The document says that suricata is able to deal with 5 kinds of protocols: http, smb, ftp, nfs, smtp. I tried all of them, but only http and smb work. Suricata 6.0.3 was used in my experiment, and the configuration yaml is as follows:

outputs:
- http
- files
- ftp
…

file-store:
version: 2
enabled: yes # set to yes to enable

Summary

This text will be hidden

  dir: filestore    # directory to store the files
  force-hash: [md5]
  force-filestore: yes # force storing of all files
  stream-depth: 0
  write-fileinfo: yes
  max-open-files: 2000

app-layers:
ftp:
enabled: yes
nfs:
enabled: yes

rules:
alert ftp-data any any → any any (msg:“FTP files”; filestore; sid:3; rev:1;)
alert nfs any any → any any (msg:“File found within NFS and stored”; filestore; sid:34; rev:1;)

I have tried afpacket, pcap and pfring module, but none of them completely extract files successfully. Suricata couldn’t even extract any files sometimes. It seems that pfring and afpacket loss some packets. Please help me with the issue.
My pcap files:
ftp_nta_file_new_zip.pcap (7.4 KB)
nfs-little.pcap (3.0 MB)

suricatalfon · October 15, 2021, 8:49am

Hi,

Could it be that ftpdata_command is missing ?

ellenzhu · October 15, 2021, 9:06am

I tried alert ftp-data any any → any any (msg:“FTP store password”; filestore; ftpdata_command:stor; sid:3; rev:1;) also failed.

suricatalfon · October 15, 2021, 9:44am

Hí,

Well, something is missing. I think you have to specify something else:
fileext
filemagic

Or you must have RUST activated for nfs.

And if not … force-filestore: yes

ellenzhu · October 15, 2021, 10:23am

I enabled rust with “force-filestore:yes” open, but still not work… I am wondering if there were some traffic loss, but why this works on http and smb.

suricatalfon · October 15, 2021, 11:43am

Hí,

With your pcap, meerkat does not extract any files from me. Also, I get pcap error. With Zeek if you extract the file.

Info Suricata:

15/10/2021 -- 13:32:31 - <Notice> - all 3 packet processing threads, 4 management threads initialized, engine started.
15/10/2021 -- 13:32:31 - <Info> - Starting file run for ../pcap/4kbDlXiHid9q7AsQlVx0ZOrckQP.pcap
15/10/2021 -- 13:32:31 - <Info> - pcap file ../pcap/4kbDlXiHid9q7AsQlVx0ZOrckQP.pcap end of file reached (**pcap err code 0**)

:Zeek_IDS:

/extract_files$ file extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

binwalk extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Zip archive data, at least v1.0 to extract, name: nta_file/
146           0x92            End of Zip archive

ftp.log Zeek_IDS:

  "ts": "2021-10-13T10:12:42.676111Z",
  "uid": "CM9FwZ2WbgTo3M3o88",
  "id.orig_h": "192.168.179.1",
  "id.orig_p": 60061,
  "id.resp_h": "192.168.179.132",
  "id.resp_p": 21,
  "user": "<unknown>",
  "command": "RETR",
  "arg": "ftp://192.168.179.132/nta_file_new.zip",
  "file_size": 168,
  "reply_code": 150,
  "reply_msg": "Opening BINARY mode data connection for /nta_file_new.zip (168 bytes)."

files.log Zeek_IDS:

"ts": "2021-10-13T10:12:42.677021Z",
  "fuid": "F24qAU19BovcyWnjik",
  "tx_hosts": [
    "192.168.179.132"
  ],
  "rx_hosts": [
    "192.168.179.1"
  ],
  "conn_uids": [
    "Czq0fS2m7wtdatI2T1"
  ],
  "source": "FTP_DATA",
  "depth": 0,
  "analyzers": [
    "MD5",
    "EXTRACT",
    "SHA256",
    "SHA1"
  ],
  "mime_type": "application/zip",
  "duration": 0,
  "local_orig": true,
  "is_orig": false,
  "seen_bytes": 168,
  "missing_bytes": 0,
  "overflow_bytes": 0,
  "timedout": false,
  "md5": "2d6dfbed41e85db491a5e50ada4def28",
  "sha1": "689531c2cab5202bc23588e7796ab01241a00b60",
  "sha256": "eea81758058f63141a520b58d18abe7e21976e0b21f6c9fcaccf278a29ca0fe7",
  "extracted": "extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik",
  "extracted_cutoff": false

.

ellenzhu · October 18, 2021, 3:01am

Zeek didn’t get the file, either, if I understand correctly. Do you mean the ftp pcap file has some problem? Could you please try with the nfs pcap?

suricatalfon · October 18, 2021, 5:10am

Hí,

The pcap should not be fine, but Zeek does extract the file.

“arg”: “ftp://192.168.179.132/nta_file_new.zip”,

  "mime_type": "application/zip",
  "duration": 0,
  "local_orig": true,
  "is_orig": false,
  "seen_bytes": 168,
  "missing_bytes": 0,
  "overflow_bytes": 0,
  "timedout": false,
  "md5": "2d6dfbed41e85db491a5e50ada4def28",
  "sha1": "689531c2cab5202bc23588e7796ab01241a00b60",
  "sha256": "eea81758058f63141a520b58d18abe7e21976e0b21f6c9fcaccf278a29ca0fe7",
  "extracted": "extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik",

extracted": "extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

binwalk extract-1634119962.677021-FTP_DATA-F24qAU19BovcyWnjik

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Zip archive data, at least v1.0 to extract, name: nta_file/
146           0x92            End of Zip archive

ellenzhu · October 18, 2021, 6:59am

Thanks, I will try other ways to extract ftp file. BTY, do you know why Zeek works but meerkat doesn’t work?

suricatalfon · October 18, 2021, 7:12am

It’s not that it doesn’t work, it’s that it works differently. I believe that it extracts if it is part of an alert.

ellenzhu · October 18, 2021, 7:27am

I tried another pcap, adding alert rules. It generated a different number of alerts every time, that also because pcap file’s error? “pcap err code 0” means error?
Here is the rule:alert ftp-data any any → any any (msg:“FTP store password”; filestore; ftpdata_command:stor; sid:3; rev:1;)

suricatalfon · October 18, 2021, 7:51am

With which tool to generate the pcap file?
Suricata version ?
Config file share. (suricata.yaml)

ellenzhu · October 18, 2021, 8:08am

Use vsftpd to send ftp traffic and Wireshake 3.4.8 to get the pcap. Suricata 6.0.3 is used to detect. suricata.yaml (68.3 KB)

suricatalfon · October 18, 2021, 9:51am

files.rules by default everything is commented. Check it.

Although Suricata says:

" - pcap file …/pcap/nfs-little.pcap end of file reached (pcap err code 0)"

It does not appear to be true. The pcap is correct:

pcapfix ../pcap/../pcap/nfs-little.pcap -s -v

[*] Your pcap file looks proper. Nothing to fix!

suricata:


{
  "timestamp": "2021-10-15T10:07:26.894961+0200",
  "flow_id": 1455606137289499,
  "pcap_cnt": 45,
  "event_type": "fileinfo",
  "src_ip": "192.168.179.140",
  "src_port": 80,
  "dest_ip": "192.168.179.1",
  "dest_port": 51344,
  "proto": "TCP",
  "http": {
    "hostname": "192.168.179.140",
    "url": "/var/www/html/nfs",
    "http_user_agent": "Microsoft-WebDAV-MiniRedir/10.0.19042",
    "http_content_type": "text/html",
    "http_method": "PROPFIND",
    "protocol": "HTTP/1.1",
    "status": 405,
    "length": 242
  },
  "app_proto": "http",
  "fileinfo": {
    "filename": "/var/www/html/nfs",
    "sid": [],
    "magic": "HTML document, ASCII text",
    "gaps": false,
    "state": "CLOSED",
    "md5": "e6d28b075d068c95666118478c07fe75",
    "sha1": "c7d3bc24244077b5e999bc5105da1c7162ec9e6f",
    "sha256": "74f2739bc72a3b339d55f24ada476de399190a06b4e527a5dd8c5d308aed6cec",
    "stored": true,
    "file_id": 1,
    "size": 242,
    "tx_id": 0
  }
}
{
  "timestamp": "2021-10-15T10:07:27.441853+0200",
  "flow_id": 1514208818566712,
  "pcap_cnt": 90,
  "event_type": "fileinfo",
  "src_ip": "192.168.179.140",
  "src_port": 80,
  "dest_ip": "192.168.179.1",
  "dest_port": 63611,
  "proto": "TCP",
  "http": {
    "hostname": "192.168.179.140",
    "url": "/var/www",
    "http_user_agent": "Microsoft-WebDAV-MiniRedir/10.0.19042",
    "http_content_type": "text/html",
    "http_method": "PROPFIND",
    "protocol": "HTTP/1.1",
    "status": 405,
    "length": 233

ellenzhu · October 19, 2021, 7:03am

But the pcap is fine…

I use the same yaml run twice, the number of alerts is also different.

Jue · September 12, 2023, 4:00am

I also encountered this problem, can you tell me how you solved it?

Topic		Replies	Views
Suricata-4.1.2 ftp extraction Help file-extraction	14	1729	May 27, 2020
Filestore module doesn't seem to work when using the -r XXX.pcap Help ips , file-extraction , suricata	5	715	January 3, 2022
The file restoration function fails to restore the ftp pcap package Help file-extraction	19	373	August 11, 2023
Suricata 5.0.2 File Extraction Help file-extraction	10	1970	February 3, 2021
Can suricata recognize ftp-data protocols? Rules	10	982	November 18, 2022

File extraction on FTP, NFS protocol only works sometimes

Related Topics