Suricata no longer write into Eve files

Help
1

Hello everyone,

I searched for a long time a topic that raises the same problem as me but I did not find anything.

I have an ELK SIEM which works fine and I have a Suricata IDS which no longer works. So, what’s the problem :

My Suricata probe is supposed to send data to my Logstash server on 5044 port but actually it send nothing because suricata stopped logging.

First we can see with the stats.log file that suricata is not sleeping :

------------------------------------------------------------------------------------
Date: 5/3/2021 -- 16:07:33 (uptime: 0d, 00h 50m 30s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 30774998
capture.kernel_drops                       | Total                     | 10105
decoder.pkts                               | Total                     | 30767997
decoder.bytes                              | Total                     | 11939861999
decoder.invalid                            | Total                     | 4525
decoder.ipv4                               | Total                     | 30718475
decoder.ipv6                               | Total                     | 2062
decoder.ethernet                           | Total                     | 30767997
decoder.tcp                                | Total                     | 12216582
decoder.udp                                | Total                     | 18484967
decoder.icmpv4                             | Total                     | 9397
decoder.icmpv6                             | Total                     | 30
decoder.avg_pkt_size                       | Total                     | 388
decoder.max_pkt_size                       | Total                     | 1518
flow.tcp                                   | Total                     | 274722
flow.udp                                   | Total                     | 6350263
flow.icmpv4                                | Total                     | 1003
flow.icmpv6                                | Total                     | 15
decoder.ipv4.trunc_pkt                     | Total                     | 4525
decoder.ipv4.opt_pad_required              | Total                     | 1852
tcp.sessions                               | Total                     | 157040
tcp.pseudo                                 | Total                     | 30
tcp.syn                                    | Total                     | 161043
tcp.synack                                 | Total                     | 153828
tcp.rst                                    | Total                     | 158603
tcp.pkt_on_wrong_thread                    | Total                     | 18711
tcp.reassembly_gap                         | Total                     | 1344
tcp.overlap                                | Total                     | 3285886
tcp.insert_list_fail                       | Total                     | 9256
app_layer.flow.http                        | Total                     | 3534
app_layer.tx.http                          | Total                     | 5398
app_layer.flow.ftp                         | Total                     | 227
app_layer.flow.tls                         | Total                     | 2473
app_layer.flow.ssh                         | Total                     | 480
app_layer.flow.smb                         | Total                     | 9939
app_layer.tx.smb                           | Total                     | 413210
app_layer.flow.dcerpc_tcp                  | Total                     | 14573
app_layer.flow.enip                        | Total                     | 27
app_layer.flow.nfs_tcp                     | Total                     | 2
app_layer.tx.nfs_tcp                       | Total                     | 3
app_layer.flow.ntp                         | Total                     | 1740
app_layer.flow.ftp-data                    | Total                     | 310
app_layer.flow.krb5_tcp                    | Total                     | 28923
app_layer.tx.krb5_tcp                      | Total                     | 28998
app_layer.flow.dhcp                        | Total                     | 3697
app_layer.flow.failed_tcp                  | Total                     | 61927
app_layer.flow.dcerpc_udp                  | Total                     | 4
app_layer.flow.dns_udp                     | Total                     | 5802488
app_layer.tx.dns_udp                       | Total                     | 11916424
app_layer.tx.enip                          | Total                     | 54
app_layer.tx.ntp                           | Total                     | 2675
app_layer.flow.krb5_udp                    | Total                     | 689
app_layer.tx.krb5_udp                      | Total                     | 689
app_layer.tx.dhcp                          | Total                     | 10722
app_layer.flow.failed_udp                  | Total                     | 541618
flow_mgr.closed_pruned                     | Total                     | 123215
flow_mgr.new_pruned                        | Total                     | 136772
flow_mgr.est_pruned                        | Total                     | 332789
flow.spare                                 | Total                     | 19
flow.emerg_mode_entered                    | Total                     | 1
flow.tcp_reuse                             | Total                     | 9214
flow_mgr.flows_checked                     | Total                     | 790
flow_mgr.flows_notimeout                   | Total                     | 790
flow_mgr.rows_checked                      | Total                     | 65536
flow_mgr.rows_skipped                      | Total                     | 65318
flow_mgr.rows_maxlen                       | Total                     | 9
tcp.memuse                                 | Total                     | 1146880
tcp.reassembly_memuse                      | Total                     | 4849708
http.memuse                                | Total                     | 72947
ftp.memuse                                 | Total                     | 18970
app_layer.expectations                     | Total                     | 22
flow.memuse                                | Total                     | 33554320

Here is an extract of my suricata.yaml file.

%YAML 1.1
---

#========================== Captures ===========================
af-packet:
  - interface: ens224
    defrag: yes

#=========================== Règles ============================
default-rule-path: /var/lib/suricata/rules
rule-files:
  - suricata.rules
  - pass.rules

#=========================== Ajout unix-command JLB ===========================
unix-command:
enabled: yes
##=========================== Sorties ===========================
default-log-dir: /var/log/suricata
stats:
  enabled: yes
  interval: 30
outputs:
  - eve-log:
      enabled: yes
      filetype: regular
      filename: eve/eve-%d-%m-%Y-%H:%M.json
      rotate-interval: 1h
      json:
        preserve-order: yes
        compact: yes
        ensure-ascii: yes
        escape-slash: yes
      types:
        - alert:
            payload-buffer-size: 4kb
            payload-printable: yes
            packet: yes
            metadata: yes
            http-body-printable: yes
            tagged-packets: yes

  - stats:
      enabled: yes
      filename: stats/stats.log
      append: yes
      totals: yes
      threads: no
logging:
  default-log-level: debug
  outputs:
  - file:
      enabled: yes
      level: info
      filename: /var/log/suricata/suricata.log

  - dns-log:
       enabled: yes
       filename: /var/log/suricata/dns.log
       append: yes

  - http-log:
       enabled: yes
       filename: /var/log/suricata/http.log
       append: yes

And here my filebeat.yml file :

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/suricata/eve/eve-*.json
  fields:
    application: suricata

#============================= Outputs ===================================
output.logstash:
  hosts: ["s-logstash:5044"]
  ssl.enabled: true
  ssl.certificate_authorities: ["/usr/local/share/ca-certificates/ca.crt"]

#============================= Logging ===================================
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  rotateeverybytes: 10485760
  keepfiles: 10
  permissions: 0644

So my filebeat agent sends eve-*.json files to logstash but they are empty.

In suricata.log I have no errors :

3/5/2021 -- 14:50:15 - <Notice> - This is Suricata version 4.1.2 RELEASE
3/5/2021 -- 14:50:15 - <Info> - CPUs/cores online: 2
3/5/2021 -- 14:50:15 - <Info> - eve-log output device (regular) initialized: eve/eve-%d-%m-%Y-%H:%M.json
3/5/2021 -- 14:50:15 - <Info> - stats output device (regular) initialized: stats/stats.log
3/5/2021 -- 14:50:15 - <Info> - 2 rule files processed. 9 rules successfully loaded, 0 rules failed
3/5/2021 -- 14:50:15 - <Info> - Threshold config parsed: 0 rule(s) found
3/5/2021 -- 14:50:15 - <Info> - 9 signatures processed. 0 are IP-only rules, 8 are inspecting packet payload, 5 inspect application layer, 0 are decoder event only
3/5/2021 -- 14:50:16 - <Info> - Going to use 1 thread(s)
3/5/2021 -- 14:50:16 - <Notice> - all 1 packet processing threads, 4 management threads initialized, engine started.
3/5/2021 -- 14:50:16 - <Info> - All AFP capture threads are running.
3/5/2021 -- 14:50:33 - <Notice> - Signal Received.  Stopping engine.
3/5/2021 -- 14:50:34 - <Info> - time elapsed 18.033s
3/5/2021 -- 14:50:34 - <Info> - Alerts: 0
3/5/2021 -- 14:50:34 - <Info> - cleaning up signature grouping structure... complete
3/5/2021 -- 14:50:34 - <Notice> - Stats for 'ens224':  pkts: 181596, drop: 3134 (1.73%), invalid chksum: 0
3/5/2021 -- 15:17:02 - <Notice> - Signal Received.  Stopping engine.
3/5/2021 -- 15:17:02 - <Info> - time elapsed 1965.702s
3/5/2021 -- 15:17:02 - <Info> - Alerts: 0
3/5/2021 -- 15:17:02 - <Info> - cleaning up signature grouping structure... complete
3/5/2021 -- 15:17:02 - <Notice> - Stats for 'ens224':  pkts: 18456374, drop: 2822 (0.02%), invalid chksum: 0
3/5/2021 -- 15:17:03 - <Notice> - This is Suricata version 4.1.2 RELEASE
3/5/2021 -- 15:17:03 - <Info> - CPUs/cores online: 2
3/5/2021 -- 15:17:03 - <Info> - eve-log output device (regular) initialized: eve/eve-%d-%m-%Y-%H:%M.json
3/5/2021 -- 15:17:03 - <Info> - stats output device (regular) initialized: stats/stats.log
3/5/2021 -- 15:17:03 - <Info> - 2 rule files processed. 9 rules successfully loaded, 0 rules failed
3/5/2021 -- 15:17:03 - <Info> - Threshold config parsed: 0 rule(s) found
3/5/2021 -- 15:17:03 - <Info> - 9 signatures processed. 0 are IP-only rules, 8 are inspecting packet payload, 5 inspect application layer, 0 are decoder event only
3/5/2021 -- 15:17:03 - <Info> - Going to use 2 thread(s)
3/5/2021 -- 15:17:03 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.
3/5/2021 -- 15:17:03 - <Info> - All AFP capture threads are running.

When I run the suricata -T command it gives me two warnings :

3/5/2021 -- 16:21:08 - <Info> - Running suricata under test mode
3/5/2021 -- 16:21:08 - <Warning> - [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Invalid logging method: dns-log, ignoring
3/5/2021 -- 16:21:08 - <Warning> - [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Invalid logging method: http-log, ignoring

I think that there is no permissions issue because nothing changed. The root user is running suricata and it can access and write in the /var/log/suricata/eve folder

Sorry, I know I don’t have a lot of information but I’m trying to find some clues to fix this issue.

Thank you in advance. If you need more information ask me.

2

First, please note that Suricata 4.1.2 has been end of lifed. You should consider upgrading to version 6 now.

As for your eve output, I see you are only sending alerts to it. So you need to make sure you have some alert generating traffic. I’d suggest enabling something like flow and/or dns as well, as those will produce records in the absence of alerts.

Also, please check your suricata.yaml against the default. This configuration has dns-log etc under logging, but it belongs under outputs instead. outputs is for data that Suricata produces about the network. logging is about Suricata’s own application log.

1 Like
3

Hello @ish and thank you for your answer !

First, I did the upgrade. I should have paid attention to this before taking the tool back in hand.

Next, I replaced logging options correctly in suricata.yaml comparing with the default configuration file and I have no more errors about this.

For your suggestion, it was right. I didn’t have alert generating traffic. The fact is that the default-rule-path was pointing on a file which was empty. I don’t know when this happened and it looks weird to me. , I hadn’t thought of looking at this file because it hasn’t been modified since installation. :sweat:
So, with a simple ICMP test alert I could see everything was working. I feel stupid …

However, I’m having some new troubles with the different files containing the rules because I have some errors and warnings about their location.

This is no longer the subject of this topic and I already have one of your answers to help me.
(cf. "/var/lib/suricata/rules" directory or "/usr/share/suricata/rules/" directory? - #3 by ish)
I think I’ll be able to fix the problem.

Thanks for your help !