Evebox log realtime

Please include the following information with your help request:

  • Suricata version
  • Operating system and/or Linux distribution
  • How you installed Suricata (from source, packages, something else)

Hi friends the Suricata Community,

I have recently installed Suricata on a small appliance with a xeon server (CPU E3-1245 v3) and 16gb ram running ubuntu 24.04. It is currently set up in IDS mode and it works fine.

The problem I have is that Evebox started capturing data very well, but for two days it has not shown any data, and if I click events → All, it shows events from a day ago and does not show the current events, lastly the access is slow, and if I do a status of the service I see that it is consuming a lot of RAM.

I show some logs from Suricata and Evebox:

suricata log:

{"timestamp":"2024-09-26T13:53:34.796345-0600","flow_id":1134836489579860,"in_iface":"eno2","event_type":"alert","src_ip":"45.57.103.133","src_port":443,"dest_ip":"192.168.13.224","dest_port":63942,"proto":"TCP","pkt_src":"wire/pcap","alert":{"action":"allowed","gid":1,"signature_id":2210020,"rev":2,"signature":"SURICATA STREAM ESTABLISHED packet out of window","category":"Generic Protocol Command Decode","severity":3},"app_proto":"tls","direction":"to_client","flow":{"pkts_toserver":9388,"pkts_toclient":69347,"bytes_toserver":632778,"bytes_toclient":104861047,"start":"2024-09-26T13:50:20.264224-0600","src_ip":"192.168.13.224","dest_ip":"45.57.103.133","src_port":63942,"dest_port":443}}
{"timestamp":"2024-09-26T13:53:34.796345-0600","flow_id":1134836489579860,"in_iface":"eno2","event_type":"alert","src_ip":"45.57.103.133","src_port":443,"dest_ip":"192.168.13.224","dest_port":63942,"proto":"TCP","pkt_src":"wire/pcap","alert":{"action":"allowed","gid":1,"signature_id":2210020,"rev":2,"signature":"SURICATA STREAM ESTABLISHED packet out of window","category":"Generic Protocol Command Decode","severity":3},"app_proto":"tls","direction":"to_client","flow":{"pkts_toserver":9388,"pkts_toclient":69348,"bytes_toserver":632778,"bytes_toclient":104862561,"start":"2024-09-26T13:50:20.264224-0600","src_ip":"192.168.13.224","dest_ip":"45.57.103.133","src_port":63942,"dest_port":443}}
root@sensor:~$ systemctl status evebox
● evebox.service - EveBox Server
     Loaded: loaded (/usr/lib/systemd/system/evebox.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-09-25 21:47:05 CST; 16h ago
   Main PID: 99782 (evebox)
      Tasks: 10 (limit: 18985)
     Memory: 5.0G (peak: 11.2G swap: 39.2M swap peak: 39.2M)
        CPU: 1h 52min 38.724s
     CGroup: /system.slice/evebox.service
             └─99782 /usr/bin/evebox server -c /etc/evebox/evebox.yaml

Monitoring the Evebox “Sqlite” db, the file grows quickly and I’m not sure if this could be the problem. In the Evebox configuration file I have the retention set to 7 days and no more than 20GB.

root@sensor:/var/lib/evebox$ du -h events.sqlite
39G	events.sqlite

any ideas suggestions? , the software I’m using is:

  • EveBox 0.18.2
  • Suricata version 7.0.6
  • Ubuntu 24.04

Given the presence of these alerts:

{"timestamp":"2024-09-26T13:53:34.796345-0600","flow_id":1134836489579860,"in_iface":"eno2","event_type":"alert","src_ip":"45.57.103.133","src_port":443,"dest_ip":"192.168.13.224","dest_port":63942,"proto":"TCP","pkt_src":"wire/pcap","alert":{"action":"allowed","gid":1,"signature_id":2210020,"rev":2,"signature":"SURICATA STREAM ESTABLISHED packet out of window","category":"Generic Protocol Command Decode","severity":3},"app_proto":"tls","direction":"to_client","flow":{"pkts_toserver":9388,"pkts_toclient":69347,"bytes_toserver":632778,"bytes_toclient":104861047,"start":"2024-09-26T13:50:20.264224-0600","src_ip":"192.168.13.224","dest_ip":"45.57.103.133","src_port":63942,"dest_port":443}}
{"timestamp":"2024-09-26T13:53:34.796345-0600","flow_id":1134836489579860,"in_iface":"eno2","event_type":"alert","src_ip":"45.57.103.133","src_port":443,"dest_ip":"192.168.13.224","dest_port":63942,"proto":"TCP","pkt_src":"wire/pcap","alert":{"action":"allowed","gid":1,"signature_id":2210020,"rev":2,"signature":"SURICATA STREAM ESTABLISHED packet out of window","category":"Generic Protocol Command Decode","severity":3},"app_proto":"tls","direction":"to_client","flow":{"pkts_toserver":9388,"pkts_toclient":69348,"bytes_toserver":632778,"bytes_toclient":104862561,"start":"2024-09-26T13:50:20.264224-0600","src_ip":"192.168.13.224","dest_ip":"45.57.103.133","src_port":63942,"dest_port":443}}

I’m going to make a guess that your logs are being flooded with these, perhaps at a rate that EveBox with SQLite can’t cleanup after itself, or process events for.

  • You could run journalctl -xf -u evebox to see if there are any signs in the logs.
  • Add a -v to EveBox command line, restart, check the journal output
  • Also check under the events tab, is the timestamp of the most recent event increasing? But just really behind?

Thanks Jason Ish for your quick response.

the output of this command:

journalctl -xf -u evebox
Sep 26 14:19:51 sensor.domain.com evebox[99782]: 2024-09-26 14:19:51  INFO evebox::sqlite::retention: Events purged in last 60s: 0
Sep 26 14:21:46 sensor.domain.com evebox[99782]: 2024-09-26 14:21:36  INFO evebox::sqlite::retention: Events purged in last 60s: 0
Sep 26 14:23:02 sensor.domain.com evebox[99782]: 2024-09-26 14:23:02  INFO evebox::sqlite::retention: Events purged in last 60s: 0
Sep 26 14:24:11 sensor.domain.com evebox[99782]: 2024-09-26 14:24:11  INFO evebox::sqlite::retention: Events purged in last 60s: 0
Sep 26 14:25:11 sensor.domain.com evebox[99782]: 2024-09-26 14:25:11  INFO evebox::sqlite::retention: Events purged in last 60s: 0
Sep 26 14:26:12 sensor.domain.com evebox[99782]: 2024-09-26 14:26:12  INFO evebox::sqlite::retention: Events purged in last 60s: 0
Sep 26 14:27:14 sensor.domain.com evebox[99782]: 2024-09-26 14:27:14  INFO evebox::sqlite::retention: Events purged in last 60s: 0

  • the output of the second command:
Sep 26 14:38:43 sensor.domain.com systemd[1]: Stopping evebox.service - EveBox Server...
░░ Subject: A stop job for unit evebox.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit evebox.service has begun execution.
░░
░░ The job identifier is 267757.
Sep 26 14:38:43 sensor.domain.com systemd[1]: evebox.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit evebox.service has successfully entered the 'dead' state.
Sep 26 14:38:43 sensor.domain.com systemd[1]: Stopped evebox.service - EveBox Server.
░░ Subject: A stop job for unit evebox.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit evebox.service has finished.
░░
░░ The job identifier is 267757 and the job result is done.
Sep 26 14:38:43 sensor.domain.com systemd[1]: evebox.service: Consumed 1h 56min 884ms CPU time, 11.2G memory peak, 39.2M memory swap peak.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit evebox.service completed and consumed the indicated resources.
Sep 26 14:38:43 sensor.domain.com systemd[1]: Started evebox.service - EveBox Server.
░░ Subject: A start job for unit evebox.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit evebox.service has finished successfully.
░░
░░ The job identifier is 267757.
Sep 26 14:38:43 sensor.domain.com (evebox)[170159]: evebox.service: Referenced but unset environment variable evaluates to an empty string: ELASTICSEARCH_URL, EVEBOX_OPTS
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::version: This is EveBox version 0.18.2 (rev: dbf08d0); x86_64-unknown-linux-musl
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::server::main: Using data directory /var/lib/evebox
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::sqlite::connection: Auto-vacuum: 1
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::sqlite::connection: Updating SQLite indexes
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::server::main: FTS enabled: true
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::sqlite::retention: Database retention settings: days=7, size=0
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::server::main: Configuration database filename: "/var/lib/evebox/config.sqlite"
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::geoip: Loaded GeoIP database: /etc/evebox/GeoLite2-City.mmdb: Ok(2024-09-06 11:56:43.0 +00:00:00)
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::server::main: Starting server on 0.0.0.0:5636, tls=true
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::server::main: Using directory /var/lib/evebox for self signed TLS certificate and key files
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::cert: Found existing TLS certificate and key: /var/lib/evebox/cert.pem, /var/lib/evebox/key.pem
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::eve::watcher: Found EVE input file /var/log/suricata/eve.json
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::eve::watcher: Starting EVE processor for /var/log/suricata/eve.json
Sep 26 14:38:43 sensor.domain.com evebox[170159]: 2024-09-26 14:38:43  INFO evebox::eve::processor: Valid bookmark found, jumping to record: 16201498

I’m not sure if this command was executed like this:

evebox server -v -D . --datastore sqlite --input /var/log/suricata/eve.json
2024-09-26 14:49:31  INFO evebox::version: This is EveBox version 0.18.2 (rev: dbf08d0); x86_64-unknown-linux-musl
2024-09-26 14:49:31 DEBUG evebox::server::main: Certificate checks disabled: false
2024-09-26 14:49:31  INFO evebox::server::main: Using data directory .
2024-09-26 14:49:31 DEBUG evebox::sqlite::connection: Opening database ./events.sqlite
2024-09-26 14:49:33  INFO evebox::sqlite::connection: Attempting to enable auto-vacuum
2024-09-26 14:49:39  INFO evebox::sqlite::connection: Auto-vacuum: 1
2024-09-26 14:50:23 DEBUG evebox::sqlite::connection: Result of setting database to WAL mode: Ok("wal")
2024-09-26 14:50:23  INFO evebox::sqlite::connection: Updating SQLite indexes
2024-09-26 14:50:23  INFO evebox::sqlite::connection: Enabling FTS
2024-09-26 14:50:23  INFO evebox::server::main: FTS enabled: true
2024-09-26 14:50:23 DEBUG evebox::sqlite::eventrepo: SQLite event store created: fts=true
2024-09-26 14:50:23  INFO evebox::sqlite::retention: Database retention settings: days=7, size=0
2024-09-26 14:50:23  INFO evebox::server::main: Configuration database filename: "./config.sqlite"
2024-09-26 14:50:23 DEBUG evebox::sqlite::connection: Opening database ./config.sqlite
2024-09-26 14:50:23 DEBUG evebox::sqlite::retention: Auto-vacuum in mode full, size based retention available
2024-09-26 14:50:23  WARN evebox::server::main: Username/password authentication is required, but no users exist, creating a user
2024-09-26 14:50:23  WARN evebox::server::main: Created administrator username and password: username=admin, password=uXEaQP8stNwx
2024-09-26 14:50:23 DEBUG evebox::geoip: Found geoip database file /etc/evebox/GeoLite2-City.mmdb
2024-09-26 14:50:23  INFO evebox::geoip: Loaded GeoIP database: /etc/evebox/GeoLite2-City.mmdb: Ok(2024-09-06 11:56:43.0 +00:00:00)
2024-09-26 14:50:23  INFO evebox::server::main: Starting server on 127.0.0.1:5636, tls=true
2024-09-26 14:50:23  INFO evebox::server::main: Using directory . for self signed TLS certificate and key files
2024-09-26 14:50:23  INFO evebox::eve::watcher: Found EVE input file /var/log/suricata/eve.json
2024-09-26 14:50:23 DEBUG evebox::server::main: Checking "/root/b264daf6271f51125d20d5a7715e8947.bookmark" for writability
2024-09-26 14:50:23  INFO evebox::eve::watcher: Starting EVE processor for /var/log/suricata/eve.json
2024-09-26 14:50:23  WARN evebox::eve::processor: Fail to load bookmark: EOF while parsing a value at line 1 column 0
2024-09-26 14:50:23  INFO evebox::cert: Created new TLS certificate and key: ./cert.pem, ./key.pem
2024-09-26 14:50:23 DEBUG evebox::server::main: TLS key filename: Some("./key.pem")
2024-09-26 14:50:23 DEBUG evebox::server::main: TLS cert filename: Some("./cert.pem")
2024-09-26 14:50:23 ERROR evebox::server::main: Failed to start TLS HTTP service: Address in use (os error 98)

All events shown in the “events” tab are from one day ago.

I think I will change the box with better hardware, but I have some doubts if something fails in Evebox.

It looks like you somehow have 2 instances of EveBox running given this one from systemd can’t bind to its port. And perhaps the other that is running isn’t configured quite correctly.

Typically I don’t see this when using systemd, unless running one instance in the foreground (in a terminal) for testing.

Kill all instances, and restart the one from systemd and see how it goes.

Jason, I think the error you see there is because when I ran one of the commands, evebox was already running, my mistake!, but the error persists and I have no events in evebox, but if you want I can upload my evebox configuration file.