What to use for logs at high rate 10Gb/s or 100G/s for Eve logs

Help
1

Hi
I want to know what can we use to store logs of eve at a rate of 10G/s , 100G/s
Eve have many output types like regular , syslog , redis also kafka with a plugin.

But regular writes on disk that will effect performance because I/O operations.

Is there any alternative method can be used ?

Thanks

2

Are you assuming that logs at 100GB/s will be produced by a single Suricata host?

Or the cluster of Suricata hosts will produce that sum of 100GB/s?

3

@IDSTower , that will be a future task , But i expect it to handle atleast 10 Gb
using something , like kafka that will save my I/O .
there is only kafka plugin but that is not able to hand even 100mb

4

You would run multiple instances of suricata with a load balancer distributing the traffic via sessions into more manageable bites for each suricata worker.
Multiple worker threads on a multi-core box, multiple boxes. Traffic distributed with session tracking round robin over all workers.

Aggregating the data back together would be a similar process

5

If you have enabled alerting and network metadata extraction, a single Suricata host shouldn’t produce that much data (10GB/s), even if it is reading 10Gbps.

So it will normally be multiple Suricata hosts doing that, and that should not be a problem.

To handle ingestion of alerts at that scale, a kafka cluster can easily handle that and will scale linearly, you will need to tune your logshipper to be able to read eve files fast enough and send them downstream.

6

With all high-traffic deployments, make sure that the threaded option for eve.json outputs is enabled

outputs:
  # a line based alerts log similar to Snort's fast.log
  - fast:
      enabled: yes
      filename: fast.log
      append: yes
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve.json
      # Enable for multi-threaded eve.json output; output files are amended with
      # an identifier, e.g., eve.9.json
      threaded: on
1 Like
7

QQ: You want to analyze 100Gbps? if yes then there is an initial DPDK implementation or the other GA way via XDP/eBPF

or you need to store the output/the result(alerts) of eg.(1Tbps IDS analysis) in a manageable way that will (not)likely generate 100Gbps? If you get 10Gbps of eve json(or syslog,kafka,redis,kafka,etc.) alerts then you area already doing something wrong!
Tobias has presented an use case where they need to analyze ~800Gbps but much of the traffic is flow shunted(dropped)

In my case @Uni Bern Switzerland we analyze “only” ~100Gbps and the eve output is ingested(via filebeat/Agent) into our Elastic SIEM. Just for your reference, since last year November I got 433GB of eve logs.

Depending on the Rules used by your Suricata and what happens in your network you might get some more alarms but nowhere near 10Gbps or 100Gbps that would be absurd. If you try to store more packet metadata you might look into https://arkime.com/ (former moloc)

8

I forgot to say that we write to an very fast NVMe disk

9

If its the write to disk that is your concern I’d look at the following options:

  • redis: There is a built-in Redis output that can act like an in-memory queue. You’d need your own application to read from the queue, and it should be able to read from, and process the events as fast as Suricata can write them as you want to avoid Redis from using too much memory where it may need to go to disk.
  • unix_socket: Again, you need your own app to be the server side fo the unix_socket, and again, you need to process the events as fast as Suricata is publishing them, otherwise you risk back-pressure which could result in Suricata dropping.

Files, either on real disk or ram disk do make a lot of things simpler though, so I’d test to make sure that it really is a point of performance degradation before trying to alternatives.

10

Hi @florinsfetea , is suricata able to write in eve.json at 100Gbps/s , as IO speed may be the issue. Even I tried to write in disk then packets were dropped

11

@Hacker7
it does not necessary need to write @100 Gbps line-rate all packets as only some interesting analyzed packet will generate alerts.