You would run multiple instances of suricata with a load balancer distributing the traffic via sessions into more manageable bites for each suricata worker.
Multiple worker threads on a multi-core box, multiple boxes. Traffic distributed with session tracking round robin over all workers.
Aggregating the data back together would be a similar process
If you have enabled alerting and network metadata extraction, a single Suricata host shouldn’t produce that much data (10GB/s), even if it is reading 10Gbps.
So it will normally be multiple Suricata hosts doing that, and that should not be a problem.
To handle ingestion of alerts at that scale, a kafka cluster can easily handle that and will scale linearly, you will need to tune your logshipper to be able to read eve files fast enough and send them downstream.
With all high-traffic deployments, make sure that the threaded option for eve.json outputs is enabled
# a line based alerts log similar to Snort's fast.log
#filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
# Extensible Event Format (nicknamed EVE) event log in JSON format
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
# Enable for multi-threaded eve.json output; output files are amended with
# an identifier, e.g., eve.9.json
QQ: You want to analyze 100Gbps? if yes then there is an initial DPDK implementation or the other GA way via XDP/eBPF
or you need to store the output/the result(alerts) of eg.(1Tbps IDS analysis) in a manageable way that will (not)likely generate 100Gbps? If you get 10Gbps of eve json(or syslog,kafka,redis,kafka,etc.) alerts then you area already doing something wrong!
Tobias has presented an use case where they need to analyze ~800Gbps but much of the traffic is flow shunted(dropped)
In my case @Uni Bern Switzerland we analyze “only” ~100Gbps and the eve output is ingested(via filebeat/Agent) into our Elastic SIEM. Just for your reference, since last year November I got 433GB of eve logs.
Depending on the Rules used by your Suricata and what happens in your network you might get some more alarms but nowhere near 10Gbps or 100Gbps that would be absurd. If you try to store more packet metadata you might look into https://arkime.com/ (former moloc)
If its the write to disk that is your concern I’d look at the following options:
redis: There is a built-in Redis output that can act like an in-memory queue. You’d need your own application to read from the queue, and it should be able to read from, and process the events as fast as Suricata can write them as you want to avoid Redis from using too much memory where it may need to go to disk.
unix_socket: Again, you need your own app to be the server side fo the unix_socket, and again, you need to process the events as fast as Suricata is publishing them, otherwise you risk back-pressure which could result in Suricata dropping.
Files, either on real disk or ram disk do make a lot of things simpler though, so I’d test to make sure that it really is a point of performance degradation before trying to alternatives.