I noticed that there are several ways to store Suricata’s alert output: regular/redis, so I thought of several ways to store alert data in Mysql. Which one is more recommended?
Regular. In regular mode, Suricata will store the alert result in the eve.json file, so I can write a script in the background of the system to automatically read the contents of eve.json into mysql, but this will encounter some performance problems such as eve. The json file is too large, the disk IO problem caused by the large amount of data, the synchronization problem of the read data, and so on.
Redis. In redis mode, Suricata will store the alert results in the redis server, so that I can write scripts to automatically persist Redis data to Mysql. This method also encounters some problems, such as memory usage, data consistency, network bandwidth etc.
Unix-command. I see that the official Suricata documentation also recommends interacting through Unix sockets, setting the unix-command output mode in the Suricata configuration file, and specifying the output to a temporary file, and creating a Python script /path/to/ suricata-unix-command.py, which uses the PyMySQL module to connect to the MySQL database and insert Suricata alarm data into the database, but this will also encounter performance problems: for example, the amount of alert data is too large, causing the python script to Connect to the database many times and so on.
I would like to ask, which method do you recommend? Or is there any other recommended method? Can guarantee low latency and high efficiency of data synchronization?
Looking forward to your reply, sincerely thank you
Although I haven’t measured performance impact and related concerns, it seems that sending the data to a socket (using
unix_dgram (UDP) or
unix_stream (TCP)) could yield the best results.
You’d write the other side of the connection, of course
There are also ways to write output plugins – here’s a few examples:
- GitHub - Center-Sun/suricata-kafka-output: provides a Suricata Eve output for Kafka with Suricate Eve plugin (last updated Nov 2021)
- GitHub - jasonish/suricata-redis-output: Suricata Eve Redis Output Plugin (Jason Ish is a core Suricata developer)
Jason’s is written in Rust and gives an example of how to receive alerts from Suricata and persist them. His example’s using Redis but the same concepts will apply to other storage methods.
If I choose sending the data to a socket (using filetype of unix_dgram (UDP) or unix_stream (TCP)), what contents of Suricata.yaml need to be changed to configure correctly?
In fact, I modified the following two parts of the configuration before, and configured to automatically execute toMysql.py in the directory I specified to write to my mysql, but it didn’t seem to work. I would like to ask if there is a problem with this configuration ? If there is a problem, how should it be configured?
unix-command section is for
suricatasc (suricata socket control) – not for alert output. You can read about suricatasc here
outputs section by
unix_stream). This will instruct Suricata to open a UNIX domain socket with UDP (TCP) if you use
unix_stream). Your program will setup a listening socket to receive requests.
Hi Jeff ，First of all thank you for your kind help
"In the section by set to (or ). This will instruct Suricata to open a UNIX domain socket with UDP (TCP) if you use (). "
What do you mean by the passage you mentioned? I don’t quite understand it, could you elaborate more? Or Could you give an example of how to configure it?What does “set to (or)” mean, and what does “use ()” mean? i’m a little confused
This shows what you’d do for a TCP UNIX domain socket –
outputs.eve-log.filetype = unix_stream
You’ll have to write a program that creates the UNIX domain socket, and then listens for incoming connection requests.
Ok, i got it.It was the question displayed on my side before. I saw that some highlighted parts of your reply were not displayed, so I didn’t understand it. Thank you very much.