Hi, I am currently using the Suricata socket client (suricatasc) to send pcap files to Suricata. Here is how my eve.json file is configured in suricata.yaml:
Is my eve.json configuration the recommended way to create a separate eve.json file for each pcap? I was thrown off because it seems that the Suricata is ignoring the rotate-interval: minute value. A new eve.json file was not created to every minute, but after ever new pcap file was processed.
Hi,
If you want one eve.json file for each pcap then your current configuration might not work.
This is because processing a pcap might take multiple minutes, the rotate interval will kick in and you will get multiple eve.json files for one pcap.
As for the files being created after each pcap, I do not know the details but my experience is that Suricata will flush some state between each pcap file when processing them in “continuous mode”. That is when Suricata is not stopped between each pcap.
Rule profiling data is for example printed when each pcap file is completed in continuous pcap mode, but will only be printed when Suricata is stopped if it’s listening to an interface. Something similar might be happening with the eve.json log files.
In that case, I might just remove rotate-interval. Hopefully that works.
Suricata will flush some state between each pcap file when processing them in “continuous mode”. That is when Suricata is not stopped between each pcap.
I’m not sure if this affects me, I’m not running Suricata with the pcap-file-continuous option, instead I am having a separate tool explicitly send pcap files to Suricata by using suricatasc with the pcap-file command.
When you say, “flush some state.” does that affect the output found in an eve.json alert?
I was just doing some high level reasoning based on my own experience feeding multiple pcaps to suricata (using the same suricatasc method as you) vs running suricata on an interface.
I have not looked at the source code and I and do not know the details about what the differences are between the modes of operation.
Is there a way to know whether Suricata is finished writing to an eve.json file?
I’m asking because I’d like to eventually remove the file once Suricata is finished writing to it and a script has finished processing the eve.json file. I don’t want to start processing or remove the eve.json file before Suricata is finished with it.
Could you explain more about your use case? While Suricata is running and processing network packets, it will write to the eve.json file according to the configuration.
You can configure what goes into the eve.json file –
Log entries for flows, dns, and other traffic. In this way, Suricata acts as a network security monitor and creates log entries according to the traffic it inspects
Alerts triggered by the rules in use.
Statistics.
Each of these can be independently enabled or disabled. You can also have multiple entries with logging information and direct each to a separate file (or redis or syslog).
Suricata doesn’t have builtin mechanisms to interact with other software to synchronize writes to eve.json but there might be other ways to achieve the goal you’re looking for.
I’m using Suricata to generate alerts which are fed into an external program for normalization. I’m trying to figure out the best way to do step 3 below. How can I know that Suricata is done processing a particular pcap file? Should I follow a different approach perhaps by having Suricata write using a unix_stream? I figured using plain ASCII files on the filesystem would be easier.
A program sends a pcap file to Suricata which is running in --unix-socket mode
Suricata writes to a eve.json file(s). 3. An external program extracts the fields it needs fromthe eve.json file(s). Then converts into a pipe separate file.
The pipe separated file is sent to yet another tool for further processing.
I am aware of the output directory argument for alerts. But, I’m not sure how to know when Suricate is finished writing to, in your example, /tmp/file1.output.
Do you have any off the cuff suggestions to achieve this?
To be clear, my worry is that I process /tmp/file1.output before Suricata is finished writing to it.
Unfortunately, I don’t think I can use --pcap-file-continuous, as another tool is in charge of cleaning up the the PCAPs. Basically, Suricata, along with a bunch of other tools are inspecting the same PCAP files simultaneously.
Though, using that flag might simplify things, will have to think about it more.
It sounds like we’d have to launch multiple instances of Suricata each in --runmode=single configuration to fit with our current tools and use case. It feels a little unfortunate (because of my weird use case, not you all lol) though since Suricata has multithreading built in.
Here’s a little more about my use case… We have a tool which will split the network traffic into multiple pcaps each belonging to a “stream”. Our tools will only inspect Pcaps belonging to a particular stream. So if there are 3 streams we’d have 3 instances of tool A and 3 instances of tool B.
But, I think --runmode=single sounds is the most straightforward solution for my use case and to fit in with our other tools… Thanks for the help, Jeff.