How to create a separate eve.json for each processed pcap file?

Hi, I am currently using the Suricata socket client (suricatasc) to send pcap files to Suricata. Here is how my eve.json file is configured in suricata.yaml:

  - eve-log:
      enabled: yes
      filetype: regular
      #filename: eve-alerts-%Y-%m-%d-%H:%M:%S.json
      filename: eve-alerts-%s.json
      rotate-interval: minute
      types:
        - alert:
            payload: yes             
            packet: yes              
            metadata: yes      
            tagged-packets: yes

Currently, a separate timestamped eve.json file is created for each pcap that is sent to Suricata:

eve-alerts-1624916513.json
eve-alerts-1624916515.json

Is my eve.json configuration the recommended way to create a separate eve.json file for each pcap? I was thrown off because it seems that the Suricata is ignoring the rotate-interval: minute value. A new eve.json file was not created to every minute, but after ever new pcap file was processed.

Thanks

Hi,
If you want one eve.json file for each pcap then your current configuration might not work.
This is because processing a pcap might take multiple minutes, the rotate interval will kick in and you will get multiple eve.json files for one pcap.

As for the files being created after each pcap, I do not know the details but my experience is that Suricata will flush some state between each pcap file when processing them in “continuous mode”. That is when Suricata is not stopped between each pcap.
Rule profiling data is for example printed when each pcap file is completed in continuous pcap mode, but will only be printed when Suricata is stopped if it’s listening to an interface. Something similar might be happening with the eve.json log files.

Hey syoc, thanks for your response.

In that case, I might just remove rotate-interval. Hopefully that works.

Suricata will flush some state between each pcap file when processing them in “continuous mode”. That is when Suricata is not stopped between each pcap.

I’m not sure if this affects me, I’m not running Suricata with the pcap-file-continuous option, instead I am having a separate tool explicitly send pcap files to Suricata by using suricatasc with the pcap-file command.

When you say, “flush some state.” does that affect the output found in an eve.json alert?

I was just doing some high level reasoning based on my own experience feeding multiple pcaps to suricata (using the same suricatasc method as you) vs running suricata on an interface.
I have not looked at the source code and I and do not know the details about what the differences are between the modes of operation.

When submitting via pcap-file you can specify an output directory per pcap. That is probably the best way to have each pcap file log to its own file.

Thanks Jason and syoc.

Is there a way to know whether Suricata is finished writing to an eve.json file?

I’m asking because I’d like to eventually remove the file once Suricata is finished writing to it and a script has finished processing the eve.json file. I don’t want to start processing or remove the eve.json file before Suricata is finished with it.

Could you explain more about your use case? While Suricata is running and processing network packets, it will write to the eve.json file according to the configuration.

You can configure what goes into the eve.json file –

  • Log entries for flows, dns, and other traffic. In this way, Suricata acts as a network security monitor and creates log entries according to the traffic it inspects
  • Alerts triggered by the rules in use.
  • Statistics.

Each of these can be independently enabled or disabled. You can also have multiple entries with logging information and direct each to a separate file (or redis or syslog).

Suricata doesn’t have builtin mechanisms to interact with other software to synchronize writes to eve.json but there might be other ways to achieve the goal you’re looking for.

Hey Jeff,

I’m using Suricata to generate alerts which are fed into an external program for normalization. I’m trying to figure out the best way to do step 3 below. How can I know that Suricata is done processing a particular pcap file? Should I follow a different approach perhaps by having Suricata write using a unix_stream? I figured using plain ASCII files on the filesystem would be easier.

  1. A program sends a pcap file to Suricata which is running in --unix-socket mode
  2. Suricata writes to a eve.json file(s).
    3. An external program extracts the fields it needs fromthe eve.json file(s). Then converts into a pipe separate file.
  3. The pipe separated file is sent to yet another tool for further processing.

The pcap-file command can accept multiple arguments including a directory where logs and alerts for the pcap file are stored.

E.g, instead of pcap-file file.pcap add a directory, e.g., pcap-file file.pcap /tmp/file1.output

See 22.2. Suricata Socket Control — Suricata 7.0.0-rc1-dev documentation and 20. Interacting via Unix Socket — Suricata 7.0.0-rc1-dev documentation

I am aware of the output directory argument for alerts. But, I’m not sure how to know when Suricate is finished writing to, in your example, /tmp/file1.output.

Do you have any off the cuff suggestions to achieve this?

To be clear, my worry is that I process /tmp/file1.output before Suricata is finished writing to it.

Appreciate the help.

Are you using --pcap-file-continuous?

If so, when that’s used, the pcap file will be delete when processing has been completed.

Unfortunately, I don’t think I can use --pcap-file-continuous, as another tool is in charge of cleaning up the the PCAPs. Basically, Suricata, along with a bunch of other tools are inspecting the same PCAP files simultaneously.

Though, using that flag might simplify things, will have to think about it more.

Another possibility is to use --runmode=single when launching Suricata – this will guarantee that pcap file processing is done by one thread.

When pcap-file for the second pcap is issued, once that logfile exists you can be certain that the processing of the previous one is complete.

This has performance implications due to the single-threaded processing.

It sounds like we’d have to launch multiple instances of Suricata each in --runmode=single configuration to fit with our current tools and use case. It feels a little unfortunate (because of my weird use case, not you all lol) though since Suricata has multithreading built in.

Here’s a little more about my use case… We have a tool which will split the network traffic into multiple pcaps each belonging to a “stream”. Our tools will only inspect Pcaps belonging to a particular stream. So if there are 3 streams we’d have 3 instances of tool A and 3 instances of tool B.

But, I think --runmode=single sounds is the most straightforward solution for my use case and to fit in with our other tools… Thanks for the help, Jeff.

There’s a console log entry displayed after a pcap file has completed processing.

The message is “pcap file end of file reached (pcap err code

The message is logged to the console and Suricata log but only at log level INFO (which is not always the default).

Perhaps this would help?

Ooo, I actually think that would work really well!

There’s a console log entry displayed after a pcap file has completed processing.

Does this mean that Suricata is finished writing to eve.json for that particular PCAP?

This means that processing for the named pcap file has completed.

Any previous output to eve.json was synchronously written.

Ok, excellent. Thanks again for the help! :grinning: