I’m running Suricata in unix socket mode, and submitting PCAP files using pcap-file. I’m trying to figure out the best way to determine if a PCAP file’s processing is finished.
I know I can call pcap-file-list to “display the list of queued files”. So, logically, if my filename is in the files list returned back from pcap-file-list then it is not finished processing.
However, if my filename is not in the pcap-file-list results, can I safely rely on that to mean “yes, it’s done processing?” My current thought is for my check to be “if my filename is NOT in pcap-file-list and the eve.json in the output directory for this PCAP file I’ve specified DOES exist, then the PCAP is finished processing.”
My only concern about this check would depend on when the PCAP’s filename is removed from the pcap-file-list list of queued files. Is it removed at the beginning or the end of processing? Because it it’s at the beginning of processing, then just the existence of eve.json wouldn’t be enough, since it could still be being appended to.
Does that logic make sense for a good check to make sure the PCAP file has finished processing? I want to ensure it’s done before I start processing eve.json.
And for what it’s worth, I did read these forum posts (here and here) but those weren’t particularly helpful, as the suggestions there mostly just recommended reading the documentation (which I’ve done) and doing the PCAPs one at a time (which I don’t think solves the problem).
More specifically, the feedback I’m looking for is on my idea of checking the results of pcap-file-list and the existence of eve.json (I’m specifying a different output directory for each pcap I send with pcap-file) to determine if it’s finished processing or not.
After playing around with it this morning, I don’t think my assumption is going to work. I think that I’m going to have to have another process that tails the suricata instance’s log and looks for the “finished” message that gets written out to it. I don’t like it, but I don’t see anything else that will work.
So it appears that once a pcap file starts being processed it is removed from the list. Which doesn’t appear to be ideal for your scenario. I think it would be more logical to be appear in the list until processing was done. That would be a “breaking” change though and would require consideration.
I was going to suggest so submit pcaps using a blocking call and manage the state from your submitter, but we don’t appear to have a way to wait for the pcap to be finished either, which is probably how I’d want to work with the interface if I had a job sumitting pcaps, I think that would be a valid feature request.
But unfortunately, with the latest release, I think you’ve narrowed in on the issues and a not great work-around.
Thanks @ish , I will log a feature request for a blocking mode (that would work for my scenario) and in the meantime I’ll resort to parsing suricata.log for the message like:
[1449 - RX#01] 2025-01-21 17:55:54 Info: pcap: pcap file /data/pcap/processed/whatever.pcap end of file reached (pcap err code 0)