Logrotate problem whith Suricata 6 in Docker container

Help
1

Hi all !

My Suricata 6.0 is running in a Docker container. It works perfectly but i have a problem with the logrotate.
The first time, it works perfectly. The file eve.json-20201128 has been created and a new eve.json has
been created too and this new file is filled correctly
The second times, the file eve.json-20201129 has been created with a new eve.json.
The problem is that the eve.json remains at zero and the old file eve.json-2020112 is filled

I have tried a lot of configuration and it’s the same every time

Here is my suricata file in /etc/logrotate.d/

/var/log/suricata/.log /var/log/suricata/.json
{
daily
missingok
rotate 45
compress
create
sharedscripts
postrotate
/usr/bin/docker exec id_container suricatasc -c reopen-log-files
endscript
}

Do you have an idea on how resolve this problem ?

Thanks

2

What happens when you execute:

/usr/bin/docker exec <id_container> suricatasc -c reopen-log-files

on the command line? Normally it should output something like:

$ docker exec 3fd5772850b1 suricatasc -c reopen-log-files
{"message": "done", "return": "OK"}

Something else you can also do is start suricata with -vvv. Its not a good log message, but you should see this everytime suricatasc -c reopen-log-files is executed:

30/11/2020 -- 21:20:39 - <Info> - Unix socket: lost connection with client

which just means that the suricatasc client has disconnected.

3

Thank you for your quick reply.

Here is the result.

docker exec ee4312961f1a suricatasc -c reopen-log-files
{“message”: “done”, “return”: “OK”}

In the /var/log/suricata/suricata.log, i have

30/11/2020 – 22:19:28 - - Unix socket: lost connection with client

I try to force 3 times the logrotate but it’s working at each times with the good files

logrotate -f /etc/logrotate.d/suricata
{“message”: “done”, “return”: “OK”}

Perhaps the problem occurs after a certain period of operation.
I will type your commands when it doesn’t work anymore and I will give you the results

4

This morning I have the problem.
What is strange is that the expected result is not that of my logrotate file
In my suricata file, i want to compress the files. When i force the logrotate, it works. This night, it didn’t work. There was no compression and in addition there is a date in the name of the file.
This night, new files have been created like eve.json, which does not fill and eve.json-20201201 which is filled. My old eve.json has become eve.json-20201201and keep filling up and the new eve.json remains at 0

1 déc. 03:31 eve.json
30 nov. 23:19 eve.json.1.gz
1 déc. 08:45 eve.json-20201201
30 nov. 23:14 eve.json.2.gz
1 déc. 03:31 fast.log
30 nov. 23:15 fast.log.1.gz
1 déc. 08:33 fast.log-20201201
30 nov. 23:07 fast.log.2.gz
1 déc. 03:31 stats.log
30 nov. 23:19 stats.log.1.gz
1 déc. 08:44 stats.log-20201201
30 nov. 23:13 stats.log.2.gz
1 déc. 03:31 suricata.log
30 nov. 23:19 suricata.log.1.gz
1 déc. 06:25 suricata.log-20201201
30 nov. 23:07 suricata.log.2.gz

Now if I place your orders, i have this result

docker exec ee4312961f1a suricatasc -c reopen-log-files
{“message”: “done”, “return”: “OK”}

and everything goes back to normal and the right files fill up

Do you have an idea ?

5

Are you mixing Suricata’s support for time based rotation with logrotate by any chance? Easy to see if you post that section of your config.

I also see that the RPM logrotate configuration has delaycompress, you may want to try adding that, ie)

/var/log/suricata/*.log /var/log/suricata/*.json {
    daily
    missingok
    rotate 5
    compress
    delaycompress
    minsize 500k
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/suricata.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

I believe this removes a possible race condition between rotating the log file, and Suricata re-opening the log files, but it means the most recently rotated log file is not compressed until the next cycle. But its obviously a common enough scenario that logrotate supports this case.

6

I’m not sure to understand what you meant by : Are you mixing Suricata’s support for time based rotation with logrotate by any chance? Easy to see if you post that section of your config.

Currently, I wanted to manage the logrotate from my host machine.
By reading your answer, I may have understood that it was necessary to manage the logrotate in the container.
So, I added it in and we’ll see if it works better.
I don’t know if that’s what you meant ?

Thanks

7

Suricata has the ability to encode the date in the filename of the eve log. When you do this, it will open new log files as the date changes. This is not the default, but I was just checking, as generally if you enable this feature, logrotate is not going to do well.

You should be able to logrotate from the host just fine. Make sure you volume mount the directory, and not just the filename and you should be good. The idea is that logrotate moves the files out of the way to their new name then sends the signal to Suricata somehow, in case of a container the command you used is probably the best.

8

Ok, I didn’t know about this possibility.
On my side, I did not activate it

I use a docker-compose to start my container and I configured it like this:

volumes:
- /var/log/suricata/:/var/log/suricata

Is it good ?

9

Yeah, that should be fine. I have not run for extended periods of time in Docker recently and I don’t know of any issues myself. So best check the output when you see it in this state and report back.

Thanks

10

Here is my feedback

After a few days of testing, I think I have found the solution to my problem. This is probably not the ideal solution but it seems to work fine.
I couldn’t get the logrotate to work on my host machine
I tried to install crond in my container under Alpine to run my logrotate in this one but I encountered a lot of problems.
Finally, I found a solution that currently works well after a few days of operation
I add logrotate in my docker image and I copied this suricata file in /etc/logrotate.d/ of my docker image.

/var/log/suricata/.log /var/log/suricata/.json {
su root root
daily
missingok
rotate 45
compress
sharedscripts
postrotate
suricatasc -c reopen-log-files
endscript
}

on my host machine, I added this line in my crontab -e

00 06 * * * docker exec 19e52111f2e0 logrotate -f /etc/logrotate.d/suricata

The host machine crontab triggers the logrotate in the container and this solution works well for me at the moment.

1 Like
11

This is a fine solution as well. I will be keeping my eye for your original attempt not working as well. But this does the rotate and the signal inside the container, so less chance of Docker getting in the way.

12

Hi all,

Here is a quick report.
Day 33 : It works perfectly !

++

13

Can you mark your solution as a “solution”? It will helps others in the future. Thanks!

14

Ok. It is done @vjulien

15

Just as a reference point. I have encountered the same problem.

Suricata 6.0.2 running in a docker container. Ubuntu 18.04