Suricata-IDS eating memory!

Help
1

Hello,
My server is so slow and when I want to write something is console then writing is delayed. I think, it is because of Suricata-IDS. I offer some information about my server and I’m thankful if anyone show me some ways to boost is:

# free 
              total        used        free      shared  buff/cache   available
Mem:        3867856     3094076      628904       11392      144876      575532
Swap:       2170876     1382284      788592

And:

# iostat
Linux 4.18.0-193.14.2.el8_2.x86_64 (System) 	10/17/2020 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.17    0.02    0.63    0.03    0.00   97.14

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               3.51        44.39        45.40  203959475  208613480
dm-0              1.44        38.62        33.62  177484677  154477077
dm-1              4.36         5.74        11.69   26358028   53740852
dm-2              0.00         0.01         0.08      52746     354114

And:

# vmstat 
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 1375628 539688     72 224988    1    3    11    11    1    2  2  1 97  0  0

And https://pastebin.fun/lCK8mah605 and https://paste.ubuntu.com/p/2yj2kdM5tB/.

Suricata
Suricata1043×497 82.3 KB

Thank you.

2

It seems it is entering swap, could be because too high memcaps are configured in suricata.yaml for the available 4GB RAM on the machine.
What is the output of

suricata --dump-config |grep mem

If you are running af-packet - how does your af-packet section look in suricata.yaml?

3

The output is:

# suricata --dump-config |grep mem
defrag.memcap = 32mb
flow.memcap = 128mb
stream.memcap = 64mb
stream.reassembly.memcap = 256mb
host.memcap = 32mb
4

What is you start command ? And could you please also share

suricata --dump-config |grep af-packet

if you are using the default settings (af-packet).

As a test run you could also try to start Suricata without rules and see if any difference

suricata ...... -S /dev/null
1 Like
5

Thank you.
Output is:

# suricata --dump-config |grep af-packet
af-packet = (null)
af-packet.0 = interface
af-packet.0.interface = ens192
af-packet.0.cluster-id = 99
af-packet.0.cluster-type = cluster_flow
af-packet.0.defrag = yes
af-packet.1 = interface
af-packet.1.interface = default

I’m using Suricata-IDS in IPS mode.

6

Are you using af-packet for IPS?

7

I set this option:

# Linux high speed capture support
af-packet:
  - interface: ens192
    # Number of receive threads. "auto" uses the number of cores
    #threads: auto
    # Default clusterid. AF_PACKET will load balance packets based on flow.
8

How are you starting Suricata?

9

I read this article: https://kifarunix.com/install-and-setup-suricata-on-centos-8/
Which is wrong?
I used that article because I can’t find any tutorial about configuration Suricata-IDS in IPS mode on CentOS 8.

If you try to test AF-PACKET fanout using tcpreplay locally, please note that load balancing will not work properly and all (or most) traffic will be handled by the first worker in the AF-PACKET cluster. If you need to test AF-PACKET load balancing properly, you can run tcpreplay on another machine connected to your AF-PACKET machine.

10

No, I don’t think its wrong. Previously you were running in nfqueue mode, not af-packet. So I just want to to narrow that down.

1 Like
11

Thus, I must disable this option? If yes, then can it make any problem for Suricata-IDS in IPS mode?

12

If running in NFQUEUE mode (-q 0, etc) the af-packet configuration section is not used, so won’t cause any issues. Thats why its important to tell us if you are in fact using the af-packet input, or at least how you are starting Suricata so we know.

13

My configuration is:

$ sudo cat /etc/sysconfig/suricata 
# The following parameters are the most commonly needed to configure
# suricata. A full list can be seen by running /sbin/suricata --help
# -i <network interface device>
# --user <acct name>
# --group <group name>

# Add options to be passed to the daemon
#OPTIONS="-i ens192 --user suricata "
OPTIONS="-q 0 --user suricata "

Can I disable af-packet: option?

14

You could remove it. But as you are running with -q (nfqueue mode) you are not using af-packet at all. So the af-packet section of the configuration file is completely ignored.

I don’t have any good answers to your memory problem, I just wanted to get clear how it was running before debugging af-packet related issues.

Some quick thoughts tho. You think its Suricata slowing it down. This should be easy enough to test. Disable Suricata and see how things perform.

Suricata doesn’t look to be using excessive amounts of memory, but you do appear to be hitting swap. It could simply be that the machine does not have enough memory to run Suricata plus whatever other application workloads you have on it.

Also note that you are running Suricata in IPS mode on a single host, protecting the host that also has your “application” workload on it. I don’t think we see this scenario very often so you may not see that many responses.

1 Like
15

It was because of pm.min_spare_servers, pm.max_spare_servers and pm.max_children values in /etc/php-fpm.d/www.conf file.