In suricata, does mpm mean that it can generate 2 alerts for 1 packet?
Can I set the detect to stop when a alert occurs while 1 packet is being inspected?
Q2.
If the flow is removed due to timeout or prune, will the memory used by the stream and reassembly engines be returned?
Q3.
My suricata is in a leak of memory in the flow, stream, reassembly and http analysis engines. When the memory reaches memcap, tcp gap occurs, and after a long time, ssn drop occurs, and traffic leakage occurs. My traffic environment is mainly web traffic, and it occurs as low as 5G and as many as 20G.
What settings can be applied to eliminate the memory leak? The settings related to flow are as follows. Should I increase the number of flow recyclers? Any ideas are great, so let me know what settings to try.
First of all what version are you using on which OS?
Some memory will be preallocated but if you have some details about a real memory leak I would argue that this should be reported as a bug.
Ubuntu 18.04, Suricata 6.0.2, 750G memory for test
2 CPUs, 80cores(hyperthreading enabled, each CPU with 20 physical cores)
Napatech 4x10G card(firmware v.3.18.0.34) using 64 Napatech streams.
1GB hostbuffer for each stream but this card can’t use the flow matcher.
64 Cores assigned to Suricata because of the number of Napatech streams.
Problem
Our in/out traffic is up to 20GB/s, mostly http and https traffic, and many other types of protocol traffic occur. And I’m using suricata as IDS in such a condition.
Looking at the graph below, memory is not returned and continues to accumulate. In order to solve this problem, various settings such as short timeout setting, inspection limit of http analysis, depth for bypass, etc. were made, but the leak was not resolved. I couldn’t find any other configuration options that could facilitate memory return.
The configuration file and stat log are attached below.
What happens when you also enable memcap for http and if you drastically reduce the values?
At least for flows you have set 80gb so that’s okay, stream also within, reassembly looks too high.
Is the traffic mostly bidirectional and “clean” so you can except that there is an issue with the traffic that would result in more and more flows being seen but not the end of it for example?
I might suggest moving to 30 Napatech streams (and hence, 30 suricata worker threads) and adjusting the memcap values down. 200GB for reassembly is far too high, imo.
Is the traffic “well-balanced”? To determine if it is, you can use the “profiling” tool (from napatech) that shows traffic distribution to each stream. You should see balanced traffic. “monitoring” will show traffic by port as well as traffic packet size distribution.
@Andreas_Herz
After the memcap is lowered, the reassembly gap increases when the reassembly memory reaches memcap, and the suricata process is terminated. When checking the log, it seems that the abnormal termination of hyperscan is the cause, but more detailed reason could not be found.
suricata.log
{"timestamp":"2021-04-23T00:33:59.962890+0900","log_level":"Error","event_type":"engine","engine":{"error_code":171,"error":"SC_ERR_FATAL","message":"Hyperscan returned error -1"}}
{"timestamp":"2021-04-23T00:33:59.964289+0900","log_level":"Error","event_type":"engine","engine":{"error_code":214,"error":"SC_ERR_NAPATECH_OPEN_FAILED","message":"Napatech Error: NT library is terminating"}}
{"timestamp":"2021-04-23T00:33:59.964327+0900","log_level":"Info","event_type":"engine","engine":{"message":"Failed to read from Napatech Stream 8: NT library is terminating"}}
{"timestamp":"2021-04-23T00:34:00.115033+0900","log_level":"Error","event_type":"engine","engine":{"error_code":219,"error":"SC_ERR_NAPATECH_INIT_FAILED","message":"Napatech Error: NTAPI is terminating"}}
stats info using 32 streams
@Jeff_Lucovsky
This time, I kept the memcap above and cut the number of napatech streams by half (32ea) following your advice. Hash5TupleSorted is being used, and the packet and packet data are properly distributed when checked in profiling.
Obviously, the number of threads decreased and the CPU usage rate increased, but the memory was returned as shown below. Since the memory is operating stably, the number of napatech streams has been increased to 48 and is currently being tested.
Isn’t it better the more napatech streams (=Suricata worker threads) are? Why?
As indicated by the red circle below, why is the reassembly gap memory usage dropping sharply? Is it because the flow engine has been switched to emergency mode?
Yes and no. In my experience, it’s best to provide an isolated core for each Suricata worker thread. Your system has 2 CPU sockets, each with 20 physical cores (plus 20 virtual with hyper-threading enabled).
Each Suricata worker thread will be assigned to a specific Napatech stream (one worker per stream). Napatech supports a max of 128 streams.
You want to ensure that you don’t oversubscribe the CPU resources on your system. Using a physical core and it’s hyper-threaded sibling core, may degrade performance on both cores. I’ve witnessed this effect personally on a system capable of handling 10’s of gigabits/sec of ingest network traffic.
You’ll have to investigate the details of the CPUs to find out how resources are shared between physical and hyper-threaded cores – the important ones are the caches (L3) but all resources would have to be considered.
Thus, my suggestion to use 30 cores, split between the 2 CPU sockets. For best performance, placing the host buffers in the same NUMA node the NIC is attached to will help. The Napatech log files (use ntlog) contain the NUMA information.
And 30 might be 32, or even 34, 36 depending on your traffic level. Each Suricata worker thread can handle a given amount of traffic – so (Total Traffic) / (Worker thread capacity) will approximate the stream count needed for your deployment.
I see the same memuse increase with 6.0.2 but not with 5.0.6 so would be helpful if you could check if it’s NOT happening when you downgrade to 5.0.6 for testing.
