The IP rep functionality uses the host
table, which by default has a fairly low memcap. Have you tried increasing that to see if it changes anything?
Hi and thank you for answering.
What a coincidence! I was just checking the following article
Speed
I’ve been playing with data sets of up to a million entries. Loading it takes hardly any time and I’m confident larger numbers will work just fine. The host table just needs bigger memcaps and hash sizes.
and I was wondering if what you suggest could help.
I was about to ask you guys what values would you recommend to start testing?
Current values:
# Host table:
#
# Host table is used by the tagging and per host thresholding subsystems.
#
host:
hash-size: 4096
prealloc: 1000
memcap: 32mb
Thank you in advance!
I’d say double it, test it, repeat, until it works
I’ll start working on it! Question: In each iteration is it advisable to double the values of the three variables or only the first two?
Thanks
The memcap
is the only critical one, but I think its good to double all of them to avoid possible performance issues.
OK, thanks. I’ll share feedback later
I misunderstood that when all are in one file it behaves properly , otherwise not - sorry about this!
Hi,
Just to be in the same page, when all the IPs are in a single list AND there are just a few lines (I don’t know what the limit is), the blocking mechanism works as expected.
But, by mixing all the lists into a single list, without any limits, the number of lines amounts to more than 50,000 and only a few IPs are blocked.
Some updates:
Take for instance IP 95.141.17.10
. If I add that IP to either file /etc/suricata/rules/scirius-iprep.list
or /etc/suricata/rules/iprep/scirius-iprep.list
, as line # 21605 ( 95.141.17.10,31,100
), then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.
Other combination is mixing some blacklists into file /etc/suricata/rules/iprep/test-iprep.list
. Here IP 95.141.17.10
will be in line # 3136 ( 95.141.17.10,31,100
). Then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.
However, if I add same IP to file /etc/suricata/rules/iprep/test-iprep.list
, as line # 12 ( 95.141.17.10,31,100
), then restart Suricata and test access from TEST PC behind Suricata, the IP is successfully blocked.
I’m still modifying variables under Host Table
to see if with higher values would help IPREP mechanism to work properly.
Current values are:
# Host table:
#
# Host table is used by the tagging and per host thresholding subsystems.
#
host:
hash-size: 4194304
prealloc: 1024000
memcap: 16384mb
Do you advise to keep increasing (doubling) all values or should I stop and try something else? Variable memcap
just reached 16GB, but the issue has not been resolved yet, nor I can notice any other impact in Suricata.
Thank you
The amount of space used by memcap
is taken from HDD or from RAM?
RAM while Suricata is running.
Thank you @syoc . That’s was I was suspecting, since now I see the swap and also CPU increasing consumption for suricata process.
Current values for Host table
are the following:
host:
hash-size: 536870912
prealloc: 131072000
memcap: 2097152mb
After last restart Suricata is not coming up yet (got stuck). I am not an expert, but for me, the above values are huge already. I am not sure if @vjulien agree with me. Current config it is starting to impact the machine performance.
I think I have reached the limit. Perhaps it is time to start trying something different.
Any recommendations @vjulien ? If you need more details, please let me know.
Thank you
Another update
Currently using only one categories file and two reputation files
# IP Reputation
reputation-categories-file: /etc/suricata/rules/scirius-categories.txt
default-reputation-path: /etc/suricata/rules/iprep/
reputation-files:
- scirius-iprep.list
- test-iprep.list
I mixed all blacklists into test-iprep.list
, having around 50500 IPs in the list. List includes the IPs I used as small sample in previous post. I am including their line numbers
10067 176.221.42.32,31,100
6326 113.212.69.128,31,100
4177 108.62.59.27,31,100
3370 95.141.17.244,31,100
33835 216.151.137.155,36,100
32561 173.234.225.161,36,100
30574 108.62.56.222,36,100
30106 95.141.17.10,36,100
None of these IPs are being blocked by Suricata. Same result if I leave all blacklists separated in individual iprep lists.
However if I move these IPs next to the top in the same list, then restart Suricata, the same IPs are being blocked as expected. Again, I am not an expert, but it looks to me that Suricata is not reading/loading/processing the entire list.
After few tests I have narrowed the working range up to 1170 lines of IPs within the list. If IP falls beyond that point, it won’t be blocked. That’s not a large number of IPs to check.
Unfortunately solution suggested by Victor Julien did not work.
In this article, somebody was testing iprep with data sets up to million entries with positive results! Having a small list with only 1170 entries vs a million entries working fine, makes me think that perhaps Suricata is not reading values properly from the Host table
or Host table
mechanism is not working properly. Is there a way to check the host table
or a command to force loading data (other that USR2
or restart Suricata)?
Any other ideas or anything else I could check or try?
Thanks
Hi,
While you guys are still researching what could be wrong with IPREP in my Suricata, I would like to try a parallel approaching so I can keep moving forward in my project.
I have been reading about datasets
and apparently I could achieve the same goal (block many IPs given one or several blacklists) with a mechanism similar to IPREP. The problem is that after reading the official documentation I still have a lot of questions and I would appreciate if somebody could give me a hand and some clues.
I have configured a dataset
in /etc/suricata/selks6-addin.yaml
. I think dataset
keyword is easier than datarep
. Please check and comment what I have done so far:
datasets:
defaults:
memcap: 100mb
hashsize: 2048
bad-ip :
type: string
state: bad-ip.lst
Also I am not sure about the fields for the rules, but I want to build something similat to IPREP. Please check and comment what I have done so far:
From original doc …
alert http any any -> any any (msg: “http user-agent test”; http.user_agent; dataset:set,ua-seen; sid:234; rev:1;)
Customizing …
drop ip any any -> any any (msg:"TEST Bad IPs"; dataset:isset, bad-ip; sid:10; rev:1;)
I’m not sure what other values I can use for the field where dns.query
is, so I am just guessing ip
. Not sure if this make sense.
If somebody could please help me with these settings.
Thank you in advance
I have been reading a lot about dataset
keyword in Suricata, but I’m still unable to configure it properly.
I have configured a dataset
in /etc/suricata/selks6-addin.yaml
.
datasets:
defaults:
memcap: 100mb
hashsize: 2048
test-badip :
type: string
state: test-badip.lst
Restarted Suricata
And also I have created two signatures:
dataset
alert ip any any -> any any (msg:"TEST Bad IP"; dataset:isset,test-badip; sid:10; rev:1;)
GeoIP
alert ip any any -> any any (msg:"TEST Bad GeoIP"; geoip:any,RU,CN,KR,KP,UA; sid:11; rev:1;)
Both signatures are rejected by Suricata when attempt to load them.
- Where can I get a list of available “sticky buffers” that I can use in the
dataset
signature (specially to get the IP address for the signature)? - I have tried different combinations for
geoip
and I have compared my rule against some examples I have found on the Net. I can’t find what it’s wrong with mygeoip
signature.
Can somebody please help me whit this?
Thanks
Try running Suricata with --list-keywords
. Then check the documentation for the keyword you want to use to see if it is sticky buffer. Stick buffer keywords are used before the corresponding content match.
I do not think that dataset feature can be used with IP matching, like you are trying to do.
IPrep is really the way to go, honestly not sure why it’s not working with a not very large list.
As for the GeoIP issue, can you post the message suricata prints when rejecting the geoip signatures?
Hi @syoc ,
Thank you for answering. At this point I have read a lot of documentation (including the official doc), but there are parts or details that are not clear to me. I wish I could do it on my own, but so far I keep failing. That is why I am asking for help here.
Having IPREP working would be great! I do think it is the right engine to be used. I have tried everything I can, with many different combinations and also tried recommendations from Suricata and SELKS teams, but it looks like there is a bug and now I am stuck. So I decided to try Peter’s advice from here:
Apparently datasets
can be used for the same purpose. The mechanism is similar to IPREP. So I am trying hard to have it operational, but again I’m facing errors due to my lack of knowledge.
Here are the last errors I got. Please remember that I am running Suricata as part of SELKS suite, so I must enter rules via Scirius or I might lose them after a Scirius restart.
* **SC_ERR_DATASET** : dataset test-badip not defined
* **SC_ERR_INVALID_SIGNATURE** : failed to set up dataset 'test-badip'.
* **SC_ERR_INVALID_SIGNATURE** : error parsing signature "alert ip any any -> any any (msg:"TEST Bad IP"; ipv4.hdr; dataset:isset,test-badip; sid:10; rev:1;) "
* **SC_ERR_INVALID_SIGNATURE** : error parsing signature "alert ip any any -> any any (msg:"TEST Bad GeoIP"; geoip:RU,CN,KR,KP,UA; sid:11; rev:1;)"
I also tried to add a dataset
as described here, from another Suricata forum post, but failed as well.
* **SC_ERR_DATASET** : dataset dns-seen not defined
* **SC_ERR_INVALID_SIGNATURE** : failed to set up dataset 'dns-seen'.
* **SC_ERR_INVALID_SIGNATURE** : error parsing signature "alert dns any any -> any any (msg: "dns list test"; dns.query; dataset:isset,dns-seen; sid:123; rev:1;)"
I really appreciate if you or somebody else could help me.
Thank you
Ah, never knew about the ipv4.hdr buffer. Very interesting.
The issue, however is that you are matching against the raw ipv4 header.
This means that the source and destination IP is not represented as a plaintext IP.
If you want a rule to match traffic towards 8.8.8.8 for instance you need a rule with ipv4.hdr; content:"|08 08 08 08|";
as you are matching the binary content directly.
Can you upload you iprep list that’s not working somewhere?