It took me a little bit to ‘mod’ in suricata-update to replace OPNSense’s update/policy rule management - which is nice, but, I want to modify my rules now (change any → $EXTERNAL_NET and such).
Success! Disabled their cron entry, added my own Web GUI cron entries to run a few Shell scripts to run ‘suricata-update’ or copy over some files that OPNSense tends to replace (a very important custom.yaml to get the rest of the functionality and a rule file list)
It seems that “suricata-update” runs Python in a single process/single thread and it takes a very long time (sometimes 30+ minutes) on a decent multiprocessor host running OPNsense 24.7.8-amd64 / FreeBSD 14.1-RELEASE-p6 (python 3.11).
Are there plans to speed up/multiprocess/thread the “suricata-update” script/project? As a python script writer I would love to help improve but do not want to interrupt flow.
Thank you for all suggestions/replies, I want to help and am not trying replicate Aristole2 or any other feature, just interested in how we can leverage modern Python3 and MP/Threads to speed up what amounts to a lot of data analysis.
Not personally involved with Suricata-update works, but if I recall correctly, I think that during last SuriCon, at some point, @ish mentioned that there are plans of moving some of our currently-Python-tools to Rust. Not sure if SU would be covered by this in the near future, but thought it would be worth mentioning that we may be moving away from Python.
I think you’ll have to include more like your full configuration, etc. For example, on my nearly 15 year old home firewall, suricata-update --no-test --no-reload --force which pulls down 182000+ rules, does some disables and modifies completes in 42s. Its slightly longer than the test which is having Suricata load the rules, but still acceptable.
Typically this isn’t an issue we hear about, so there might be something in your configuration that is causing the slow down that would be worth investigating.
suppress gen_id 0, sig_id 0, track by_dst, ip 10.30.0.8
suppress gen_id 1, sig_id 0, track by_dst, ip 10.30.0.8
/root/suricata/homelab.rules (converted from threshold, and convert/testing these on the ‘suricata.yaml’/‘custom.yaml’ netmap bpf line):
alert ip 10.30.14.0/24 any -> any any (msg:"Network is life - Network"; bypass; sid:1000000; rev:1;)
alert ip 10.30.5.5 any -> any any (msg:"Network is life - Host"; bypass; sid:1000001; rev:1;)
alert ip 10.30.5.4 any -> any any (msg:"Network is life - Host"; bypass; sid:1000002; rev:1;)
alert ip 10.30.5.3 any -> any any (msg:"Network is life - Host"; bypass; sid:1000003; rev:1;)
alert ip 10.30.10.16 any -> any any (msg:"Network is life - Host"; bypass; sid:1000004; rev:1;)
alert ip 10.30.10.1 any -> any any (msg:"Network is life - Host"; bypass; sid:1000005; rev:1;)
alert ip 10.30.10.36 any -> any any (msg:"Network is life - Host"; bypass; sid:1000006; rev:1;)
alert ip 10.30.1.3 any -> any any (msg:"Network is life - Host"; bypass; sid:1000007; rev:1;)
alert ip 10.30.10.62 any -> any any (msg:"Network is life - Host"; bypass; sid:1000008; rev:1;)
alert ip 10.30.1.33 any -> any any (msg:"Network is life - Host"; bypass; sid:1000009; rev:1;)
26/11/2024 -- 14:07:14 - <Info> -- Loading /root/suricata/update.yaml
26/11/2024 -- 14:07:14 - <Info> -- Using /usr/local/share/suricata/rules for Suricata provided rules.
26/11/2024 -- 14:07:14 - <Info> -- Found Suricata version 7.0.7 at /usr/local/bin/suricata.
From modify.conf:
- https://www.snort.org/rules/snortrules-snapshot-29111.tar.gz?oinkcode=aaaaaaaaaaaaaa(redacted)aaaaaaaaaaaaaaaaaaaaaa
Local files/directories:
- /root/suricata/homelab.rules
Enabled sources:
- abuse.ch/feodotracker
- malsilo/win-malware
- abuse.ch/urlhaus
- aleksibovellan/nmap
- et/open
- oisf/trafficid
- pawpatrules
- tgreen/hunting
- abuse.ch/sslbl-c2
- abuse.ch/sslbl-ja3
- ptrules/open
- etnetera/aggressive
- abuse.ch/sslbl-blacklist
- stamus/lateral
A Shell script named “suricataupdate.sh” will start the suricata-update process and output the standard out and error out to the log file ‘suricata-update.log’, thus this command gives an idea about run time:
Output from head suricata-update.log && tail suricata-update.log && grep Disabling suricata-update.log | wc -l:
26/11/2024 -- 14:54:16 - <Info> -- Loading /root/suricata/update.yaml
26/11/2024 -- 14:54:16 - <Debug> -- Setting configuration value subcommand -> update
26/11/2024 -- 14:54:16 - <Debug> -- Setting configuration value verbose -> True
26/11/2024 -- 14:54:16 - <Debug> -- Setting data directory to /usr/local/etc/suricata
26/11/2024 -- 14:54:16 - <Debug> -- Setting configuration value config -> /root/suricata/update.yaml
26/11/2024 -- 14:54:16 - <Debug> -- Setting configuration value suricata-conf -> /usr/local/etc/suricata/suricata.yaml
26/11/2024 -- 14:54:16 - <Debug> -- Setting configuration value suricata -> /usr/local/bin/suricata
26/11/2024 -- 14:54:16 - <Debug> -- Setting configuration value version -> False
26/11/2024 -- 14:54:16 - <Debug> -- Setting configuration value show-advanced -> False
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2053531] ET TA_ABUSED_SERVICES Observed Commonly Actor Abused Online Service Domain (data-seed-prebsc-1-s2 .binance .org in TLS SNI)
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2051746] ET TA_ABUSED_SERVICES Observed Commonly Actor Abused Online Service Domain (egnyte .com in TLS SNI)
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2051743] ET TA_ABUSED_SERVICES DNS Query to Commonly Actor Abused Online Service (egnyte .com)
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2053729] ET TA_ABUSED_SERVICES Commonly Actor Abused Online Service Domain (cdn .ethers .io)
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2053730] ET TA_ABUSED_SERVICES Observed Commonly Actor Abused Online Service Domain (cdn .ethers .io in TLS SNI)
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2046653] ET TA_ABUSED_SERVICES Commonly Abused File Sharing Domain (wasabi .com) in DNS Lookup
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2052706] ET TA_ABUSED_SERVICES Observed Abused File Sharing/CRM Platform (pipedrive-files-*-pipedrive .com .s3 .* .amazonaws .com) in TLS SNI
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2052634] ET TA_ABUSED_SERVICES DNS Query to Abused File Sharing/CRM Domain (flg .to)
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2052635] ET TA_ABUSED_SERVICES DNS Query to Abused File Sharing/CRM Domain (getflg .com)
26/11/2024 -- 15:26:09 - <Debug> -- Disabling: [1:2052636] ET TA_ABUSED_SERVICES Observed Abused File Sharing/CRM Platform (flg .to in TLS SNI)
5382
Scripts:
/root/suricataupdate.sh
#!/bin/sh
# Get current date and time
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
# Check if I am already running
if [ ! -e /root/suricata/inRunSuricataUpdate ]; then
touch /root/suricata/inRunSuricataUpdate
else
echo "$TIMESTAMP: I am already running... exiting."
exit 0 # Exit with a 0
fi
# Define file paths
ROOT_CUSTOM1="/root/suricata.rules"
SURICATA_CUSTOM1="/usr/local/etc/suricata/opnsense.rules/suricata.rules"
echo "$TIMESTAMP: Checking for rule updates..." > /root/suricatarules.log
script_name="rule-updater.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarules.log
exit 0 # Exit with a 0
else
echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatarules.log
fi
script_name="installRules.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarules.log
exit 0 # Exit with a 0
else
echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatarules.log
fi
cp /usr/local/etc/suricata/opnsense.rules/suricata.rules /root/suricata.rules
suricata-update --config /root/suricata/update.yaml --suricata-conf /usr/local/etc/suricata/suricata.yaml --suricata /usr/local/bin/suricata --data-dir /usr/local/etc/suricata --threshold-in=/root/suricata/threshold.in --threshold-out=/usr/local/etc/suricata/threshold.config --output /usr/local/etc/suricata/opnsense.rules -v --no-test --no-reload 2>&1 | tee /root/suricata-update.log
RESTART_NEEDED="NO"
# Check if files are identical
if cmp -s "$ROOT_CUSTOM1" "$SURICATA_CUSTOM1"; then
echo "$TIMESTAMP: $ROOT_CUSTOM1 Files are identical." >> suricatarules.log
else
echo "$TIMESTAMP: Rules file is different, restarting Suricata" >> /root/suricatarules.log
RESTART_NEEDED="YES"
fi
if [ "$RESTART_NEEDED" == "YES" ]; then
service suricata restart
echo "$TIMESTAMP: Suricata service restarted." >> /root/suricatarules.log
fi
# Removing run check
rm /root/suricata/inRunSuricataUpdate
exit 0
Please let me know if there is another detail / conf / log to add to this collection
The custom.yaml if you are interested in it is in a related blog post here:
and I’m working this all across the OPNSense forum as well since you have to mod the environment just a little to get this level of functionality - the other unrelated shell script that allows this to take affect and keep the ‘custom.yaml’ and the rule-file as well in shape ( thankfully minimal mod necessary due to all the parameters that ‘suricata-update’ takes <3 ):
After reading another post about Suricata rule management, I just replicated the ‘suricata-update’ deployment onto an Ubuntu host (just to see if FreeBSD might be the fly in the ointment) and I’m seeing a similar time frame to finish making the new rules file (otherwise replicating the settings/config level on the OPNSense FreeBSD host as far as the enabled sources and disable/enable/drop/modify conf files and update/suricata/custom yaml files and homelab rules file).
It might be the high volume disable.conf list?
Feels like there should be a way to pythonically pandas / multiprocess / thread that ‘walk’ of disable items.
Otherwise I’m curious if other Suricata / homelab members have shared their confs for disable/enable/drop/modify against which list of enabled rule sources?
Looks like a classic quadratic complexity with the implementation of the disable by sid, and your large set of sids. I have in fix in mind and will hopefully address it soon.
The time appears to happen between the rule disable gid:sid and sid matching, almost specifically in the sid only matches but both seem to take quite some time. It could be regex matches as I have 11 of those, but, the debug output doesn’t do much to mention regex match interaction.
The ‘suricata-output’ command being executed broken down:
This is done because this installation is on an OPNSense router which places/manages files as such. Instead of deviating, I’m leaning in and seeing if I can replace the primary rule management orchestration with suricata-update as the existing OPNSense quite fast feature, does not allow for rule modification which I require (and know how to do). Finally got some of my first modify.conf rules changed - that double \ before $ in match but not output got me lol.
Cannot seem to edit the previous comment/reply where I was detailing the environment/state of my IDS Host, so, I will simply state here that I also figured out that having ‘alert’ on a bypass rule is a bit of odd form and have since made those ‘pass’ rules with the bypass at the end.
homelab.rules:
pass ip 10.30.10.1 any -> any any (msg:"Bypass - Host01"; bypass; sid:1000001; rev:1;)
pass ip any any -> 10.30.10.1 any (msg:"Bypass - Host01"; bypass; sid:1100001; rev:1;)
As someone that has used sets (and pandas) to overcome list/loop long runs, sets are generally amazingly faster (pandas can be great when later you have to make specific ‘cuts’ across the data for reporting or other ‘smart’ features of the script/project later - 200k items updated or filtered in seconds - and you can import/export to json/native-Python objects rather easily).
Haven’t had the chance to use/implement your mod/patch. Is there a way to effectively ‘install/get’ your mod/upgrade? If not, I’d like to hard mod it in and test it, just so I can with knowledge comment on your Pull Request because I believe you are addressing the issue the same way I was going to attempt a likely very sizable optimization.
Just want to say thank you and Shivani, very responsive and thorough look at the issue. Looking forward to patch, and release, and eventually an updated python module to get!
Quite an amazing experience for a security researcher attempting bigger and better things!
One area of interest to me is the amount of planned work, and, the area of affected space?
With the disable/enable/drop/modify (staged in that order as I’ve read) if you disable things, then you can effectively re-enable a disabled thing if you enable it in some way via a re/sid match in enable.conf. Which only works so well (intimate knowledge of SID rules needed to even play).
Example of a re-ordered Stage Run:
Stage 1 - Enable - enable.conf # All
Stage 2 - Disable - disable.conf # Classtype 'policy violation' and other 'noisy' SIDs
Stage 3 - Drop - drop.conf # Set Drop where desired
Stage 4 - Modify - modify.conf # Modify the rules
There’s also the idea that it would quite clever to be able to design “configurable steps - each step w/file” to take on the whole of rule data, such that you could have more than ‘4’ steps…
N Steps Each Assignable to Action Type and File Via ‘update.yaml’:
Something like this… very draft example
Step 01 - Enable - 01-enable.conf # All
Step 02 - Disable - 02-disable.conf # Specific Classtypes or other key+value/regex match
Step 03 - Enable - 03-enable.conf # Specific SIDs you still want
Step 04 - Drop - 04-drop.conf # Set drop on rules
Step 05 - Modify - 05-modify.conf # Modify special needs
^ in above example/concept my thought is you could have more than 5 steps, but, I must admit I’m not there yet in my use of rule update/cultivation yet - goals lol
Additional ideas/wishes:
If there were a few more stats outputs between the steps via debug, one could realize they are disabling 1000 SIDs that are already disabled, and if it could re-output the ‘optimized’ conf files, further if that would work in the ‘runs’ style full run, that would be quite a tool-kit gift to all IDS admins.
Something like this per Source per Action where the source has a rule called Net ABC Trojan (.rules) and we are doing the third step which is an enable step:
If after the affect line it could list the ‘extra’ line items that when attempted ‘had no affect’ as optimizing the suricata-update whole run would be a great bonus for all us looking to remove un-necessary work from our processes. In this example we are doing a Step 03 Enable and this is showing the pre-affect-post.
Had some brainstorming that I was doing on the side, had to share. Again, thank you all for making this and sharing with us.
On the OPNSense I had to: ln -s /usr/local/bin/python3 /usr/local/bin/python
Then get the Git repo: git clone https://github.com/OISF/suricata-update.git
Then the command using the Python Module from the Git Repo: ./suricata-update/bin/suricata-update update --config /root/suricata/update.yaml --suricata-conf /usr/local/etc/suricata/suricata.yaml --suricata /usr/local/bin/suricata --data-dir /usr/local/etc/suricata --threshold-in=/root/suricata/threshold.in --threshold-out=/usr/local/etc/suricata/threshold.config --output /usr/local/etc/suricata/opnsense.rules -v --no-test --no-reload | tee /root/suricataupdate.log
Output (abbreviated…):
3/12/2024 -- 12:58:53 - <Debug> -- This is suricata-update version 1.3.3 (rev: None); Python: 3.11.10 (main, Nov 4 2024, 23:32:15) [Clang 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15
3/12/2024 -- 12:58:53 - <Info> -- Loading /root/suricata/update.yaml
3/12/2024 -- 12:58:53 - <Debug> -- Setting configuration value subcommand -> update
3/12/2024 -- 12:58:53 - <Debug> -- Setting configuration value verbose -> True
3/12/2024 -- 12:58:53 - <Debug> -- Setting data directory to /usr/local/etc/suricata
3/12/2024 -- 12:58:53 - <Debug> -- Setting configuration value config -> /root/suricata/update.yaml
3/12/2024 -- 12:58:53 - <Debug> -- Setting configuration value suricata-conf -> /usr/local/etc/suricata/suricata.yaml
3/12/2024 -- 12:58:53 - <Debug> -- Setting configuration value suricata -> /usr/local/bin/suricata
3/12/2024 -- 12:58:53 - <Debug> -- Setting configuration value version -> False
3/12/2024 -- 12:58:53 - <Debug> -- Setting configuration value show-advanced -> False
3/12/2024 -- 13:00:50 - <Warning> -- Found classification with same shortname "misc-attack", keeping the one with higher priority (3)
3/12/2024 -- 13:00:50 - <Warning> -- Found classification with same shortname "policy-violation", keeping the one with higher priority (3)
3/12/2024 -- 13:00:50 - <Warning> -- Found classification with same shortname "external-ip-check", keeping the one with higher priority (3)
3/12/2024 -- 13:00:50 - <Warning> -- Found classification with same shortname "social-engineering", keeping the one with higher priority (3)
3/12/2024 -- 13:00:50 - <Warning> -- Found classification with same shortname "coin-mining", keeping the one with higher priority (3)
3/12/2024 -- 13:00:50 - <Info> -- Writing /usr/local/etc/suricata/opnsense.rules/classification.config
3/12/2024 -- 13:00:50 - <Debug> -- Recording existing file /usr/local/etc/suricata/threshold.config with hash '72c65f1d1092ac8d96a6f42d0d7abe91'.
3/12/2024 -- 13:00:50 - <Info> -- Generated 0 thresholds to /usr/local/etc/suricata/threshold.config.
3/12/2024 -- 13:00:51 - <Info> -- Skipping test, disabled by configuration.
3/12/2024 -- 13:00:51 - <Info> -- Done.
5381
What used to take a little over an hour (5381 disables, 224 enables, Modified 182679 rules.), takes less than 2 minutes, brilliant update, thank you @ish
