Rules database connectivity

Developers
1

So I was thinking that suricata could offer a sql database connection to load rules from a database. Just supply the sql connection settings to suricata.yaml file). This could save some time in organising and keeping rules in a searchable format (I also wrote an app that can store, search and edit rules, then dump them as text, so shameless plug here as well…) Anyway python’s suricata-ids idstools can parse rules files into a format easier to work with than a sets of rules files… It would also offer the ability to connect remotely to a centralised rules database for control over what happens there.

In my schema I loaded the ET ruleset into a table like this:

| id | raw | sid | message | comment | enabled | classtype | content | rev

And can search pretty accurately to check what threats are being checked (eg. trojan), but there’s no setting to allow loading these rules into suricata engine…

1 Like
2

Hi,

Using a database and toolset to help manage rules is good but I don’t think that necessarily means suricata should be able to access and load the rules from the database.

However, this is just my opinion. Please feel free to create a feature request at Overview - Suricata - Open Information Security Foundation for broader feedback.

Community involvement in Suricata’s development is encouraged so feel free to create a PR with the changes (and tests!) that demonstrate this feature.

3

Happy to do that. Can you help me though in pointing out where I might start? In src directory there are +1000 files!

4

Perhaps suricata update could have som SQL integration?
Sounds like a much simpler job, though I’m not familiar with the codebase at all.

5

This is also a good idea – IMO, preferable to modifying suricata.

suricata-update is described here – suricata-update - A Suricata Rule Update Tool — suricata-update 1.2.0 documentation

6

Thanks Jeff.

I’d like to configure and make suricata locally before creating a PR with changes. Are there any guides to changing the parameters to configure? I’ve exported LIBS=-lmysqlclient -lpthread -lz -lm -lrt -lssl -lcrypto -ldl -lresolv and LDFLAGS=/usr/lib/x86_64-linux-gnu to the environment, then tried:

./configure --includedir=’/usr/include/mysql’

And it generated makefile for make and make install! However the include mysql.h header isn’t picked up when compiling detect-engine-loader as it should. How can I ensure that the include directories & headers that get included when I run configure?

FYI automake | developers guide

7

I’d like to learn a bit more about how you’d like to Suricata to work with this, but overall I’m inclined to push you towards suricata-update as well. We don’t like extending Suricata with external connections, instead we prefer to extend the unix socket functionality. Then an external tool (such as suricata-update could handle the complexity of talking over the network, etc).

Keep in mind that Suricata’s rules can rely on dataset files, iprep files as well as some other custom file formats. Ideally a solution would be able to support all of these.

8

Sure thing. I mean when it comes to the suricata-update it seems to work well to collect the signatures and load them into suricata. I have encountered situations where there is no real transparency around which SID’s are loaded, or if a threat has been mitigated by a particular rule file, or if the collection of rules includes a particular threat etc. Many situations where different servers require different configurations, and the update utility only collects the bleeding edge so to speak… Generally a good way to handle a tonne of files (in this case rules files) is to use a database for organisation. It just seemed clearer to organise these SIDs in that way, and it kind of saves you having to search through files for particular content, to see if that threat has been caught. I certainly gained a lot of transparency around these rules by building a database that way, being able to search on content, comment, SID, you know, frequency of the alerts etc.

I think it would help the cause to incorporate some kind of database connectivity. IN the least, I’ve already managed to connect to my database and read these lines as if they were from a file, but you know, I’ve been struggling around the build process these last couple of days…

I’ll take your advice and look towards suricata-update. certainly you can still get a great configuration from using this, it’s just, you know, I was scratching my head thinking why you haven’t done this yet…

9

You could do something like this:
./configure <stuff> CXXFLAGS=${CXXFLAGS}" CFLAGS="${CFLAGS}"

And then make sure the include dirs are in CXXFLAGS, CFLAGS

10

The solution is to use

mysql_config --cflags --libs to print out the needed include directories and libraries, in this case it was:

-I/usr/include/mysql 
-L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -lrt -lssl -lcrypto -ldl -lresolv

Then include those when configuring;

sudo ./configure LIBS="-L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -lrt -lssl -lcrypto -ldl -lresolv" CFLAGS="-I/usr/include/mysql"

Prior to make && make install-full. Hope that helps someone else with the same problem…

11

One issue when wanting to back this by a database is which database, what should the schema look like. It needs to be optional, etc. It’s probably only worth following through if we can create something that will be used by 80% of the users. Just a few of the things to consider if wanting to support it directly in Suricata-Update.

12

Jason,

The changes I’d like to submit would be to,

  1. Create the connection string values in a db.conf file.
  2. Pass options in the config parser list to create the rule database if not exists.
  3. Pass options in the config parser list to update the database from a file or directory. This would keep the most recent revision of the rule.

The schema would mimic the rule class rule.py in suricata-update. This however has way too many fields to be useful, and I’d keep the numbers down to limit useful data. The rule class offers elements of the rule:

enabled, action, proto, source_addr, source_port, direction, dest_addr, dest_port, group, gid, sid, rev, msg, flowbits, metadata, references, classtype, priority, noalert, features, raw.

But I would keep, raw, priority, sid, msg, enabled, gid, classtype, rev, proto, metadata and introduce a content and comment field. I’d also mark all of these fields as varchar in the db to avoid too many problems with value types (even if it’s bad practice)

I’d also start with mysql, then optionally support postgres and MSSQL (in Azure) if the change got accepted.

13

So who is the user of this database? I see Suricata-Update as the data inserter/updater, but never really querying the database. If this is the case, I wonder if some sort of post-processing plugin support would be better. There is already another feature that could benefit from this (not database related), and then when it comes to the database, if done as a plugin users could adjust the schema as needed.

14

Jason,

I see Suricata-Update as the data inserter/updater, but never really querying the database.

The first step is to add to the suricata-update utility to sync those rules it gets with a database. I’ve successfully achived this locally and will submit a PR for review.

I’d be hopeful that the raw rule becomes the commodity, and all the peripheral stuff around that becomes the basis for more features. Eg. Hyperlinks to the internet where there is information on the threat, or a whois lookup to explain the network traffic etc. Something else that could help an analyst investigate a threat or explain a particular threat vector, or even help a system administrator prioritize his alerts efficiently by knocking out low priory rules…

15

sorry for dig out this topic again.

I think managing rules through the unix socket is cool, we are currently working on a project integrating suricata as the detection engine, but we provide a GUI for the user to modify their rules, so this is an interactive procedure. It’s possible to update the rules file and let suricata reload the rules, but it’s a bit wired to achieve through this way.

May I know if suricata team has ever worked on this feature? :grinning:

16

I’m still keen as well. I know Jason Ish wrote py-idstools that helps me to load rules into a db table, raw rule and a list of interesting fields. GitHub - jasonish/py-idstools: idstools: Snort and Suricata Rule and Event Utilities in Python (Including a Rule Update Tool) but it’s never been officially factored into Suricata.

I did get it working GitHub - CosmoRied/suricata at mysql. and GitHub - CosmoRied/suricata-update at database .

You just need to swap out the rules file for /etc/suricata/my.cnf file that contains your connection details if you really need it.

Sadly, it doesn’t use the recommended unix socket file, but I don’t see the benefits of doing that if all your connection details could be supplied in the epically long suricata.yaml file anyway. Is it just neater to connect to a unix socket?

17

I feel your pain, I myself attempted to solve the same issue by creating IDSTower, however I will lean toward Victor opinion that you are better off doing this outside Suricata, why? because you will want more than just a database, you might need

  1. a web interface to explore/search/edit/add those rules.
  2. a way to transform those rules in bulk.
  3. a way to update the rules in the database.
  4. a way to inform suricata to reload those rules
  5. …etc

so it will end up to be more than just loading the rules through the database.

1 Like