Suricata build environment

Developers
1

Hi Suricata Developers!

Firstly, i have not contributed to Suricata as of yet but i intend to! i have a few things going on regarding Suricata and one thing i have realized i do recently is that i re/build Suricata a lot depending on the deployment or CTF challenge i wish to use Suricata in etc.

What id like to know is what tooling or workflow other developers out there use for testing out dev features that may need a new flag being configured and additional dev libraries installed to test a feature against pcap data to verify the results?

Is a set of docker images best with different versions of Suricata all having access to pcaps?

or Suricata installed as .deb from the ppa on a vm do other people then compile Suricata in a different directory and have multiple instances of Suricata on the same machine?

a snippet of my build options are below. I was going to script this to do either pull latest from suricata master or development branch to build against but just looking to more experienced people than i to ask for specifics around their setup

Dependencies
Pip install pyaml or pip3 install pyaml
needs to be installed as the command “suricata-update” requires it and suricata update is used to manage the rulebase within Suricata 4+ versions

apt install libhtp-dev libcap-ng-dev libmagic-dev libluajit-5.1-dev libhiredis-dev liblz4-dev libjansson-dev lua-zlib-dev libgeoip-dev zlib1g-dev pkg-config python-yaml libyaml-dev cargo pfring nprobe ntopng ntopng-data cento nbox libnet1-dev libevent1-dev ragel cmake libboost-dev libpcap-dev libnspr4-dev nss-devel libmaxminddb-dev && export CFLAGS="-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security" && export SECCFLAGS="-fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security" && export PCAP_CFLAGS="-I/usr/include" libmaxminddb-dev libnspr4-dev libfl-dev libnetfilter-log-dev

HyperScan configuration

Hyperscan is a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, yet functions as a standalone library with its own API written in C. Hyperscan uses hybrid automata techniques to allow simultaneous matching of large numbers (up to tens of thousands) of regular expressions, as well as matching of regular expressions across streams of data.
#download hyperscan source onto the system
Wget https://github.com/intel/hyperscan/releases/tag/v5.2.1
unzip hyperscan-master.zip
cd hyperscan-master/
mkdir build
cd build/
apt install libpcre2-dev libpcre2 libpcre2-32-0
pcre-config
cmake -DBUILD_STATIC_AND_SHARED=1 …/ && make && make install
echo “/usr/local/lib” |tee --append /etc/ld.so.conf.d/usrlocal.conf
cat /etc/ld.so.conf.d/usrlocal.conf
Ldconfig

Now lets download latest source file from APT for Suricata Apt source suricata
apt source suricata
cd suricata-5.0.1/

./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --datarootdir=/usr/share --enable-gccprotect --enable-gccmarch-native --enable-python --enable-afl --enable-nfqueue --enable-pfring --enable-non-bundled-htp --enable-luajit --enable-geoip --enable-pie --enable-hiredis --enable-rust --with-libpfring-includes=/usr/local/include --with-libpfring-libraries=/usr/local/lib

make install

Cd build
#build a deb
fpm -s dir -t deb -n dev-suricata-5.0.1 /home/user/suricata-5.0.1/
#compile from source
make
do make install-rules to install binaries and download latest rules
make install-rules
do make install-full if you want configuration files as well as rules
make install-full

Also, Is it common for researchers or students to use ET Pro rulesets for personal use of signature development? I have used them previously professionally and think it would be a great addition for an individual to use them for their own research? if the cost was different from an enterprise to an individual?

2

Hmm this looks pretty good

image
image913Ă—537 54.6 KB

Thanks to the talk at Suricon

3

Any reason for not just using hyperscan from the Debian/Ubuntu repositories?

You can also use the PPA https://suricata.readthedocs.io/en/suricata-5.0.2/quickstart.html#installation

Wrt pcaps, the main public repo is Suricata-Verify: https://github.com/OISF/suricata-verify (in there see the tests/ directory)

4

Yeah this stems from a configuration i have in prod that uses Pf_ring with a FPGA card so i cant use the ppa and need to compile it. So i thought that if i kept with that theme of compiling from source i could get clone the daily or beta repo to build within a fresh docker build for different features such as the forensics work Eric Leblond is doing.

Plus ive never needed to do this so part of is i just want to setup the workflow so i can share it with others later :slight_smile:

Nathan

5

I find that docker works well. Just use a build arg for when you need a repo updated so you’re not having to do a completely fresh build: https://github.com/protectwise/suricata-ipc/blob/master/docker/Dockerfile#L69

6

There was a system like this in place a number of years ago, but it was heavily abused, so it was abandoned. I would love to have more arguments to take up the chain regarding re-implementing an ETPRO researcher program, so if you have use cases, please let me know.

1 Like
7

I can understand if they removed it due to abuse, I just think it’s the best rule set out there and is an amazing starting base for signature development. If there was some way you could tie it to an individual account for research purposes so it won’t be used commercially id be very interested. I just think that researchers are the ones contributing to the advancement of suricata and rule detection alongside proofpoint but an individual researcher can’t afford the cost as a corporation can to have an equal playing field. Such as creating a rule in community not knowing it may already exist in et pro.

8

Thanks Danny. So if I had this built. I could then swap out the got clone repo with the forensics mode in development now but all other build parameters the same to replay over same pcaps and identify differences in log output etc

9

Yep. And then use build args when you need a new pull of that development repo.

10

Yea, we don’t have a good way to avoid the abuse. We love to receive rules and pcaps at ET, if we have someone send us something with ETPRO coverage, we generally move it into the ET OPEN ruleset. If it’s something we don’t have coverage on it goes into the ET OPEN set. When we have researchers contributing/interacting with us a lot, there are various things that we can do in that situation. For the purposes of learning signature generation, there is a lot of good stuff in ET OPEN and we put stuff in there often. Always happy to chat about that. For the purposes of detection, theres the ET Telemetry thats bundled with opnsense which is free.

11

Did you have to compile tcpdump via the latest pf_ring git? Did you run into a failure looking for pcap_loop? Could you share how you made your pf_ring/pcap/tcpdump libraries to compile suricata against?