From November 10 to 19 we held four virtual community brainstorm events. These replaced the in-person session we normally do at Suricon. It was a new experience for us to meet virtually this way, but we feel it worked out well.
Of the four sessions, three focused on a specfic topic area leaving the last open for ongoing discussions and other topics:
- Deployment
- Rules and Analysts
- Development: extending Suricata
- Overflow
Each was planned for 90 minutes with 30 minutes “extra time”, which we ended up using in every session. So in all we talked about the Suricata roadmap for 8 hours.
We used three components in the sessions: Zoom for the call, a dedicated Slack space for chat and announcements, and our regular Redmine install. All were used by many throughout the two-week event.
The redmine contains a special tracking ticket for the event: https://redmine.openinfosecfoundation.org/issues/4097
The 3 topic sessions had the same overall structure:
- Overview of relevant Suricon 2019 tickets
- Overview of relevant work done in Suricata 6
- Ideas and plans by the development team
- Open discussion
The following is a summary of the outcomes from all of the sessions.
Deployment
New protocols: one of the main requests was more protocols. Mentioned specifically were LDAP, HTTP/3, ICS protocols. But also (better) support for oldies like IRC, SMTP was pointed out.
We discussed cloud deployments, something where we want to do more improvements. This includes traffic capture in the cloud, better support for (reverse) proxy setups and better cooperation with TLS decrypt tools.
PCAP output
A recurring topic at the brainstorm sessions over the years is the desire to have more PCAP data around alerts. Interest in just having the correct packet logged to EVE was expressed when in IDS mode (currently the ACK is logged), with more advanced interest in logging pcap after an alert is triggered to logging packets leading up to an alert.
Interest in a more official tool to convert EVE logs to pcap files was also expressed. Such a tool would operate on the “payload” field, not the “packet” as in IDS deployments, as the “packet” would not contain the correct data. Also, due to many alerts being triggered after reassembly of packets, payloads, etc, often the “triggering packet” may be gone, or no single packet actually makes sense as the triggering packet.
The team feels that for the most part the “payload” that is logged is the best option for seeing the raw data behind an alert, and we’re also looking at implementing a form of PCAP logging to log more data around alerts, logging what is in memory and then some data after the alert.
The team does feel that if full raw packet data leading up to an alert is also required that an external full packet capture system would be ideal.
Rules and Analysts
The conversation focussed on what rule writers and analysts would like to see added and improved.
Protocols: more protocols, more buffers to match on. In general it was agreed that there needs to be parity between what we log and what can be matched on.
Entropy: it was suggested to create a way to calculate and match on entropy. Possible as a transform keyword.
Matching logic: rule writers seemed unanimous about the need for a number of rule language improvements:
- Bidirectional rule (e.g. match on http request and response in a single rule)
- Flowbits prefilter: make rules that heavily rely on flowbits more efficient
- Negated matching wrt missing buffers. E.g. if a rule has a negated cookie match the rule should match if there is no cookie at all. Currently negation is only evaluated if a buffer exists.
- app_records/pdu: expose records/pdus to the rule lang so raw tcp stream matching can be avoided.
Scripting
Significant time was spent on discussing scripting support in Suricata. Currently implemented with Lua (or luajit). Problems with our lua implementation are:
- Locked to lua 5.1 due to luajit
- Implementation is not sandboxed/jailed
- Functionality beyond basic lua capabilities is complex due to differences in availability of lua modules on various platforms.
The 2nd two issues make it hard for rule vendors to include Lua based detection logic.
The lua 5.1 situation: 5.1 is no longer supported upstream. Luajit development seems to be stalled. Moving to a more modern lua version means breaking compatibility and thus giving up on luajit.
An alternative could be to vendor in lua, so we take responsibility for maintaining it for Suricata. This could mean a significant burden on the team, and distribution packagers have indicated that this generally against distro policy.
The team feels that Lua is most likely a dead end for us. So the question then becomes: what to replace it with? It’s clear that we don’t want to design our own language. In the next few months we’ll give this more thought and define requirements. Some of the suggestions made were javascript, typescript and python.
Although not a scripting language, there was discussion about WASM/WASI as well. The potential seems to be a cross platform, secure way of extending capabilities, where extensions can be developed in multiple languages. Scripting language support is considered a weak component of this ecosystem so far.
We’ve concluded that WASM/WASI is something we want to revisit in about a year, while we will continue to track the development of this ecosystem.
Development: extending Suricata
From the development team 2 major topics were presented:
- The plugin work already in progress (experimental at this time)
- Libsuricata planning
There was a lot of conversation over the plugins, about what use cases are and what capabilities should be accessible. Some plugin development is already being done and several members offered to provide feedback on the API. We hope to expand and stabilize the API in Suricata 7.
We also discussed the plans for libsuricata. Several use cases were discussed:
- Full suri pipeline in a different tool: avoid duplication of efforts
- Partial pipeline in a 3rd party tool. E.g. a proxy where you’d use suri’s app-layer, detect and output, but not packet decoding and stream engine.
- Use of components in a 3rd party tool. E.g. just the detection engine.
- Special mention for supporting the libnids API, or at least an API that supports the same use cases.
We’re going to cautiously move ahead here. It may take some time before we’ll have a stable API, but we hope to introduce a first experimental version in Suricata 7.
In general, making it easier to extend Suricata is one of the main themes for Suricata 7.
Wrapping up
Overall the event was very useful to the team, and we hope also to the community. Keep an eye on the forum here, on https://redmine.openinfosecfoundation.org/issues/4097 for updates to the “2020 brainstorm” tickets