Suricata with passive optical TAP

Hello,

I am running Suricata in IDS mode, connected to my network using passive optical TAP. Passive optical TAP are very handy solution, which is very reliable, do to being completely passive.

To run Suricata in IPS mode, you need to send to Suricata both incoming and outgoing packets from given monitored link (i.e. both sides of the traffic). You have two options:

  • use switch and functionality of “port monitoring”. To get both sides of the traffic, you typically need to copy traffic from at least two ports, (the ones where your link is coming in to the switch and out from the switch), to the “monitoring port”, where you will connect Suricata. Suricata than see all packets as unidirectional - i.e. incoming on its NIC. This is in many cases not optimal solution. Just imagine, that you have 10Gbit/s link over you switch, which you would like to monitor and Suricata is also connected by 10Gbit/s port. As both directions of traffic on your link will be copied as outgoing traffic to monitoring port, you can’t exceed 10Gbit/s traffic in SUM on your link, otherwise you will saturate monitoring port! Also if you switch will die or miss-function your link will go down.

  • second way how to get traffic sniffed and copied to Suricata is easier and more reliable - using optical passive TAPs, where simple light going in fiber optics cable is split between your link and another monitored link. It is typically using duplex fibre optics links, and therefore from one monitored link you will get two outputs. One having copied downstream data and one with upstream data. Therefore you are not limited in capacity as in case as above upstream data up to full 10Gbit/s capacity are copied to one 10Gbit/s interface and downstream data again in full capacity is copied to second independent 10Gbit/s port.

The issue is, that there is no documentation about using passive TAPs with Suricata at all. As mentioned Suricata need to see both sides of the traffic to reconstructs properly flows.

  • In first case, both sides are present on the same Suricata interface.

  • In the second case each side of the traffic is present on independent Suricata interface

Is this a problem? How two interfaces shall be configured in Suricata?

I really think, that such setup shall be possible, as it is for example possible to Suricata setup in IPS mode, where it natively sees both directions of the traffic incoming on two different interfaces. Even more as with passive TAPs you only use receiving fibre (IN) on Suricata NIC SFP, and transmit fibre (OUT) is even not connected, you can even set packet copy mode on the both NICs as in IPS mode. But copying packets will be only wasting resources as these wont be transmitted anywhere.

Therefore I really hope, that Suricata can work natively with passive TAPs, only such functionality is not documented. I am seeing this as opportunity to improve on this.

I would be very thankful, if someone with Suricata code knowledge could step in and enlighten us, how we can use Suricata in TAP mode.

Best regards
L.

Unfortunately there is no explanation in Suricata documentation, how to handle

I agree that we could improve the documentation on such parts.

You brought up the IPS mode but there it’s also relevant to have both directions on each interface. Given a flow 10.0.0.1 to 10.0.0.2 you receive the initial packet from 10.0.0.1 to 10.0.0.2 on eno1 and copy this to eno2 where it’s moving to 10.0.0.2. The response from 10.0.0.2 is coming in into eno2 and copied to eno1 and passing to 10.0.0.1. So on eno1 and eno2 you see the "same’ traffic bidirectional.

Passive TAP works as long as you install them on a wire where both directions of a traffic is seen. I have several of those setups where it’s ensured to see it.
Within the “RX” from the TAP towards the Sensor where Suricata is running no “TX” is needed but within the RX is the actual traffic that was seen by the TAP.

Alternatives are port mirroring as you already mentioned, SPAN ports of if you need to merge traffic another option are packetbroker.

You could try bridging the two interfaces in Linux and set the bridge as capture device, the last time I tried this it was even worse. But maybe it’s working now.
Despite that I would recommend to ensure that you find a way to place the TAP on a place where bidirectional traffic is seen.

Please correct me if there are any errors here. Mostly a gut feel explanation of how I remember the inner workings.

The optical tap will send TX on one fiber strand and RX on the other.
The cable can as mentioned be split and put into the RX part of two SFP ports on the sensor machine.
Suricata can then be set up to read packets from both interfaces, use the AF_PACKET cluster-type: cluster-flow so the packets for a layer 4 session will be sent to the same socket / suricata worker even if they arrive on different interfaces.
The problem is that you are now stuck with a race condition, Suricata does not buffer packets and if you get for instance a SYN on the RX direction for the fiber tap and a SYN ACK on the TX they will of course go to different ports on the sensor device and can be processed at different speeds. The SYN ACK packet might be processed by the NIC before the SYN packet and sent to Suricata first.
This break reassembly logic. Interface bonding in Linux seems to buffer packets and solve the problem to some extent but costs a lot of kernel CPU time. pfring seems to handle this to some extent as well.

I am afraid, that I don’t understand your reply. Fibre optic TAP see both directions of the traffic, but it will copy it to separate fibres, which than has to be connected to TWO separate network interfaces. This is the issue I tried to describe.

So I am looking to advise, how Suricata shall be setup, to be able to reconstruct flows, where outgoing traffic is seen on interface1, while incoming traffic is seen on interface2.

I am definitely ready to prepare docs update by myself, but I will need information how Suricata works internally.

I highly doubt, that Suricata inspect both eno1 and eno2 traffic, because it will simply do scanning of the same traffic twice! Based on my expectations Suricata will probably inspect only incoming traffic, not outgoing on both interfaces. Or both directional traffic but only on one interface.

But in the current IPS mode documentation, BOTH interfaces are setup to be monitored by Suricata. So there has to be some internal mechanism, at least I hope so.

It actually HAS to be split, there are separate cables and you can not combine light on the physical level… into packets :slight_smile:

So for passive optical TAP you need to have TWO input interfaces in Suricata, each one seeing opposite monitored traffic direction. Also there will be no outgoing traffic from these interfaces.

So what you are saying is, that Suricata is not suitable for passive TAPs? (at least not for higher preformance setups ~ 10Gbit/s)

To me that seems to be huge practical limitation.

What will happen, I will setup Suricata with passive TAP in IPS mode, having two clusters each on one interface, using RSS and cluster_qm and copy-mode ips to copy packets between interfaces?

Edit: actually I found this explanation of AF_Packet copy mode. And from here:

The copy-mode variable can take the following values:

  • ips : the drop keyword is honored and matching packet are dropped.
  • tap : no drop occurs, Suricata act as a bridge

So is this a solution?

In your scenario this might be correct if the traffic coming INTO the tap is not already coming from a mirror or SPAN port. In such a case there is not much you can do except trying the bridge attempt or look for a NIC or capture card that could to the reconstruction. If you feed traffic into eno1 and eno2 there is no communication between those two.

The relevant part is that even before Suricata is inspecting that the NIC is still seeing both parts of the communication and thus RSS etc. still work and ensure that it’s correctly forwarded to Suricata.
AFAIK only in the end the packet is passed from eno1 to eno2.

But I agree that this might make one think about how it could work in IDS mode.

In addition to what Andreas’ said, I know people used to use the Linux “bond” interface to bond the 2 incoming interfaces together. In fact it looks like Security Onion still provides this feature. I think this does limit the amount of tuning you can do with RSS queues and such, as Suricata would be reading off the virtual bond interface.

I think in high speed environments that port aggregator taps or packet brokers are now more common - where you tap 2 10Gb interfaces into one 40Gb interface for instance.

But I don’t think IPS mode will you provide what you’re after. In the af-packet ips and tap modes Suricata is a bridge passing packets, so the ordering is controlled. If both sides were being fed from a tap, the ordering would not be controlled and I suspect things won’t work as expected.

That is not correct, RSS stands for Recive Side Scaling ONLY. Therefore what you are trying to imply will not work, as with RSS and cluster_mq packets applied, you will not be able to guarantee packets from same flow with opposite directions to hit same core / Suricata worker.

On the other hand it will work perfectly fine with IDS mode and mirrored port, where symetrical hash will steer both direction of the traffic mirrored to suricata port to the always same core. As this is key function of RSS symetrical hash.

you’re right about RSS. What I wanted to underline is that Suricata itself sees the complete flow on one interface. There is the async-oneside option that could help in some environments but won’t solve your case I guess.

I don’t think that I will be able to handle my traffic without RSS. So I am afraid, that I am actually correct. Suricata is not suitable to be used with high performance setup on passive TAPs. And there seems to be no workaround for that. :-o

I was also trying to find any meaningful way of the difference between cluster_flow and cluster_qm but except one liner comment in config I got no luck. Could please anyone explain, how it would work in my setup?

But I am afraid this will ruin completely Suricata performance I need anyway, as packets would need to be copied between L3 caches of different CPU.

Ok, so if you setup IPS mode, is traffic inspected by Suricata twice, independently on each of two configured interfaces? As these interfaces see same traffic.

I think this is true for any of the IDS/NSM type applications, not just Suricata. Passive taps have their place, but not for stateful deep packet inspection like Suricata, Zeek and such.