I am running Suricata in IDS mode, connected to my network using passive optical TAP. Passive optical TAP are very handy solution, which is very reliable, do to being completely passive.
To run Suricata in IPS mode, you need to send to Suricata both incoming and outgoing packets from given monitored link (i.e. both sides of the traffic). You have two options:
use switch and functionality of “port monitoring”. To get both sides of the traffic, you typically need to copy traffic from at least two ports, (the ones where your link is coming in to the switch and out from the switch), to the “monitoring port”, where you will connect Suricata. Suricata than see all packets as unidirectional - i.e. incoming on its NIC. This is in many cases not optimal solution. Just imagine, that you have 10Gbit/s link over you switch, which you would like to monitor and Suricata is also connected by 10Gbit/s port. As both directions of traffic on your link will be copied as outgoing traffic to monitoring port, you can’t exceed 10Gbit/s traffic in SUM on your link, otherwise you will saturate monitoring port! Also if you switch will die or miss-function your link will go down.
second way how to get traffic sniffed and copied to Suricata is easier and more reliable - using optical passive TAPs, where simple light going in fiber optics cable is split between your link and another monitored link. It is typically using duplex fibre optics links, and therefore from one monitored link you will get two outputs. One having copied downstream data and one with upstream data. Therefore you are not limited in capacity as in case as above upstream data up to full 10Gbit/s capacity are copied to one 10Gbit/s interface and downstream data again in full capacity is copied to second independent 10Gbit/s port.
The issue is, that there is no documentation about using passive TAPs with Suricata at all. As mentioned Suricata need to see both sides of the traffic to reconstructs properly flows.
In first case, both sides are present on the same Suricata interface.
In the second case each side of the traffic is present on independent Suricata interface
Is this a problem? How two interfaces shall be configured in Suricata?
I really think, that such setup shall be possible, as it is for example possible to Suricata setup in IPS mode, where it natively sees both directions of the traffic incoming on two different interfaces. Even more as with passive TAPs you only use receiving fibre (IN) on Suricata NIC SFP, and transmit fibre (OUT) is even not connected, you can even set packet copy mode on the both NICs as in IPS mode. But copying packets will be only wasting resources as these wont be transmitted anywhere.
Therefore I really hope, that Suricata can work natively with passive TAPs, only such functionality is not documented. I am seeing this as opportunity to improve on this.
I would be very thankful, if someone with Suricata code knowledge could step in and enlighten us, how we can use Suricata in TAP mode.
Best regards
L.
Unfortunately there is no explanation in Suricata documentation, how to handle
I agree that we could improve the documentation on such parts.
You brought up the IPS mode but there it’s also relevant to have both directions on each interface. Given a flow 10.0.0.1 to 10.0.0.2 you receive the initial packet from 10.0.0.1 to 10.0.0.2 on eno1 and copy this to eno2 where it’s moving to 10.0.0.2. The response from 10.0.0.2 is coming in into eno2 and copied to eno1 and passing to 10.0.0.1. So on eno1 and eno2 you see the "same’ traffic bidirectional.
Passive TAP works as long as you install them on a wire where both directions of a traffic is seen. I have several of those setups where it’s ensured to see it.
Within the “RX” from the TAP towards the Sensor where Suricata is running no “TX” is needed but within the RX is the actual traffic that was seen by the TAP.
Alternatives are port mirroring as you already mentioned, SPAN ports of if you need to merge traffic another option are packetbroker.
You could try bridging the two interfaces in Linux and set the bridge as capture device, the last time I tried this it was even worse. But maybe it’s working now.
Despite that I would recommend to ensure that you find a way to place the TAP on a place where bidirectional traffic is seen.
Please correct me if there are any errors here. Mostly a gut feel explanation of how I remember the inner workings.
The optical tap will send TX on one fiber strand and RX on the other.
The cable can as mentioned be split and put into the RX part of two SFP ports on the sensor machine.
Suricata can then be set up to read packets from both interfaces, use the AF_PACKET cluster-type: cluster-flow so the packets for a layer 4 session will be sent to the same socket / suricata worker even if they arrive on different interfaces.
The problem is that you are now stuck with a race condition, Suricata does not buffer packets and if you get for instance a SYN on the RX direction for the fiber tap and a SYN ACK on the TX they will of course go to different ports on the sensor device and can be processed at different speeds. The SYN ACK packet might be processed by the NIC before the SYN packet and sent to Suricata first.
This break reassembly logic. Interface bonding in Linux seems to buffer packets and solve the problem to some extent but costs a lot of kernel CPU time. pfring seems to handle this to some extent as well.
I am afraid, that I don’t understand your reply. Fibre optic TAP see both directions of the traffic, but it will copy it to separate fibres, which than has to be connected to TWO separate network interfaces. This is the issue I tried to describe.
So I am looking to advise, how Suricata shall be setup, to be able to reconstruct flows, where outgoing traffic is seen on interface1, while incoming traffic is seen on interface2.
I am definitely ready to prepare docs update by myself, but I will need information how Suricata works internally.
I highly doubt, that Suricata inspect both eno1 and eno2 traffic, because it will simply do scanning of the same traffic twice! Based on my expectations Suricata will probably inspect only incoming traffic, not outgoing on both interfaces. Or both directional traffic but only on one interface.
But in the current IPS mode documentation, BOTH interfaces are setup to be monitored by Suricata. So there has to be some internal mechanism, at least I hope so.
It actually HAS to be split, there are separate cables and you can not combine light on the physical level… into packets
So for passive optical TAP you need to have TWO input interfaces in Suricata, each one seeing opposite monitored traffic direction. Also there will be no outgoing traffic from these interfaces.
So what you are saying is, that Suricata is not suitable for passive TAPs? (at least not for higher preformance setups ~ 10Gbit/s)
To me that seems to be huge practical limitation.
What will happen, I will setup Suricata with passive TAP in IPS mode, having two clusters each on one interface, using RSS and cluster_qm and copy-mode ips to copy packets between interfaces?
Edit: actually I found this explanation of AF_Packet copy mode. And from here:
The copy-mode variable can take the following values:
ips : the drop keyword is honored and matching packet are dropped.
In your scenario this might be correct if the traffic coming INTO the tap is not already coming from a mirror or SPAN port. In such a case there is not much you can do except trying the bridge attempt or look for a NIC or capture card that could to the reconstruction. If you feed traffic into eno1 and eno2 there is no communication between those two.
The relevant part is that even before Suricata is inspecting that the NIC is still seeing both parts of the communication and thus RSS etc. still work and ensure that it’s correctly forwarded to Suricata.
AFAIK only in the end the packet is passed from eno1 to eno2.
But I agree that this might make one think about how it could work in IDS mode.
In addition to what Andreas’ said, I know people used to use the Linux “bond” interface to bond the 2 incoming interfaces together. In fact it looks like Security Onion still provides this feature. I think this does limit the amount of tuning you can do with RSS queues and such, as Suricata would be reading off the virtual bond interface.
I think in high speed environments that port aggregator taps or packet brokers are now more common - where you tap 2 10Gb interfaces into one 40Gb interface for instance.
But I don’t think IPS mode will you provide what you’re after. In the af-packet ips and tap modes Suricata is a bridge passing packets, so the ordering is controlled. If both sides were being fed from a tap, the ordering would not be controlled and I suspect things won’t work as expected.
That is not correct, RSS stands for Recive Side Scaling ONLY. Therefore what you are trying to imply will not work, as with RSS and cluster_mq packets applied, you will not be able to guarantee packets from same flow with opposite directions to hit same core / Suricata worker.
On the other hand it will work perfectly fine with IDS mode and mirrored port, where symetrical hash will steer both direction of the traffic mirrored to suricata port to the always same core. As this is key function of RSS symetrical hash.
you’re right about RSS. What I wanted to underline is that Suricata itself sees the complete flow on one interface. There is the async-oneside option that could help in some environments but won’t solve your case I guess.
I don’t think that I will be able to handle my traffic without RSS. So I am afraid, that I am actually correct. Suricata is not suitable to be used with high performance setup on passive TAPs. And there seems to be no workaround for that. :-o
I was also trying to find any meaningful way of the difference between cluster_flow and cluster_qm but except one liner comment in config I got no luck. Could please anyone explain, how it would work in my setup?
But I am afraid this will ruin completely Suricata performance I need anyway, as packets would need to be copied between L3 caches of different CPU.
Ok, so if you setup IPS mode, is traffic inspected by Suricata twice, independently on each of two configured interfaces? As these interfaces see same traffic.
I think this is true for any of the IDS/NSM type applications, not just Suricata. Passive taps have their place, but not for stateful deep packet inspection like Suricata, Zeek and such.
Hi @ish , I am working on a similar setup and facing the same problem as @litinoveweedle . We built a Suricata probe with one Intel X710 10 GbE dual port NIC. We manage to configure the hash function to allow for symmetric hashing and the use of RSS, isolated CPUs from the OS and pinned them to Suricata, set IRQ affinity, bypass elephant flows with the eBPF filter, etc. Basically all the recomendations regarding high performance. The problem we faced was packet loss, since we were using a port mirror. Now we are trying to use a passive TAP on the setup, and after configuring both interfaces we see some SURICATA STREAM pkt seen on wrong thread alerts.
you can look into those flows that trigger the wrong thread and check if they are properly forwarded to th einterface. Are they by any chance split over both interfaces?
Hello @Andreas_Herz , thank you very much for your response. I do not understand what you mean by properly forwarded, since Suricata is working “passively”, only listening in both interfaces.
On top of the wrong thread alerts a lot of alerts regarding missing or out-of-order packets are also fired (SURICATA STREAM ESTABLISHED packet out of window, SURICATA STREAM ESTABLISHED invalid ack, etc.)
My network setup is as follows:
Red arrow is ens2f0 and blue arrow ens2f1. My AF_Packet configuration is bringing up a set of worker threads for each of the interfaces. If I undertand correctly, each set of worker threads (W#XX-ens2f0, W#XX-ens2f1) is only seen one side of the connection.
Is that correct? If this is the case, is there a way to “solve” this and send both sides of the connection to the same worker thread?
So if you have a TCP connection from Client A in the left network to the Server B in the right network you will end up with the following situation (I assume based on your setup):
TCP SYN from A to B is seen on ens2f1
TCP SYNACK from B to A is seen on ens2f0
TCP ACK from A to B is seen on ensf2f1
So this means that on one interface you only have the direction A->B and on the other A<-B. This will be an issue since Suricata tries to look into the whole flow but is unable to do so since the packets are in different threads.
The best solution for that is to ensure that the traffic is merged before it reaches the interface. This would require taps that can aggregate the traffic.
On pure software side you can only try to do a bond interface and hope that it works as expected. I tried that in the past but not in production. Maybe at some point in the future we can use eBPF or DPDK to take care of this, but you will always have the risk that due to timing the packet order is too broken.
Found this topic in the past I thought it might be a good idea to also try DPDK and Bond PMD if possible in your environment. (Though some experience in the Linux world is recommended prior to starting to work with the DPDK).
Thanks @lukashino . We had exactly the same issue using optical taps. The Bond PMD from DPDK seems to work perfectly in this case. It can even aggregate multiple tapped links, each having inbound and outbound traffic delivered to a separate interface. This might be useful when one leg of the same flow follows a different route.
