Hi everyone,
I have built a sensor on some Dell hardware with decent system capacity, it has dual gold XEON gpu and 377GB of ram and a Silicom 40Gbps capture card FPGA using PF_RING FPGA license from NTOP and Suricata 5.0.2 compiled from source.
Suricata is connected to a Packet broker and is listening to an aggregate feed of traffic. It seems approximately 5Gbps. I have configured the below config in suricata.yaml and it seems to be multithreaded and working nicely.
PF_RING configuration. for use with native PF_RING support
for more info see http://www.ntop.org/products/pf_ring/
pfring:
- interface: fbcard:0:a:0
Number of receive threads. If set to 'auto' Suricata will first try
o use CPU (core) count and otherwise RSS queue count.
'#'threads: 24
Default clusterid. PF_RING will load balance packets based on flow.
All threads/processes that will participate need to have the same
clusterid.
'#'cluster-id: 99
Default PF_RING cluster type. PF_RING can load balance per flow.
Possible values are cluster_flow or cluster_round_robin.
'#'cluster-type: cluster_flow
bpf filter for this interface
'#'bpf-filter: tcp
If bypass is set then the PF_RING hw bypass is activated, when supported
by the interface in use. Suricata will instruct the interface to bypass
all future packets for a flow that need to be bypassed.
#bypass: yes
Choose checksum verification mode for the interface. At the moment
of the capture, some packets may be with an invalid checksum due to
offloading to the network card of the checksum computation.
Possible values are:
'#'- rxonly: only compute checksum for packets received by network card.
'#' - yes: checksum validation is forced
'#' - no: checksum validation is disabled
'#' - auto: Suricata uses a statistical approach to detect when
checksum off-loading is used. (default)
Warning: 'checksum-validation' must be set to yes to have any validation
checksum-checks: auto
Second interface
'#'- interface: eth1
threads: 3
'#' cluster-id: 93
'#' cluster-type: cluster_flow
Put default values here
'#' - interface: fbcard:0:a:0
'#' threads: 2
Suricata is multi-threaded. Here the threading can be influenced.
threading:
set-cpu-affinity: yes
Tune cpu affinity of threads. Each family of threads can be bound
on specific CPUs.
These 2 apply to the all runmodes:
management-cpu-set is used for flow timeout handling, counters
worker-cpu-set is used for 'worker' threads
Additionally, for autofp these apply:
receive-cpu-set is used for capture threads
verdict-cpu-set is used for IPS verdict threads
cpu-affinity:
- management-cpu-set:
cpu: [ 0 ] # include only these CPUs in affinity settings
- receive-cpu-set:
cpu: [ 40 ] # include only these CPUs in affinity settings
- worker-cpu-set:
The problem i have is when i view my ring buffer on the card is set to 128GB of capacity but about 5 minutes into starting the capturing process i can see a large number of discards counting.
Speaking with the Vendor of the card they are saying i can increase the size of the buffer but ultimately its the application not keeping up with the ingestion rate so i need to optimize it.
As you can see from the above config that is how i have configured Suricata relating to the FPGA card using PF_RING (cluster-id commented out etc due to it crashing) and CPU-affinity settings, Does this look right and should i be making tweaks elsewhere in order for Suricata to keep up?
I am using the emerging threats pro ruleset.
Happy to provide further config or ./configure parameters. Where can i look in logs or stats file to identify potential problems with Suricata processing the load of traffic? I can setup some filters on the broker to slice some traffic if needed to reduce load. I would be expecting based on the hardware spec of the card to be able to capture around 30Gbps based on testing i have done at the card layer.
i used ‘#’ to represent the portion of config actually commented out, otherwise markdown formats it nice and large for me on here 
Edit: I thought id add the system output also checking the status of the daemon
the ioctl check is standard as linux cannot identify the interface im using correctly as its a pf_ring FPGA “fbcard:0:a:0” interface. the rest of the output is related to the commented out sections. When they are uncommented Suricata crashes
- suricata.service - Suricata Intrusion Detection Service
Loaded: loaded (/etc/systemd/system/suricata.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-03-29 22:28:04 UTC; 32min ago
Process: 29196 ExecStartPre=/bin/rm -f /var/run/suricata.pid (code=exited, status=0/SUCCESS)
Main PID: 29203 (Suricata-Main)
Tasks: 55
Memory: 1.8G
CPU: 1h 58min 48.622s
CGroup: /system.slice/suricata.service
└─29203 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --pfring
- Mar 29 22:28:04 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:04 - (suricata.c:1084) <Notice> (LogVersion) -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (runmode-pfring.c:284) <Error> (ParsePfringConfig) -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Could not get cluster-id from config
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (runmode-pfring.c:332) <Error> (ParsePfringConfig) -- [ERRCODE: SC_ERR_GET_CLUSTER_TYPE_FAILED(35)] - Could not get cluster-type from config
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (util-ioctl.c:296) <Warning> (GetEthtoolValue) -- [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get feature via ioctl for 'fbcard:0:a:0': No such device (19)
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (util-ioctl.c:296) <Warning> (GetEthtoolValue) -- [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get feature via ioctl for 'fbcard:0:a:0': No such device (19)
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (util-ioctl.c:296) <Warning> (GetEthtoolValue) -- [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get feature via ioctl for 'fbcard:0:a:0': No such device (19)
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (util-ioctl.c:296) <Warning> (GetEthtoolValue) -- [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get feature via ioctl for 'fbcard:0:a:0': No such device (19)
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (util-ioctl.c:296) <Warning> (GetEthtoolValue) -- [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get feature via ioctl for 'fbcard:0:a:0': No such device (19)
- Mar 29 22:28:44 sensor01 suricata[29203]: [29238] 29/3/2020 -- 22:28:44 - (source-pfring.c:586) <Error> (ReceivePfringThreadInit) -- [ERRCODE: SC_ERR_PF_RING_SET_CLUSTER_FAILED(37)] - pfring_set_cluster returned -7 for cluster-id: 1
- Mar 29 22:28:44 sensor01 suricata[29203]: [29203] 29/3/2020 -- 22:28:44 - (tm-threads.c:2170) <Notice> (TmThreadWaitOnThreadInit) -- all 49 packet processing threads, 4 management threads initialized, engine started.
Cheers,
Nathan
