# High overhead from rs\_dns\_state\_get\_tx causing packet loss
**URL:** https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136
**Category:** Help
**Created:** [February 22, 2021, 5:24pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136 "2021-02-22T17:24:48Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@applezip](https://forum.suricata.io/u/applezip)
#### Post date: [February 22, 2021, 5:24pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/1 "2021-02-22T17:24:48Z")
After first starting Suricata up, everything runs fine for a few hours but eventually I get (seemingly) unrecoverable packet loss.
Monitoring the processes with perf top, `rs_dns_state_get_tx' and `AppLayerDefaultGetTxIterator’ slowly creep up in overhead% and eventually overtake `DetectRun.part.16’. Once this happens, I start getting packet loss.
This was after 15hrs:
Samples: 2M of event ‘cycles’, 4000 Hz, Event count (approx.): 1171695080430 lost: 0/0 drop: 0/0
Overhead Shared Object Symbol
46.41% suricata [.] rs\_dns\_state\_get\_tx
28.55% suricata [.] AppLayerDefaultGetTxIterator
9.87% suricata [.] FlowGetProtoMapping
4.15% suricata [.] DetectRun.part.16
1.01% suricata [.] DetectEnginePktInspectionRun
0.91% suricata [.] DetectEngineInspectRulePacketMatches
0.73% suricata [.] rs\_sip\_state\_get\_tx
I have the stats output from the same time attached as text file. [statsout.log](https://forum.suricata.io/uploads/short-url/d5EojxovEpVzNsIUUEhyLDEB0QM.log) (6.6 KB)
OS: CentOS8 stream
CPU: 2x Xeon E5-2699 v4 (88 HT cores)
RAM: 128GB
NIC: Napatech NE40E3-4
Data rate: 10gbps sustained (pushing bigFlows.pcap from [tcpreplay.appneta.com](http://tcpreplay.appneta.com) through a packet broker to the napatech)
It’s slow to diagnose as it takes hours for the function to creep up to the top of the list in perf.
---
### Author:  [@vjulien](https://forum.suricata.io/u/vjulien)
#### Post date: [February 23, 2021, 8:49am UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/2 "2021-02-23T08:49:40Z")
What version of Suricata are you using?
---
### Author:  [@applezip](https://forum.suricata.io/u/applezip)
#### Post date: [February 23, 2021, 11:37am UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/3 "2021-02-23T11:37:33Z")
Whoops, that seems like a pretty important detail.
Suricata 6.0.1, compiled from source. Build info attached. [buildinfo.log](https://forum.suricata.io/uploads/short-url/8tKRHmqgU36ccdnb861DcdpX42M.log) (6.7 KB)
---
### Author:  [@applezip](https://forum.suricata.io/u/applezip)
#### Post date: [February 24, 2021, 1:23pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/4 "2021-02-24T13:23:44Z")
I disabled the DNS parsers and have been running for 17hrs with average 0.2% packet loss, versus the average 7.2% packet loss over my last 15hr run with DNS parsers enabled. When the packet loss starts with DNS parsers, I see about 30% packet loss and I cannot recover until I kill the feed or restart Suricata.
I see that the master branch has some recent changes to src/app-layer-parser.c, including some transaction cleanup. I may try to merge those changes into the 6.0.1 build locally and see how it goes.
---
### Author:  [@Andreas\_Herz](https://forum.suricata.io/u/Andreas_Herz)
#### Post date: [February 27, 2021, 8:01pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/5 "2021-02-27T20:01:29Z")
I can see that on some deployments as well, we will keep an eye on that. If you could test it with Suricata 5.0.5 that would help us to narrow it down to changes from 5 to 6.
---
### Author:  [@applezip](https://forum.suricata.io/u/applezip)
#### Post date: [March 2, 2021, 11:37am UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/6 "2021-03-02T11:37:14Z")
After an 18hr run with 5.0.5, I have 0% packet loss. Same host, same settings, still 10gbps sustained.
---
### Author:  [@vjulien](https://forum.suricata.io/u/vjulien)
#### Post date: [March 2, 2021, 12:26pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/7 "2021-03-02T12:26:24Z")
We have some fixes in the just released 6.0.2 that might help. Are you able to try it out? (See [Suricata 6.0.2 and 5.0.6 released](https://forum.suricata.io/t/suricata-6-0-2-and-5-0-6-released/1170))
---
### Author:  [@applezip](https://forum.suricata.io/u/applezip)
#### Post date: [March 3, 2021, 11:20am UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/8 "2021-03-03T11:20:37Z")
I am still seeing the same issue with 6.0.2. Averaging 8.1% packet loss after 14 hours at 10gbps.
I’ll attach the main perf top and annotations for ‘`rs_dns_state_get_tx`’ and ‘`AppLayerDefaultGetTxIterator`’
 
---
### Author:  [@vjulien](https://forum.suricata.io/u/vjulien)
#### Post date: [March 3, 2021, 3:42pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/9 "2021-03-03T15:42:44Z")
Are you able to provide a `perf top` screenshot after running it with the `-g` option? I’d like to see if we can find out which path leads to these calls.
---
### Author:  [@ish](https://forum.suricata.io/u/ish)
#### Post date: [March 3, 2021, 9:19pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/10 "2021-03-03T21:19:33Z")
Are you willing to try a patch or 2? I’ve somewhat replicated this by crafting a misbehaving DNS client, but I’ve seen similar in the real world.
---
### Author:  [@applezip](https://forum.suricata.io/u/applezip)
#### Post date: [March 8, 2021, 4:01pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/11 "2021-03-08T16:01:45Z")
Yes, I can try some local patches.
Here’s the updated output from perf:
---
### Author:  [@ish](https://forum.suricata.io/u/ish)
#### Post date: [March 17, 2021, 5:47pm UTC](https://forum.suricata.io/t/high-overhead-from-rs-dns-state-get-tx-causing-packet-loss/1136/12 "2021-03-17T17:47:05Z")
I’ve found a few cases with TCP DNS where this can happen, in particular where there are DNS TCP streams that are long lived and messages may be lost, or the client floods the server (which I have seen on the real internet).
This patch should help with the issue, but we’re looking at better ways as well. Please let me know. If you know there is no TCP DNS in your traffic, this is unlikely to help.
[https://github.com/jasonish/suricata/commit/ddb78e60de5a35f09548b6d93e55a57accfb4e05.patch](https://github.com/jasonish/suricata/commit/ddb78e60de5a35f09548b6d93e55a57accfb4e05.patch)
Thanks.