High Kernel drops count when downloading somethings

I am using suricata as IPS and there is high kernel drops.
I use a 2 LAN server to simulate http server and client (with 1Gbps link)
Then I download a binary file from server ,traffic pass suricata.
At first, packets count is very small, and kernel drops is 0
and after download(about 6g file), in stats ,packets count is still very small, and no kernel drops.
And then after about 2 mins, the packets count suddenly increase and at the same time get a high kernel drops(about 79%)
Read the stats log but nothing find and then I use perf top -p $(pidof suricata)

In the 2 mins between flow closed and high kernel drops get, I see this.

16.40% libpthread-2.28.so [.] __pthread_mutex_trylock
9.34% libpthread-2.28.so [.] __pthread_mutex_unlock_usercnt
4.20% suricata [.] DefragTimeoutHash
1.95% suricata [.] DetectRun.part.14
1.82% [kernel] [k] check_preemption_disabled

What version of Suricata are you using?

What packet capture method are you using with your IPS?

6.0.4
AF-packet mode with config

af-packet:
  - interface: ens1f1
    threads: 16
    cluster-id: 97
    cluster-type: cluster_qm
    defrag: yes
    rollover: no
    xdp-mode: driver
    xdp-filter-file:  /root/suricata-6.0.4/ebpf/xdp_filter.bpf
    bypass: yes
    copy-mode: ips
    copy-iface: ens5f1
    buffer-size: 32768
    use-mmap: yes
    mmap-locked: yes
    ring-size: 500000
    # block-size: 2097152
  - interface: ens5f1
    threads: 16
    cluster-id: 95
    cluster-type: cluster_qm
    defrag: yes
    rollover: no
    xdp-mode: driver
    xdp-filter-file:  /root/suricata-6.0.4/ebpf/xdp_filter.bpf
    copy-mode: ips
    copy-iface: ens1f1
    buffer-size: 32768
    bypass: yes
    use-mmap: yes
    mmap-locked: yes
    ring-size: 500000
    block-size: 2097152

Can you share your whole config?
Can you try a run without XDP?