Af_packet + RSS, some workers doing nothing / not loaded

You have some information here 9.2. Packet Capture — Suricata 6.0.2 documentation about the ordering.
On the commercial side we often see what you describe , only one part of the traffic mirrored or mirrored to separate interfaces. This is not optimal for statefull inspection and detection.
The main point is - detection actually, the packets/flows need to be inspected and seen by Suricata the same way the “end client” will “see” them to provide full inspection.
What often is done in such cases is employment of packet brokers. ( Suricata with passive optical TAP - #7 by ish )