How to set a custom source for suricata-update including lua?

I have tested the following versions of Suricata:
Suricata version 7.0.2-dev (af4bb917d 2023-09-15) on Arch-Linux. installed via pfsense AUR-package,
Suricata version 7.0.2-dev also Arch, but manually compiled and installed from official git (pulled on september 26)
both times using the bundled suricata-update 1.3.0
Suricata version 6.0.13 on pfsense 2.7.0 stable installed via pfsense gui
I don’t know the suricata-update version this time.

What I want to do is having a git/http-server packaging the .rules and .lua files as tarball and serving it to suricata-update as a custom source.
I added http://192.168.102:7182/customrules.tar.gz as a source to suricata-update.
On Arch directly calling “suricata-update add-source” and on pfsense using the “extra rules” form.
On both Arch versions the rules from the .rules file get loaded, but it complains about a missing testrule.lua file, which is shipped in the tarball (double checked it) and referenced by one rule using luajit:testrule.lua.
“find / -iname testrule.lua” shows no matches on the suricata machine.

On pfsense the rule updater tells me “Extra CustomRules rules were updated.” and
“Suricata has restarted with your new set of rules”, but they are in fact not loaded. (I can tell since I included a syntactically wrong rule, for easy detection of rule loading in suricata.log and there is no error in the log.)
I’ll also post this problem on pfsense forum, but included it here, since the problem might be related.

So now my question is, what happens with the files in the tarball when suricata-update processes it and how can I autodeploy custom lua-rules via suricata-update?

Placing them manually in the rules folder works, but bypassing the update-manager seems like a really bad habit and hard to maintain.

Additional info:
security.lua.allow-rules is enabled in all instances’ suricata.yaml.
The tar.gz is created using “tar -czf customrules.tar.gz *.rules *.lua”, but I also tried including the parent directory with the same result.

After solving the pfsense issue, it shows the same behaviour as on Arch.

Thanks for your help

I replied to your specific pfSense issue on the Netgate/pfSense forum. The pfSense problem is not related to your issue here. The Suricata package on pfSense does not use the suricata-update tool.

Lua rules are not supported by Suricata-Update at this time. We have similar support for datasets, so I think this should be an easy addition.

Status update:
I submitted a fix for it.
Dataset and filehash handling was a good guide.
pull request