The setup-app-layer.py script will kickstart the process of adding a new application layer protocol to Suricata by creating most of the needed functions and registrations. However, all the actual code within those functions to detect a new protocol will have to be added, accordingly to that new protocol specification, before suricata will actually detect the intended app-proto.
As Andreas said, if you did that, it would be easier to help seeing to code, easier to understand what might be missing.