The bitmask semantics derived from Snort’s behavior. We are consistent with that.
I noticed we didn’t document it for byte_test – we will correct that in the near future.
In the meantime, here’s what the documentation should’ve said for bitmask 
The AND operator will be applied to the extracted value The result will be right shifted by the number of bits equal to the number of trailing zeros in the mask