Have a look at Suricata-IDS and Nmap and Outbound: Port Scanning & Brute Force detection - #3 by FlorinMarian