First of all, sorry to post here, but I believe that mod_security with iptables makes harder for hackers to gain resources access.

I was wondering if someone could tell me if he/she has be able to use iptables strings modules with hitcount modules. Why? Easy to answer. You could be able to limit access to php forms using string (but for performance Deep Packet Inspection is not the best approach) and using hitting count.  

You could block more than 5 chances to gain access to example_form.php.

Is it a bad idea? Mod security has brute-force rules?

I know that you can develop new rules to approach this solution or use some other alternatives as captchas or honeypots fields.

Kind regards.