Overview
Use r2hex.py to parse snort or suricata rule files (matching the file names in test_types.py), AND to generate data that will match (or trip) each alert in the rule file.
Use r2hex.py and test_types.py with populate_database.py to build a postgres database containing the parsed rule information and data for all rule files (matching the names in test_types.py) in a specified directory.
Use simulateMaliciousTraffic.py, config.cfg and endpoint_server.py to generate network data that can be used to test an IDS. Developed and tested on BT 5.1, but it should work on a linux distribution with postgresql, netcat and python installed. SimulatedMaliciousTraffic.py (as configured with config.cfg) works with endpoint_server.py to simulate a client and server and pass the "malicious" data appropriately for a give rule. With these scripts you can stress test your IDS, excercise you incident response team, or test a new rule that you've developed!
Design Description
This document describes the design of the not-so-obvious python scripts, namely:
1. populate_database.py
2. r2hex.py
3. simulateMaliciousTraffic.py
Populate_Database.py script
Introduction
The simulatemalicioustraffic module obtains all information concerning how to generate traffic from a postgres database. Database creation is performed prior to using with this script, which imports r2h.py.
Overview
The Populate_database.py script is used to connect to a postgres database, create the database named IDS_Testing_Framework and the schema named ids_testing, and to populate it with information and directives derived from snort rules.
To re-run this script, the ids_testing schema must first be drop';
Script Arguments
Populate_Database.py <path to rules directory>
Database structure
Two tables are created for each rules file listed in test_types.py. The rule files used may be a subset of those listed in test_types.py, but there must be a test_type that matches the rule file name. To create a smaller test set than all suricata emerging rule files, simply point the script to a folder with the subset of rule files you are interested in.
The two tables created for each rule file are a (1) directives table, and (2) a rules table. The rules table simply contains a parsed out version of the each rule, and is not currently used by the module. The directives table however contains all information necessary to create and send network data necessary to trip the IDS. The parsing and interpretation of the rules files is performed by the rule parsing module r2h.py that is import by this script.
The structure of the rules table - and there is one created and named after each rules file - is as follows:
"timestamp" timestamp with time zone,
"action" text,
"protocol" text,
"source_ip" text,
"source_port" text,
"destination_ip" text,
"destination_port" text,
"options" text,
"sid" text
The structure of the directives table - and there is also one created and named after each rules file - is as follows:
"timestamp" timestamp with time zone,
"protocol" text,
"packet_data" text,
"source_net" text,
"source_port" text,
"destination_net" text,
"destination_port" text,
"syn_net" text,
"msg" text,
"sid" text,
"err_code" integer,
"err_msg" text
The rule files (*.rules) are parsed by the r2h class.
The rules table is populated using the follow r2h object elements:
timestamp <- time.asctime(),
action, <- r2h.action
protocol <- r2h.protocol
source_ip <- r2h.snet
source_port <- r2h.sport
destination_ip <- r2h.dnet
destination_port<- r2h.dport
options <- r2h.msg
sid <- r2h.sid
The directives table is populated using the follow r2h elements:
timestamp <- time.asctime()
protocol <- r2h.protocol
packet_data <- r2h.data
source_net <- r2h.snet_resolved
source_port <- r2h.sport_resolved
destination_net <- r2h.dnet_resolved
destination_port<- r2h.dport_resolved
syn_net <- r2h.flow_resolved
msg <- r2h.msg
sid <- r2h.sid
err_code <- r2h.err_code
err_msg <- r2h.err_msg
The some fields in the directives table require no further interpretation necessary to craft the network data, such as the field "packet_data". However, other fields cannot be used verbatum and must be interpreted, such as source_net, destination_net and syn_net. This interpretation is performed by the new module, simulateMaliciousTraffic.py.
If the error code for a rule is greater than zero, then this db rule is not used to generate network traffic.
Packet_data is hex encoded, it is reformatted to include \x so that the netcat commands can be properly formated with the "echo" command.
class R2Hex
Introduction
Snort and suricata rule files are text files that describe the salient aspects of a data that alert the IDS to the network presence of attack. These rule files are parsed by this module when the populate_database.py script is run.
Instantiating the r2hex class
An R2Hex object is instantiated with a snort/suricata rule passed as a string.
Example rule: the_rule = "alert tcp $EXTERNAL any -> $HOME_NET 80..."
Example instantiation: r2hex_object=R2Hex(the_rule)
Class attibutes
R2Hex object attributes include the following:
r2hex_object.err_code -> 0=success, error otherwise
r2hex_object.err_msg -> Success, or error msg otherwise
r2hex_object.rule -> the_rule
r2hex_object.protocol -> tranport protocol
r2hex_object.data -> packet data that will trip the alert
r2hex_object.action -> snort action (alert, log, etc)
r2hex_object.snet -> the source network that sends data
r2hex_object.sport -> the source port
r2hex_object.direction -> snort rule direction (always ->)
r2hex_object.flow -> snort flow (from/to server/client)
r2hex_object.dnet -> the network that recieves data
r2hex_object.dport -> the destination port
r2hex_object.msg -> the rule msg
r2hex_object.sid -> the rule SID
r2hex_object.sport_resolved -> actual port number (instead of $HTTP_PORTS, etc.)
r2hex_object.dport_resolved -> actual port number (instead of $HTTP_PORTS, etc.)
r2hex_object.snet_resolved -> resolved to external/internal understood by module
r2hex_object.dnet_resolved -> resolved to external/internal understood by module
r2hex_object.flow_resolved -> resolves to external/internal (initiator of 3-way handshake)
Class methods
The following class methods perform simple regular expression pattern matching to parse out the relevant field in a snort/suricata rule. As no processing or interpretation of the field is performed, the relevant methods are uninteresting and include the following methods:
self.msg = parseOutMsg(self, srule)
self.directon = parseOutDirection(self, srule)
self.action = parseOutAction(self, srule)
self.protocol = parseOutProto(self, srule)
self.sport = parseOutSport(self, srule)
self.sid = parseOutSid(self, srule)
self.dport = parseOutDport(self, srule)
self.snet = parseOutSource(self, srule)
self.dnet = parseOutDestination(self, srule)
self.flow = parseOutFlow(self, srule)
The remaining methods are more interesting, and are described in some detail next.
self.data = parseOutData(self, srule)
This method searches the rule for the content-describing key words, either "content", "uricontent", or "pcre", as well as for the content modifiers: "!", "offset", "distance" and "isdataat", and then generates the data pattern that will be used to create the necessary packet data. If the rule is related to HTTP, then this method creates the necessary HTTP header data. The data is generated and returned as a hex encoded string. Some snort rule content key words can be (and are) ignored by this class because they are really snort engine directives for efficiency. An example is the "within" key word, which directs snort to only search a portion of the data packet for content. (Note: The keywords byte_test and byte_jump are not yet addressed by this class. They are more difficult to process, and are so far used in only about 5% of snort rules. Will be addressed in a future version.)
self.flow_resolved = resolveFlowVar(self, flow, snet, dnet)
This method uses the parsed out flow, snet and dnet rule fields to determine which network will initiate the 3-way handshake (the other end-point will of course listen for a connection). The following is necessary to understand the use of flow in a rule:
1. "from_server" and "to_client" have identical meaning, and vice versa
2. The "server" is located on the network to the right of the rule direction arrow
3. The "client" is located on the network to the left of the direction arrow
4. The rule direction is alway -> (never <-) and indictates data direction
Example: Consider the partial example rule:
alert http $HOME_NET any -> $EXTERNAL any ... flow:established, from_server...
The tcp connection must be initiated from the $HOME_NET (points 2 and 3) and, once the tcp connection is established, the data must pass from $HOME_NET to $EXTERNAL. (This rule example could be used to detect implant calling out to command and control servers.)
The methods that follow could be considered as private methods:
resolveNetVar(self, var)
This method resolves the source and destination field variables to either "external", "internal", or "any". If the network variable matches EXTERNAL, then "external" is resolved, matches any, then "any" is resolved. "Internal" is resolved in all other cases.
parseOutPort(self, port)
This method is called by parseOutDport and parseOutSport, and addresses the use of the ! operator and port ranges - if applicable - in the rule.
resolvePortVar(self, var)
This method is called by sport_resolved and dport_resolved, and resolves port variables used in some rules, variable such as $HTTP and $FTP, to there respective port numbers, 80 and 21 in this example.
