Features

Log2table allows you to continuously monitor your logfiles. You can trigger actions when a specific message comes in your audited logfiles or when a specific number of occurences are present.
I've developed it as a simple Intrusion Detection System and to let away bad guys of my web servers. For example, those who are trying passwords on ssh, those who are trying cross site htpp requests, those who are looking for wp-admin.php, ...

The first match for such task was fail2ban. But I was facing some difficulties to configure it to my specific needs. After having developed a minimalist script (here), a reader of my blog has started to develop something around this idea (Vilain). After several changes of his code, he informs me that I'm deviating too much from his initial idea and inform me that he will no more merge my changes in his git repository. So I rename it log2table, mainly because this application build a link between logiles and firewall tables.

I've build log2tble to be as flexible as possible. It should be easy to adapt to other firewalls (iptable with ipset). It could eventually be configured to execute other tasks than add/remove IP from a firewall table.

The code is split in 4 parts:

• the main loop: log2table.py
• the rules: they should all be in rules.py
• the parameters requested by the rules: parameters.py
• the command file: cmd_<platform>.py</platform>

The last 3 files are available in /etc/log2table and can be adapted to your specific needs.

Rules are indenpendent of the logfiles you are tracking. Log2table consolidates the actions triggered by source IP. Indeed, a specific machine (source IP) could generate entries in your http log file, but also in your authlog file, or in your /var/log/messages file. Whatever this IP is doing on your machine will be tracked. You can imagine to apply the same rule for different actions in different logfiles. In such case you have what I call, a consolidation of actions; this will generate a consolidated repsonse.

For flexibility reasons, several parts of the program are available to the users in /etc/log2table by default. To adapt those config files a minimum set of skills of Python is required. Parameter.py is a simple Python dictionary where you can define which logfile you want to follow and which rule you want to trigger once the regex rule match a line in your tracked logfile. The rules are also available to the end-users in rules.py. Each classes in rules.py allow you to define the behavior to adopt when a line match your regular expression.

Currently I propose 2 rules. One, called Hacker, to ban unwanted connections. This rule adds the IP having performed an unwanted connection into an OpenBSD PF table. This rule remove those "bad IPs" after 1 hour at least. Inside my pf.conf file, I block all connections for IP contained in this table.

As example, here the rule I've added in my pf.conf file:

block in quick proto tcp from <bruteforce> to any

The other class in rules.py is called WifiTemp. This class registers each IP on their first connection by watching the dns.log file they are using (imagine they are forced to use this dns server). After 1 hour their IP is added in a PF table where PF block their traffic. After again 1 hour the rule remove IP from the PF table, so they can re-used the Wifi. As you can understand the goal is to provide a Wifi connection for 1 hour, after it's blocked for one hour at least, after it's available for an another 1 hour, etc...
This rule is just for demonstration purposes, it shows the flexibility those rules can offer.

Each rule can have several parameters coming either from the regex, either provided by the parameters.py file.

Feel free to share you rules, I'll include them in the next releases :-)

Pre-requisites

You just need to have python3.6 installed.

This application has been developed on OpenBSD, but with very few adaptations it should work on other systems.
Normally you should only modify the cmd_<platform>.py file. </platform>

In this cmd_<platform>.py file you must make sur that the userid defined (in parameters.py) can run the required commands. On OpenBSD we use doas. On other system you could use sudo. Thanks to assure that this use can run the command you define in this command file. </platform>

Installation

Just untar the package and run 'make' as root.

tar -xzvf log2table.tgz
cd log2table
make

This will install log2table in /usr/local/bin
and configuration files in /etc/log2table.

Feel free to adapt Makefile if you want to use other directories.

If you are not on OpenBSD, please remove from Makefile the line

install log2table.rc /etc/rc.d/log2table

log2table.rc is a file required for the rcctl command.

Configuration

Options

log2table offers few options:

• --debug: allow you to run in debug mode and be much more verbose
• --config: allow you to load an another parameter.py file. This is useful in combination with the debug mode to test some features

parameters.py

We have some global parameters like:

• 'user' : The userid on which you log2table will run, after the startup.
• 'logfile': the log2table logfile
• 'sleeptime': number of seconds you want to "sleep" before log2table check, for new data, the logfiles you want to track
• 'rules': the rules you plan to use in the sections bellow. This information will allow log2table to load IP addresses already existing in the firewall's tables. Each Rule, in rule.py, has a table's name.

For each log file you want to follow, you must provide several informations:

• One regex patterns: 'regex'. This is the main element on which you will trigger some actions. The regex can return one or several information. The name of those values must be in accordance to what the Rule expect.
• One rule name: 'name'. This is a text value describing the name of the rule. This name will be present in the log of log2table.
• The name of the rule to use: 'rule'. This is a string value where you have to provide the name of the rule present in the rules.py file.
• Some parameters: 'params'. Those parameters will be provided to the rule. Here again be careful that the provided parameters are compliant with what the rule expect. Those values are extended to the parameters provided by your regex pattern. Be careful to not overwrite some values coming from your regex.

rules.py: The hacker rule

Currently you have 1 Rule class called: Hacker. This class ban IP having crossed a weight of 100. The weight must be provided in your logfile's definition.
This class expects to have the "IP" values coming from your regex.
Moreover you have to define a weight for each regex pattern.
In the parameters.py I propose you, someone trying ssh with root userid will receive a weight of 50. Someone trying ssh with another userid will receive a weight of 30.
As consequence if a hacker try 2 times the root account, he will be added to the banned table of the firewall. If he tries once the root and 2 other users he will be banned too. If he tries 4 others users he will be banned too.

parameters.py: Test your regex

You can test your regex by copy pasting the targeted line in a file called "test.txt". Then you can simply execute your parameters.py file.

python3.6 parameters.py

Lines started by "#" will be skipped.

You will get an output like this:

Line 2 match rule ssh_invaliduser: data: {'ip': '91.197.232.107'}
Line 5 match rule ssh_invaliduser: data: {'ip': '91.197.232.107'}
Line 6 match rule ssh_root: data: {'ip': '188.18.81.201'}
Line 9 match rule cross-site: data: {'ip': '195.95.147.212'}
Line 10 match rule cross-site: data: {'ip': '176.103.56.60'}
Line 12 match rule http503_GET: data: {'ip': '91.196.50.33'}
Line 13 match rule http503_GET: data: {'ip': '104.197.186.66'}
Line 14 match rule http503_others: data: {'ip': '104.197.186.66'}
Line 15 match rule http503_GET: data: {'ip': '80.172.244.226'}
Line 18 match rule http503_GET: data: {'ip': '201.83.32.27'}
Line 19 match rule http503_GET: data: {'ip': '201.83.32.27'}

I really insist that you spend lot of time to test your rules. Copy/paste lot of lines coming from your different logfiles here to assure that the behavior is correspond to what you expect.

To my personal taste, I avoid that a specific line could match several rules

accepted_ips

You can find this empty file. The goal is to list, one per line, the IP you will NEVER see added in one of the firewall tables.
I've done this during the test phases to avoid that log2table block myself.

How to start

Table definition

First verify that each table defined in you rules (rules.py) are correctly defined in your firewall.

For OpenBSD you should have something like this in your /etc/pf.conf:

table <bruteforce> persist

start

On OpenBSD you just have to use the rcctl commands

rcctl enable log2table
rcctl start log2table

On other systems, you can execute this:

/usr/local/bin/log2table

Few parameters are defined for log2table.
The most useful is the debug mode:

log2table -d

But you could also ask log2table to use a test parameter.py file with this:

log2table -c test_params.py

doas.conf

I suggest you to add the following in your /etc/doas.conf file

permit nopass <your user> as root cmd vi args /etc/log2table/accepted_ips
permit nopass <your user> as root cmd vi args /etc/log2table/parameters.py
permit nopass <your user> as root cmd vi args /etc/log2table/cmd_openbsd.py
permit nopass <your user> as root cmd vi args /etc/log2table/rules.py
permit nopass <your user> as root cmd vi args /usr/local/bin/log2table.py
permit nopass <your user> as root cmd vi args /usr/local/bin/log2table
permit nopass <your user> as root cmd rcctl args restart log2table
permit nopass <your user> as root cmd rcctl args stop log2table
permit nopass <your user> as root cmd rcctl args start log2table
permit nopass <parameters.py user> as root cmd pfctl

I'm authorising it in order to be able to update the required files, to start, stop and restart the rcctl.
The last line is request in order to let the userid defined in parameters.py to execute the pfctl command

syslogd of OpenBSD

We are on May 2017, and as of today syslogd provide several messages in /var/log/messages like: "last message repeated 2 times".
Because of this feature, a tool like log2table cannot react correctly in some situations.

Fortunately, on April, 17th the OpenBSD developers have foreseen a switch to add in your /etc/rc.conf.local that will allow you to disable to feature.