commit eba4e6499bc66c97c4bdfeee6917624003a31d83
from: vi <vincent_delft@yahoo.com>
date: Sat May 27 17:47:39 2017 UTC

initial commit

commit - /dev/null
commit + eba4e6499bc66c97c4bdfeee6917624003a31d83
blob - /dev/null
blob + 04128fca0f4fb5b8fb0f3af32424c3055d06fea6 (mode 644)
--- /dev/null
+++ LICENSE
@@ -0,0 +1,15 @@
+ * Copyright (c) 2017 Vincent Delft <vincent.delf@gmail.com>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
blob - /dev/null
blob + 007e618b8493611cfc06ea9a1f07a27b8c70c129 (mode 644)
--- /dev/null
+++ Makefile
@@ -0,0 +1,33 @@
+# log2table - update table of your firewall
+#
+
+# Customize below to fit your system
+# paths
+PREFIX = /usr/local
+ETC = /etc/log2table/
+
+install: 
+	@echo == Modifying the log2table.py BASEDIR ==
+	sed -i "s:BASEDIR.*=.*:BASEDIR = \"$(ETC)\" :" log2table.py
+	sed -i "s:export PYTHONPATH=.*:export PYTHONPATH=$(ETC) :" log2table
+	@echo == Installing config files to ${ETC} ==
+	install -d ${ETC}
+	install -C -b rules.py ${ETC}
+	install -C -b parameters.py ${ETC}
+	install -C -b cmd*.py ${ETC}
+	install -C -b accepted_ips ${ETC}
+	@echo == Installing executable file to ${DESTDIR}${PREFIX}/bin ==
+	install -d ${DESTDIR}${PREFIX}/bin
+	install log2table.py ${DESTDIR}${PREFIX}/bin
+	install -m 0700 log2table ${DESTDIR}${PREFIX}/bin
+	@echo == Installing init script in /etc/rc.d ==
+	install log2table.rc /etc/rc.d/log2table
+
+
+uninstall:
+	@echo == Removing executable file from ${DESTDIR}${PREFIX}/bin ==
+	rm -f ${DESTDIR}${PREFIX}/bin/log2table
+	@echo == Removing init script in /etc/rc.d ==
+	rm -f /etc/rc.d/log2table
+
+.PHONY: install uninstall 
blob - /dev/null
blob + bb8ea2fb247c45d7efade06f210c001843ae560e (mode 644)
--- /dev/null
+++ README.md
@@ -0,0 +1,193 @@
+# Features
+
+Log2table allows you to continuously monitor your logfiles. You can trigger actions when a specific message comes in your audited logfiles or when a specific number of occurences are present. 
+I've developed it as a simple Intrusion Detection System and to let away bad guys of my web servers. For example, those who are trying passwords on ssh, those who are trying cross site htpp requests, those who are looking for wp-admin.php, ...
+
+The first match for such task was fail2ban. But I was facing some difficulties to configure it to my specific needs. After having developed a minimalist script ([here](http://www.vincentdelft.be/post/post_20161106)), a reader of my blog has started to develop something around this idea ([Vilain](https://yeuxdelibad.net/Blog/?d=2016/12/26/16/17/21-un-fail2ban-pour-openbsd-vilain)). After several changes of his code, he informs me that I'm deviating too much from his initial idea and inform me that he will no more merge my changes in his git repository.  So I rename it log2table, mainly because this application build a link between logiles and firewall tables. 
+
+I've build log2tble to be as flexible as possible. It should be easy to adapt to other firewalls (iptable with ipset). It could eventually be configured to execute other tasks than add/remove IP from a firewall table.
+
+The code is split in 4 parts:
+
+- the main loop: log2table.py
+- the rules: they should all be in rules.py
+- the parameters requested by the rules: parameters.py
+- the command file: cmd_<platform>.py
+
+The last 3 files are available in /etc/log2table and can be adapted to your specific needs. 
+
+Rules are indenpendent of the logfiles you are tracking. Log2table consolidates the actions triggered by source IP. Indeed, a specific machine (source IP) could generate entries in your http log file, but also in your authlog file, or in your /var/log/messages file. Whatever this IP is doing on your machine will be tracked. You can imagine to apply the same rule for different actions in different logfiles. In such case you have what I call, a consolidation of actions; this will generate a consolidated repsonse. 
+
+For flexibility reasons, several parts of the program are available to the users in /etc/log2table by default. To adapt those config files a minimum set of skills of Python is required. Parameter.py is a simple Python dictionary where you can define which logfile you want to follow and which rule you want to trigger once the regex rule match a line in your tracked logfile. The rules are also available to the end-users in rules.py. Each classes in rules.py allow you to define the behavior to adopt when a line match your regular expression. 
+
+Currently I propose 2 rules. One, called Hacker, to ban unwanted connections. This rule adds the IP having performed an unwanted connection into an OpenBSD PF table. This rule remove those "bad IPs" after 1 hour at least. Inside my pf.conf file, I block all connections for IP contained in this table. 
+
+As example, here the rule I've added in my pf.conf file:
+
+    block in quick proto tcp from <bruteforce> to any
+
+The other class in rules.py is called WifiTemp. This class registers each IP on their first connection by watching the dns.log file they are using (imagine they are forced to use this dns server). After 1 hour their IP is added in a PF table where PF block their traffic. After again 1 hour the rule remove IP from the PF table, so they can re-used the Wifi. As you can understand the goal is to provide a Wifi connection for 1 hour, after it's blocked for one hour at least, after it's available for an another 1 hour, etc...
+This rule is just for demonstration purposes, it shows the flexibility those rules can offer.  
+
+Each rule can have several parameters coming either from the regex, either provided by the parameters.py file. 
+
+Feel free to share you rules, I'll include them in the next releases :-)
+
+
+
+# Pre-requisites
+You just need to have python3.6 installed. 
+
+This application has been developed on OpenBSD, but with very few adaptations it should work on other systems.
+Normally you should only modify the cmd_<platform>.py file. 
+
+In this cmd_<platform>.py file you must make sur that the userid defined (in parameters.py) can run the required commands. On OpenBSD we use **doas**. On other system you could use **sudo**. Thanks to assure that this use can run the command you define in this command file. 
+
+
+# Installation
+Just untar the package and run 'make' as root. 
+
+	tar -xzvf log2table.tgz
+	cd log2table
+	make
+
+This will install log2table in /usr/local/bin
+and configuration files in /etc/log2table. 
+
+Feel free to adapt Makefile if you want to use other directories. 
+
+If you are not on OpenBSD, please remove from Makefile the line
+
+	install log2table.rc /etc/rc.d/log2table
+
+log2table.rc is a file required for the [rcctl](http://man.openbsd.org/rcctl) command. 
+
+
+# Configuration
+
+## Options
+log2table offers few options:
+
+- --debug: allow you to run in debug mode and be much more verbose
+- --config: allow you to load an another parameter.py file. This is useful in combination with the debug mode to test some features
+
+## parameters.py
+We have some global parameters like:
+
+- 'user' : The userid on which you log2table will run, after the startup.
+- 'logfile': the log2table logfile
+- 'sleeptime': number of seconds you want to "sleep" before log2table check, for new data, the logfiles you want to track
+- 'rules': the rules you plan to use in the sections bellow. This information will allow log2table to load IP addresses already existing in the firewall's tables. Each Rule, in rule.py, has a table's name. 
+
+
+
+For each log file you want to follow, you must provide several informations:
+
+- One regex patterns: 'regex'. This is the main element on which you will trigger some actions. The regex can return one or several information. The name of those values must be in accordance to what the Rule expect.
+- One rule name: 'name'. This is a text value describing the name of the rule. This name will be present in the log of log2table.  
+- The name of the rule to use: 'rule'. This is a string value where you have to provide the name of the rule present in the rules.py file. 
+- Some parameters: 'params'. Those parameters will be provided to the rule. Here again be careful that the provided parameters are compliant with what the rule expect. Those values are extended to the parameters provided by your regex pattern. Be careful to not overwrite some values coming from your regex.
+
+## rules.py: The hacker rule
+Currently you have 1 Rule class called: Hacker. This class ban IP having crossed a weight of 100. The weight must be provided in your logfile's definition.
+This class expects to have the "IP" values coming from your regex.
+Moreover you have to define a weight for each regex pattern.
+In the parameters.py I propose you, someone trying ssh with root userid will receive a weight of 50. Someone trying ssh with another userid will receive a weight of 30.
+As consequence if a hacker try 2 times the root account, he will be added to the banned table of the firewall. If he tries once the root and 2 other users he will be banned too. If he tries 4 others users he will be banned too.
+ 
+ 
+## parameters.py: Test your regex
+
+You can test your regex by copy pasting the targeted line in a file called "test.txt". Then you can simply execute your parameters.py file. 
+
+	python3.6 parameters.py
+
+Lines started by "#" will be skipped. 
+
+You will get an output like this:
+
+	Line 2 match rule ssh_invaliduser: data: {'ip': '91.197.232.107'}
+	Line 5 match rule ssh_invaliduser: data: {'ip': '91.197.232.107'}
+	Line 6 match rule ssh_root: data: {'ip': '188.18.81.201'}
+	Line 9 match rule cross-site: data: {'ip': '195.95.147.212'}
+	Line 10 match rule cross-site: data: {'ip': '176.103.56.60'}
+	Line 12 match rule http503_GET: data: {'ip': '91.196.50.33'}
+	Line 13 match rule http503_GET: data: {'ip': '104.197.186.66'}
+	Line 14 match rule http503_others: data: {'ip': '104.197.186.66'}
+	Line 15 match rule http503_GET: data: {'ip': '80.172.244.226'}
+	Line 18 match rule http503_GET: data: {'ip': '201.83.32.27'}
+	Line 19 match rule http503_GET: data: {'ip': '201.83.32.27'}
+
+
+I really insist that you spend lot of time to test your rules. Copy/paste lot of lines coming from your different logfiles here to assure that the behavior is correspond to what you expect. 
+
+To my personal taste, I avoid that a specific line could match several rules
+
+## accepted_ips
+
+You can find this empty file. The goal is to list, one per line, the IP you will NEVER see added in one of the firewall tables. 
+I've done this during the test phases to avoid that log2table block myself. 
+
+
+# How to start
+
+## Table definition
+
+First verify that each table defined in you rules (rules.py) are correctly defined in your firewall. 
+
+For OpenBSD you should have something like this in your /etc/pf.conf:
+
+	table <bruteforce> persist
+
+
+## start
+On OpenBSD you just have to use the rcctl commands
+
+	rcctl enable log2table
+	rcctl start log2table
+
+On other systems, you can execute this:
+
+	/usr/local/bin/log2table
+
+
+Few parameters are defined for log2table. 
+The most useful is the debug mode:
+
+	log2table -d
+
+But you could also ask log2table to use a test parameter.py file with this:
+
+	log2table -c test_params.py
+
+
+
+# doas.conf
+
+I suggest you to add the following in your /etc/doas.conf file
+
+	permit nopass <your user> as root cmd vi args /etc/log2table/accepted_ips
+	permit nopass <your user> as root cmd vi args /etc/log2table/parameters.py
+	permit nopass <your user> as root cmd vi args /etc/log2table/cmd_openbsd.py
+	permit nopass <your user> as root cmd vi args /etc/log2table/rules.py
+	permit nopass <your user> as root cmd vi args /usr/local/bin/log2table.py
+	permit nopass <your user> as root cmd vi args /usr/local/bin/log2table
+	permit nopass <your user> as root cmd rcctl args restart log2table
+	permit nopass <your user> as root cmd rcctl args stop log2table
+	permit nopass <your user> as root cmd rcctl args start log2table
+	permit nopass <parameters.py user> as root cmd pfctl
+
+I'm authorising it in order to be able to update the required files, to start, stop and restart the rcctl. 
+The last line is request in order to let the userid defined in parameters.py to execute the pfctl command
+
+
+# syslogd of OpenBSD
+
+We are on May 2017, and as of today syslogd provide several messages in /var/log/messages like: "last message repeated 2 times". 
+Because of this feature, a tool like log2table cannot react correctly in some situations. 
+
+Fortunately, on April, 17th the OpenBSD developers have foreseen a [switch](https://github.com/openbsd/src/commit/cf679774bcb9854e93e3d778e764574171a29b23) to add in your /etc/rc.conf.local that will allow you to disable to feature. 
+
+
+
+
blob - /dev/null
blob + a41f7932929595b4b05cc6b29eb1a02ff23bb3f6 (mode 644)
--- /dev/null
+++ TODO
@@ -0,0 +1,2 @@
+Adapt and test it on Linux and iptables/ipsets
+Define other Rules and/or regexp
blob - /dev/null
blob + e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 (mode 644)
blob - /dev/null
blob + a37118d350e95f07cdbbccdb8f84898468ee6f2e (mode 644)
--- /dev/null
+++ cmd_openbsd.py
@@ -0,0 +1,38 @@
+# log2table - update table of your firewall
+import subprocess
+
+#This is the file containing the commands to your firewall
+#please edit it and adapt the command to your specific system
+#rename the file cmd_<platfrom.system>.py
+
+def get_ip(table, logger):
+    #return a list of all IP currently banned by the firewall
+    try:
+        ret = subprocess.check_output(["doas","pfctl", "-t", table, "-T", "show"])
+        ret = ret.decode('utf-8')
+    except:
+        logger.warning("Failed to run doas pfctl -t {} -T show".format(table))
+        ret = ""
+    return ret.split()
+
+def ban_ip(ip, table, logger):
+    #add an IP input the log2table_table of the frewall and return the output of the command or None in case of failure
+    try:
+        ret = subprocess.check_output(["doas","pfctl", "-t", table, "-T", "add", ip],stderr=subprocess.STDOUT)
+        ret = ret.decode('utf-8').strip() 
+    except:
+        logger.warning("Failed to run doas pfctl -t {} -T add {}".format(table, ip))
+        ret = None
+    return ret
+
+def unban_ip(ip, table, logger):
+    #remove an IP from the log2table_table of the frewall and return the output of the command or None in case of failure
+    try:
+        ret = subprocess.check_output(["doas","pfctl", "-t", table, "-T", "delete", ip],stderr=subprocess.STDOUT)
+        ret = ret.decode('utf-8').strip() 
+    except:
+        logger.warning("Failed to run doas pfctl -t {} -T delete {}".format(table, ip))
+        ret = None
+    return ret
+
+
blob - /dev/null
blob + fb59a90431c797e1b884a6e9b62304f6f5330b6b (mode 644)
--- /dev/null
+++ log2table
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+export PYTHONPATH=/etc/log2table/ 
+if [ "$*" = "" ]; then
+   python3.6 /usr/local/bin/log2table.py &
+else
+   python3.6 /usr/local/bin/log2table.py $*
+fi
blob - /dev/null
blob + be5a986c50b7c1f3d5f7df00fa3f147ce815a878 (mode 755)
--- /dev/null
+++ log2table.py
@@ -0,0 +1,261 @@
+#!/usr/local/bin/python3.6
+# -*- coding:Utf-8 -*- 
+
+
+"""
+Author :      Vincent <vincent.delft@gmail.com>
+              Thuban <thuban@yeuxdelibad.net>
+Licence :     MIT
+Require : python >= 3.5
+
+Description : This tool update "tables" of your firewall
+              currently works on openBSD
+              Inspired from https://yeuxdelibad.net/Blog/?d=2016/12/26/16/17/21-un-fail2ban-pour-openbsd-vilain
+
+              In pf.conf, add : 
+                    block quick from <log2table_bruteforce> 
+
+/*
+ * Copyright (c) 2017 Vincent Delft <vincent.delf@gmail.com>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+"""
+
+import sys
+import os
+import re
+import logging
+import subprocess
+import asyncio
+import time
+import imp
+import platform
+
+
+VERSION = "0.5"
+
+def load_ignore():
+    to_ignore = []
+    try:
+        content = open(CONFIG.DEFAULT['ignore_ips'], "r").read()
+    except:
+        logger.exception('Failed to to read the ignore file (%s)' % CONFIG.DEFAULT["ignore_ips"])
+        content = ""
+    for line in content.split('\n'):
+        if line.strip() and line[0] == "#":
+            pass
+        else:
+            if line.strip():
+                to_ignore.append(line.strip())
+    logger.info("We will ingore the following elements: {}".format(to_ignore))
+    return to_ignore
+
+
+
+class Log2table():
+    def __init__(self):
+        logger.info('Start log2table version {}'.format(VERSION))
+        self.loop = asyncio.get_event_loop()
+        self.ip_seen_at = {}
+        self.load_bad_ips()
+        self.to_ignore = load_ignore()
+        
+        for logfile in CONFIG.PARAMS.keys():
+            logger.info("Start log2table for {}".format(logfile))
+            asyncio.ensure_future(self.check_logs(logfile,CONFIG.PARAMS[logfile]))
+
+
+    def load_bad_ips(self):
+        for rule_txt in CMD.DEFAULT['rules']:
+            rule_obj = eval("rules." + rule_txt)
+            table = rule_obj.table
+            ips = CMD.get_ip(table, logger)
+            for ip in ips:
+                rule = rule_obj("", logger, ip)
+                rule.set_from_table()
+                self.ip_seen_at[ip] = rule
+                logger.info("Load from {}, {}".format(table, rule.display()))
+        
+                
+
+    def start(self):
+        try:
+            logger.info('Run forever loop with sleeptime of {} sec'.format(CONFIG.DEFAULT['sleeptime']))
+            self.loop.run_forever()
+        except KeyboardInterrupt:
+            self.loop.close()
+        finally:
+            self.loop.close()
+
+    async def check_logs(self, logfile, params):
+        """
+        worker 
+        """
+        if not os.path.isfile(logfile) :
+            logger.warning("{} doesn't exist".format(logfile))
+        else :
+            # Watch the file for changes
+            stat = os.stat(logfile)
+            size = stat.st_size
+            inode = stat.st_ino
+            mtime = stat.st_mtime
+            RE = {}
+            for param in params:
+                regex = param['regex']
+                RE[param['name']] = re.compile(regex)
+            while True:
+                await asyncio.sleep(CONFIG.DEFAULT['sleeptime'])
+                stat = os.stat(logfile)
+                if size > stat.st_size and inode != stat.st_ino:
+                    logger.info("The file {} has rotated. We start from position 0".format(logfile))
+                    size = 0
+                    inode = stat.st_ino
+                if mtime < stat.st_mtime and inode == stat.st_ino:
+                    logger.debug("{} has been modified".format(logfile))
+                    mtime = stat.st_mtime
+                    with open(logfile, "rb") as f:
+                        f.seek(size,0)
+                        for bline in f.readlines():
+                            line = bline.decode().strip()
+                            logger.debug('line:{}'.format(line))
+                            for param in params:
+                                ret = RE[param['name']].match(line)
+                                logger.debug("name:{}, ret:{}".format(param['name'],ret))
+                                if ret:
+                                    data = ret.groupdict()
+                                    try:
+                                        if data['ip'] not in self.to_ignore :
+                                            logger.debug('line match {} the {} rule'.format(data['ip'], param['name']))
+                                            asyncio.ensure_future(self.treat_ips(line, {'ip' : data['ip'], 'data': data, 'params': param }))
+                                        else:
+                                            logger.info('line match {}. But element in ignore list'.format(data[param['key']]))
+                                            logger.info('line was:{}'.format(line))
+                                            logger.info('rule was:{}'.format(param['name']))
+                                    except:
+                                        logger.exception('Matching failure')
+                                        logger.warning("Values returned by your regex are:{}".format(data))
+                size = stat.st_size
+
+    async def treat_ips(self,line, ip_item):
+        logger.debug('treat_ips started')
+        try:
+            ip = ip_item['ip']
+            data = ip_item['data']                
+            parameters = ip_item['params']
+            parameters.setdefault('params',None)
+            data.update(parameters['params'])
+        except:
+            logger.exception('Paramters failure')
+        logger.debug("data:{}".format(data))
+        try:
+            rule_obj = eval("rules." + parameters['rule'])
+            rule = rule_obj("", logger, ip)
+        except:
+            logger.exception("Failed to create the object")
+            logger.info("data:{}".format(str(data)))
+            logger.info("line:{}".format(line))
+        self.ip_seen_at.setdefault(ip,rule)
+        try:
+            ret = self.ip_seen_at[ip].update(data)
+        except:
+            logger.exception("Failed to update the object")
+            logger.info("data:{}".format(str(data)))
+            logger.info("line:{}".format(line))
+        logger.info("We have treated, name {}, {}".format(parameters['name'], self.ip_seen_at[ip].display()))
+        #we check which element could be removed
+        to_remove = []
+        for recorded_ip in self.ip_seen_at.keys():
+            try:
+                self.ip_seen_at[recorded_ip].refresh()
+            except:
+                logger.exception("Failed to refresh the object")
+                logger.info("line:{}".format(line))
+            if self.ip_seen_at[recorded_ip].status == "to delete":
+                to_remove.append(recorded_ip)
+        for ip in to_remove:
+            self.ip_seen_at.pop(ip)
+        #for debugging, this line allow us to see if the script run until here
+        logger.debug('treat_ips end:{}'.format(self.ip_seen_at))
+
+
+
+
+
+def main():
+    os.chdir(os.path.dirname(os.path.abspath(__file__)))
+    v = Log2table()
+    v.start()
+    return 0
+
+if __name__ == '__main__':
+    import argparse
+    parser = argparse.ArgumentParser(description="Log2table add or remove elements from your firewall's tables")
+    parser.add_argument('--debug','-d', action="store_true", help="run in debug mode")
+    parser.add_argument('--conf','-c', nargs="?", help="location of the config files (parameters.py and cmd_<platgform>.py)")
+    parser.add_argument('--version','-v', action="store_true", help="Show the version and exit")
+    args = parser.parse_args()
+    if args.conf:
+        BASEDIR = args.conf
+    else:
+        #we import DEFAULT and PARAMS
+        BASEDIR = "/etc/log2table/" 
+    CONFIG = imp.load_source("*",BASEDIR + "parameters.py")
+    
+    # Configure logging
+    logger = logging.getLogger(__name__)
+    logging.basicConfig(filename=CONFIG.DEFAULT['logfile'],
+                        format='%(asctime)s %(module)s:%(funcName)s:%(message)s')
+    logger.setLevel(logging.INFO)
+    if args.debug:
+        print("run in debug")
+        logger.setLevel(logging.DEBUG)
+        ch = logging.StreamHandler(sys.stdout)
+        logger.addHandler(ch)
+    
+    logger.info("  ")
+    logger.info("{} started".format(sys.argv[0]))
+    if args.version:
+        print("Version: ", VERSION)
+        sys.exit(0)
+
+
+    #we first load parameters.py, create the logfile object, then switch the userid
+    try:
+        os.setuid(CONFIG.DEFAULT['user'])
+    except:
+        logger.exception('Failed to set uid to {}. Please verify it'.format(CONFIG.DEFAULT['user']))
+        sys.exit(1)
+    logger.info("switch to userid: {}".format(CONFIG.DEFAULT['user']))
+
+    logger.info("Rules loaded:{}".format(str(CONFIG.DEFAULT['rules'])))
+    
+    #Load command specific functions
+    plform = platform.system().lower()
+    try:
+        CMD = imp.load_source("*",BASEDIR + "cmd_{}.py".format(plform))
+    except:
+        print("ERROR to load cmd_{}.py. Check your log file".format(plform))
+        logger.exception("Error to load cmd_{}.py".format(plform))
+        sys.exit(1)
+    
+    #Load rules and assign it to the rules obects
+    import rules
+    rules.cmd_to_ban=CMD.ban_ip
+    rules.cmd_to_unban=CMD.unban_ip
+    
+    main()
+
+
+# vim: tabstop=4 expandtab shiftwidth=4 softtabstop=4
+
blob - /dev/null
blob + 7e94e1203318dd88a0e84ec50341b01e9a36955d (mode 755)
--- /dev/null
+++ log2table.rc
@@ -0,0 +1,13 @@
+#!/bin/sh
+#
+# $OpenBSD: log2table.rc,v 1.0 2017/03/28 17:58:46 Thuban$
+
+daemon="/usr/local/bin/log2table"
+
+. /etc/rc.d/rc.subr
+
+pexp=".*log2table\.py"
+rc_reload=NO
+
+rc_cmd $1
+
blob - /dev/null
blob + a0a8d6617a442fb5cdb5ef16ced5c1b533c73e8d (mode 644)
--- /dev/null
+++ parameters.py
@@ -0,0 +1,103 @@
+# log2table - update table of your firewall
+
+DEFAULT = {
+'logfile': "/var/log/log2table.log",
+'rules': ['Hacker'],  #Hacker is a class defined in rules.py. 
+                      #Hacker is using the table called "bruteforce"
+                      #hacker requires "ip" in the regex and "weight" in the params
+
+'user': 1001,      #Numeric id of the user you want to use for this daemon 
+
+#The ignore_ips file contains all IP you will NEVER block
+#To respect the format the file must contain one IP per line
+#Comment line are allow, but the first character must be a #
+'ignore_ips': '/etc/log2table/accepted_ips',
+
+#duration between each file update check
+'sleeptime': 1,
+}
+
+
+
+PARAMS = {
+        #name of the log file
+        '/var/log/authlog':
+        [
+            {
+                #May 12 22:08:42 myvultr sshd[42196]: Failed password for root from 190.51.62.216 port 48498 ssh2
+                'name':'ssh_root',  #Name fo the rule
+                'regex':'.* Failed password for root from (?P<ip>\S+) .*', #each regex must at least create a ip parameter
+                'rule': "Hacker",
+                'params': {'weight':50} #each rule can have his own weight
+            },
+            {
+                'name':'ssh_others',
+                'regex':'.* Failed password for (?!root)\S+ from (?P<ip>\S+) .*',
+                'rule': "Hacker",
+                'params': {'weight':35} 
+            },            
+            {
+                #May 12 20:40:06 myvultr sshd[96263]: Failed password for invalid user alex from 188.16.70.24 port 48325 ssh2
+                'name':'ssh_invaliduser',
+                'regex':'.* Failed password for invalid user .* from (?P<ip>\S+) .*',
+                'rule': "Hacker",
+                'params': {'weight':20} #each rule can have his own weight
+            },
+        ]
+    ,
+        #name of the log file
+        '/var/www/logs/access.log':
+        [
+            {
+                'name':'wp-login',
+                'regex':'.*wp-login.php.* (?P<ip>\S+)$',
+                'rule': "Hacker",
+                'params': {'weight': 100} #each rule can have his own weight
+            },
+            {
+                #default 176.103.56.60 - - [13/May/2017:02:06:46 +0200] "GET http://ya.ru/ HTTP/1.1" 400 0
+                'name': 'cross-site',
+                'regex':'\S+ (?P<ip>\S+) .*] \"GET http://.*',
+                'rule': "Hacker",
+                'params': {'weight': 100}
+            }
+        ]
+    ,
+        '/var/log/messages':
+        [
+            {
+                'name':'http503_GET',
+                'regex':'.* pound: .* e503 .* \"GET .* from (?P<ip>\S+) .*',
+                'rule': "Hacker",
+                'params': {'weight':20}
+            },
+            {
+                'name':'http503_others',
+                'regex':'.* pound: .* e503 .* \"(?!GET).* from (?P<ip>\S+) .*',
+                'rule': "Hacker",
+                'params': {'weight':50}
+            },
+        ]
+    }
+
+
+
+
+
+
+if __name__=="__main__":
+    import re
+    #You can build your test file in order to assure your rules are correct
+    #To avoid hasardous problems, you should avoid that a line match several rules
+    data = open('test.txt').readlines()
+    i = 0
+    for line in data:
+        i += 1
+        for logfile in PARAMS.keys():
+            for elem in PARAMS[logfile]:
+                reg = re.compile(elem['regex'])
+                ret = reg.match(line)
+                if ret:
+                    print("Line {} match rule {}: data: {}".format(i, elem['name'], ret.groupdict()))
+
+
blob - /dev/null
blob + 8775959793551dcf24b3e476d5a025d6f297ca09 (mode 755)
--- /dev/null
+++ restart.sh
@@ -0,0 +1,4 @@
+#!/bin/sh
+
+doas pkill -9 -f log2table.py
+doas python3.6 log2table.py &
blob - /dev/null
blob + 29a378663243a3c9385ef2f3ee91e8333b5de95a (mode 644)
--- /dev/null
+++ rules.py
@@ -0,0 +1,139 @@
+# log2table - update table of your firewall
+import time
+
+#Those parameters must be set in log2table.py
+def cmd_to_ban(ip, table, logger):
+    return "cmd_to_ban to implement"
+
+def cmd_to_unban(ip, table, logger):
+    return "cmd_to_unban to implement"
+
+class Generic:
+    """
+    Gereric rule class for log2table. 
+    Please sub-class it to match you need. 
+    In any case, objects having a status="to delete" will be removed by log2tail from his list of ip to watch
+
+    """
+    table = "bruteforce" #Do not forget to add this in your pf.conf file: table <bruteforce> persist
+                         #If you are not using the OpenBSD's firewall, please refer to your manuals 
+                         #for the exact command required to create this table
+    def __init__(self, status, logger, ip):
+        self.status = status
+        self.logger = logger
+        self.time = time.time()
+        self.ip = ip
+    def update(self, data):
+        "This method is used because we have received a new instance of this ip"
+        pass
+    def to_ban(self):
+        pass
+    def to_unban(self):
+        pass
+    def refresh(self):
+        "This method is to regulary check if we should not stop the tracking on this ip"
+        pass
+    def display(self):
+        return ""
+    def set_from_table(self):
+        "This method is to setup paramters for an ip already present in the firewall"
+        pass
+
+class WifiTemp(Generic):
+    """
+    Block some ip addresses for 1 hour after 1 hour of serfing. 
+        statuses: working, banned, to delete, exception
+    """
+    table = "wifitemporary" #Do not forget to add this in your pf.conf file: table <wifitemporary> persist
+                         #If you are not using the OpenBSD's firewall, please refer to your manuals 
+                         #for the exact command required to create this table
+    def __init__(self, status, logger, ip):
+        super().__init__("working", logger, ip)
+        self.duration = 10
+        values = ip.split('.')
+        ipdec = int(values[0])*256*256*256 + int(values[1])*256*256 + int(values[2])*256 + int(values[3])
+        self.status = "exception"
+    def update(self, data):
+        if self.status == "exception":
+            return 
+        if time.time() - self.time < self.duration:
+            #we keep it running
+            pass
+        if (time.time() - self.time) > self.duration and (time.time() - self.time) < 2*self.duration:
+            self.to_ban()
+    def to_ban(self):
+        if self.status != "banned":
+            ret = cmd_to_ban(self.ip, self.table, self.logger)
+            self.logger.info("{} added in table {}, return code:{}".format(self.ip, self.table, ret))
+            self.status = "banned"
+            self.logger.info(self.display())
+        self.status = "banned"
+    def to_unban(self):
+        if self.status != "to delete":
+            ret = cmd_to_unban(self.ip, self.table, self.logger)
+            self.logger.info("{} removed from , return:{}".format(self.ip,self.table, ret))
+        else:
+            self.logger.info("{} silently removed".format(self.ip))
+        self.status = "to delete"
+    def refresh(self):
+        if self.status == "exception":
+            return 
+        if (time.time() - self.time) > 2*self.duration:
+            self.to_unban()
+    def display(self):
+        return "{}: {}".format(self.ip, self.status)
+    def set_from_table(self):
+        if self.status == "exception":
+            return 
+        self.status = "banned"
+        self.time = time.time() - self.duration
+        
+class Hacker(Generic):
+    """Block port scanners
+        Behaviour:
+            Each ip matching a specific regex will receive an associate weight for his attack
+            As soon as this ip has reach the limit of a weight of 100, he will be added in the firwall table
+            After at least one hour, this ip will be removed from the firewall table
+        Important parameters:
+            table: is the generic one. Do not forget to add this table in your firewall config file
+            status: can be  "working", "banned", "to delete"
+        Required parameters:
+            ip: the ip address of the hacker we want to ban
+            weight: the weight given to a certain attach
+    """
+    def __init__(self, status, logger, ip ):
+        super().__init__("working", logger, ip)
+        self.counter = 0
+        self.threshold = 100
+        self.duration = 3600
+    def update(self, data):
+        weight = data['weight']
+        self.counter += weight
+        self.time = time.time() #we re-initialize it as soon as we receive instances for this ip
+        if self.counter >= self.threshold:
+            self.to_ban()
+    def to_unban(self):
+        if self.status == "banned":
+            ret = cmd_to_unban(self.ip, self.table, self.logger)
+            self.logger.info("{} removed from {}, return:{}".format(self.ip, self.table, ret))
+        else:
+            self.logger.info("{} silently removed".format(self.ip))
+        self.status = "to delete"
+    def to_ban(self):
+        if self.status != "banned":
+            ret = cmd_to_ban(self.ip, self.table, self.logger)
+            self.logger.info("{} add in {}, return code:{}".format(self.ip, self.table, ret))
+            self.status = "banned"   #we change teh status so that the display just after 
+            self.logger.info(self.display())
+        self.status = "banned"
+    def refresh(self):
+        if time.time() - self.time > self.duration:
+            self.to_unban()
+    def display(self):
+        return "{}: {}, counter:{}".format(self.ip, self.status, self.counter)
+    def set_from_table(self):
+        self.status = "banned"
+        self.counter = self.threshold
+
+
+    
blob - /dev/null
blob + b0fe0943b67d5b456a1c49aed2ebfcdb666f888e (mode 644)
--- /dev/null
+++ test.txt
@@ -0,0 +1,21 @@
+/var/log/authlog
+May 12 20:35:34 myvultr sshd[96221]: Failed password for invalid user 0 from 91.197.232.107 port 60572 ssh2
+May 12 20:35:34 myvultr sshd[96221]: Connection closed by invalid user 0 91.197.232.107 port 60572 [preauth]
+May 12 20:35:35 myvultr sshd[6190]: Invalid user 0000 from 91.197.232.107 port 49020
+May 12 20:35:35 myvultr sshd[6190]: Failed password for invalid user 0000 from 91.197.232.107 port 49020 ssh2
+May 12 20:25:02 myvultr sshd[94214]: Failed password for root from 188.18.81.201 port 55884 ssh2
+
+/var/www/logd/access.log
+default 195.95.147.212 - - [12/May/2017:23:12:00 +0200] "GET http://ya.ru/ HTTP/1.1" 400 0
+default 176.103.56.60 - - [13/May/2017:02:06:46 +0200] "GET http://ya.ru/ HTTP/1.1" 400 0
+
+/var/log/messages from pound
+May 12 21:46:26 myvultr pound: (1eb71bdc838) e503 no service "GET http://testp3.pospr.waw.pl/testproxy.php HTTP/1.1" from 91.196.50.33 testp3.pospr.waw.pl
+May 12 22:35:15 myvultr pound: (1ec0704de38) e503 no service "GET / HTTP/1.0" from 104.197.186.66 -
+May 12 22:35:15 myvultr pound: (1ec0704d438) e503 no service "HEAD / HTTP/1.0" from 104.197.186.66 108.61.209.198
+May 12 22:48:31 myvultr pound: (1ec0704dc38) e503 no service "GET / HTTP/1.0" from 80.172.244.226 -
+May 12 23:48:19 myvultr pound: (1ebaabcf238) error read from 35.184.73.193: Operation timed out
+May 13 01:03:14 myvultr pound: (1eb33d61038) error read from 35.184.73.193: Operation timed out
+May 13 03:19:49 myvultr pound: (1eb754c7438) e503 no service "GET /hndUnblock.cgi HTTP/1.0" from 201.83.32.27 108.61.209.198
+May 13 03:19:49 myvultr pound: (1ec022b0838) e503 no service "GET /tmUnblock.cgi HTTP/1.0" from 201.83.32.27 108.61.209.198
+