Fail2ban: an enemy of script-kiddies
April 29th, 2007 edited by anaI bet there is only a little part of auth.log-aware GNU/Linux users, who has not experienced a pleasure of browsing thousand of lines of the failed authentication attempts. If you do not yet know what to look for in your auth.log, just run:
> zgrep 'Failed password for illegal user' /var/log/auth.log* | wc -l
On the system which I just tried, the result is 125835! since July of 2006. Yeah yeah — 99.999% of those failed logins are due to silly dictionary attacks, which (unfortunately) work in some % of the cases. Are you sure that your password and passwords of all the users on your system are strong enough to survive such an attack?
Also, I guess, there is a (hopefully small) group of system administrators, who experienced a pleasure of DoS attack on their services. Or web-server admins, who have a pleasure to stare at the attempts to access non-existing (most of the time) on the webserver /php/bla-admin.X.Y.bleh.
For both those groups (as well as for other problems too), there is a straightforward solution — just reject (or in other terms - ban) abuser’s IP as soon as you detect an attempt to get an unauthorized access to your box. Unfortunately, we do not stare at the log files 24×7, so we can not react in time. To substitute such a weak part of the chain in this process, i.e. a human operator, Fail2ban tool was created by Cyril Jaquier.
The idea behind Fail2ban is very simple: temporarily or permanently ban an IP which performed multiple undesired actions, such as unsuccessful authentication, access to restricted area, etc.
Originally it was developed to catch illegal SSH login attempts, but later on it grew up into an easily customizable toolkit for speedy reaction on some events (such as detected failed login attempts) recorded in the log files.
In the following sections I will describe a bit more of internals of Fail2ban configuration, but that knowledge is not really required to get the tool working for you. For that, it is sufficient to run “apt-get install fail2ban”. You might like to read the section on jails below if you simply want to enable some additional jails shipped with the Fail2ban package.
Debian/Ubuntu Presence of Fail2ban
Fail2ban is present in sarge from backports.org, and it is native to Etch and Sid. Sarge version in backports is from a 0.6 branch of the Fail2ban, and it has different configuration scheme than current 0.7 (soon 0.8) branch. 0.7 uses split configuration files and orthogonally separates notions of a filter (pretty much a python regular expression with associated set of files) and an action to be taken (banning via iptables/hosts.deny, or sending an email).
Fail2ban is also present in Ubuntu releases since Dapper release.
Configuration
Default configuration in both branches (0.6 and 0.7) enables ssh logins monitoring right out of the box, so no changes are necessary to get Fail2ban running.
If necessary, all changes in the configuration of Fail2ban 0.6.x have to be made in the original configuration file, and sections can be also enabled via command line switch (-e iirc) (N.B. this cmdline option is specific to Debian release of Fail2ban and is not present in upstream version). 0.7 branch uses completely different configuration scheme, and it is very convenient: any change or addition which has to be done in file /etc/fail2ban/X.conf can simply be made in file /etc/fail2ban/X.local — parameters in .local override ones in .conf. This way .conf file stays intact, and during your upgrade there is no necessity to mess with patching config files if they get changed upstream. Since I prefer 0.7 branch, I will describe details of its configuration.
As I mentioned above, 0.7 branch comes with an orthogonal configuration between filters and actions. A filter specifies what to
look for (like a ‘failed login attempt from …’ in auth.log, or a message ‘please brew some coffee, Mike’ in your .xchat/history/private.log), and an action describes possible scenario to play (to ban an IP, or to send an a single packet authenticator to a coffee maker to start brewing a fresh cup of coffee).
Filter
So here is an example of a filter:
> grep -v '^#' /etc/fail2ban/filter.d/sshd.conf
[Definition]
failregex = Authentication failure for .* from <HOST>
Failed [-/\w]+ for .* from <HOST>
ROOT LOGIN REFUSED .* FROM <HOST>
[iI](?:llegal|nvalid) user .* from <HOST>
ignoreregex =
“failregex” is a list of python regular expressions (with “” simply be a shortcut for “(?:::f{4,6}:)?(?P<host>\S+)” to match an IP or a host name. “ignoreregex” allows to infiltrate some false positives.
Standard sid Debian installation of Fail2ban comes with filters for various services (ssh, ftp, http), various implementations (exim, postfix; proftpd, pure-ftpd, wuftpd, etc), and for some additional events (normal illegal login in ssh vs DDOS attack on sshd).
If you want to write your own filter to store under /etc/fail2ban/filter.d/blah.conf, there is a very handy helper tool: fail2ban-regex, which can test your regular expression on the existing logfile and tell if you it works fine.
> fail2ban-regex /var/log/auth.log 'Failed [-/\w]+ for .* from <HOST>’
Running tests
=============
Use regex line : Failed [-/\w]+ for .* from <HOST>
Use log file : /var/log/auth.log
Results
=======
Failregex:
[1] Failed [-/\w]+ for .* from <HOST>
Number of matches:
[1] 2 match(es)
Addresses found:
[1]
69.115.175.240 (Sun Apr 01 23:58:20 2007)
69.115.175.240 (Sun Apr 01 23:58:27 2007)
Date template hits:
2 hit: Month Day Hour:Minute:Second
0 hit: Weekday Month Day Hour:Minute:Second Year
0 hit: Year/Month/Day Hour:Minute:Second
0 hit: Day/Month/Year:Hour:Minute:Second
0 hit: Year-Month-Day Hour:Minute:Second
0 hit: TAI64N
0 hit: Epoch
Success, the total number of match is 2
However, look at the above section ‘Running tests’ which could contain important
information.
Here instead of regular expression to test, you could simply provide the file of you tentative filter.
Action
A typical action for most of the cases would be to ban detected IP of an abuser using iptables, and that action is described in the following Fail2ban action definition:
> sudo grep -v '^#' /etc/fail2ban/action.d/iptables.conf
[Definition]
actionstart = iptables -N fail2ban-<name>
iptables -A fail2ban-<name> -j RETURN
iptables -I INPUT -p <protocol> –dport <port> -j fail2ban-<name>
actionstop = iptables -D INPUT -p <protocol> –dport <port> -j fail2ban-<name>
iptables -F fail2ban-<name>
iptables -X fail2ban-<name>
actioncheck = iptables -n -L INPUT | grep -q fail2ban-<name>
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j DROP
actionunban = iptables -D fail2ban-<name> -s <ip> -j DROP
[Init]
name = default
port = ssh
protocol = tcp
Default action in 0.7 branch of Debian package though is iptables-multiport, which can be used to ban multiple ports at once. Besides it, there are other actions available such as
- hostsdeny — ban using hosts.deny mechanism
- shorewall,ipfw — use firewall cmdline interfact to ban/allow an IP
- mail-* — email about the performed action to a sysadmin
Jail
And now we came to a point where both notions (filter + action) should be used together. “Jail” is the specification containing a filter and desired set of actions to be performed. Here is an example from original upstream version of /etc/fail2ban/jail.conf.
[ssh-iptables]
enabled = false
filter = sshd
action = iptables[name=SSH, port=ssh, protocol=tcp]
mail-whois[name=SSH, dest=yourmail@mail.com]
logpath = /var/log/sshd.log
maxretry = 5
In this example, the jail ssh-iptables defines the name of the filter to be used (so the full file name is implied to be /etc/fail2ban/filters.d/sshd.conf). Also it defines the list of actions to be performed: TCP port 22 has to be banned after 5 unsuccessful attempts, and an email has to be sent to yourmail@mail.com informing about such action.
While preparing Debian package of Fail2ban, I tuned up Debian-shipped version of jail.conf so that jail specifications becomes minimalistic, since most often all the jails should perform the same chosen action. If there is a need in a jail-specific action, it can always be specified in “action” parameter of the jail. The same jail in Debian-shipped jail.conf looks like
[ssh] enabled = true port = ssh,sftp filter = sshd logpath = /var/log/auth.log maxretry = 6
Since the rest of the jails present in jail.conf are not active by default, desired jails can easily be enabled in /etc/fail2ban/jail.local. Here you can see a part of my locally customized jail.local:
[DEFAULT] bantime = 3600 destemail = root@localhost banaction = shorewall action = %(action_mwl)s [apache] enabled = true maxretry = 4 [sasl] enabled = true [courierauth] enabled = true # custom jail which used to be not present in shipped jail.conf [apache-noscript] enabled = true port = http filter = apache-noscript logpath = /var/log/apache*/*error.log maxretry = 6
Screenshots
April 29th, 2007 at 6:30 am
Note that a similar effect can be achieved much more efficient using the “recent” match available in Iptables.
iptables -A FORWARD -i ethLRZ -p tcp –dport 22 -m state –state NEW -m recent –set –name SSH
iptables -A FORWARD -i ethLRZ -p tcp –dport 22 -m state –state NEW -m recent –update –seconds 60 –hitcount 5 –rttl –name SSH -j DROP
These two iptables statements will rate-limit incoming SSH connections to 5 in a one minute window. Regular users will rarely ever trigger this limitation, but being inside iptables makes it react *much* faster than fail2ban.
I’ve been successfully using this to deter scanners for both SSH and http://FTP. After their fifth try they are blacklisted for some time. In my experience they always give up.
With fail2ban, they ususally managed to do several dozens of tries before they were blocked.
Using iptables directly also uses much less CPU and memory. (Not that fail2ban would be expensive, though)
April 29th, 2007 at 8:59 am
Nice, I didn’t knew that one yet :-)
Will try it for sure
thx
April 29th, 2007 at 12:12 pm
Perhaps the most useless piece of software ever. As the ssh login attemps cause extremely small network traffic, cpu usage, or risk (like if any of those dictionary attacks would ever succeed.. furthermore openssh is quite solid on its own), all this does is adds unnecessary components to the system.
April 30th, 2007 at 12:59 am
donalde: I totally disagree. As the number of users on a box gets larger, some of them WILL have weak passwords!
April 30th, 2007 at 8:40 am
oh man, I spent hours a while back writing this guide http://blog.2blocksaway.com/2007/01/03/iptables-and-portsentry-the-dynamic-duo-how-to-dynamically-block-ips-and-unblock-them-automagically/
in order to achieve something similar with portsentry and iptables. Except it does it on IPs that portscan you , blocks teh ip and after some time unblocks it. I like fail2ban though this looks nice…
//flosse
http://blog.2blocksaway.com
April 30th, 2007 at 12:02 pm
Did you create the map automatically from banned IPs? It looks very cool, can you please give a description if so?
reply#1: I am sorry — I didn’t understand the question at first ;-) too many ‘maps’ these days within any programming language syntax ;-) GeoIP Map image was generated by a friend of upstream author — I inquired on details and will report as soon as I know
reply#2: unfortunately map was created ad-hoc/manually. I guess it is time for some nice tool working in conjoint with geoip…
April 30th, 2007 at 12:28 pm
joeyo: that’s why you enforce password policy (for smaller environments) or roll up smartcards (for larger setups).
April 30th, 2007 at 2:16 pm
First of all I want to express “Thanks” to ‘ana’ editor of debaday for shaping the article up to debaday formatting standard.
Quick replies to the raised questions/issues:
1. IPTables way: operating just on NEW packets would allow abuser to perform DoS attack on some IP by spoofing packets with the victim IP. It is probably possible to implement proper rate banning operating on RELATED packages. Besides that rate-limiting is not appropriate for some environments when it is needed to react depending on internal structure of the package. match target in iptables comes handy but it is yet not widely accepted and some higher level firewalls (eg shorewall) do not provide relevant semantics. And once again - operating on packet level has to be taken with additional caution to prevent DoS attacks
2. @donalde about ‘useless’: dictionary attacks not only impose security risk but are terrible ‘log file spammers’ — they make it much harder to keep an eye on the activities
3. @coNP: fail2ban keeps a list of banned IPs which can be easily obtained per each jail using “fail2ban-client status” command. Also you are welcome to add any ‘action’ on banning, such as updating some file or LDAP/DNS server with the abuser IP.
April 30th, 2007 at 2:32 pm
@flosse: I used to use portsentry myself ;-) BTW — Cyril (fail2ban upstream author) promised to implement matching based on multiple lines. Then it would make it very easy to construct rules based on the log entries of SNORT. That would allow to ban not only evil portmappers but all kinds of other beasties.
April 30th, 2007 at 6:13 pm
Whats wrong with pam_abl?
answer: hopefully nothing wrong with it — it does its job — but it works only for PAM based authentication… there seems to be no temporary banning… you can’t use it for some ftp servers/mail servers/etc which natively do not go through pam. fail2ban provides generic framework which can be used for any service which reports abnormal activity to a log file.
May 1st, 2007 at 1:16 pm
Note that before going into Fail2ban, just check if you can simply afford to disable clear password authentifcation with SSH: this is a simple as “UsePAM no” in the default Debian config. Suddenly everything is simple and secure as managing /home/*/.ssh/authorized_keys files.
Remember, passwords are obsolete. I manage to have computer-illiterate clients to use public key cryptography without much hassle (that would be puttygen+winscp for those MS users).
May 1st, 2007 at 3:38 pm
A comparison with the denyhosts package would be informative. It’s clearly simpler (works “out of the box”) but also less flexible. Is it any less secure?
May 1st, 2007 at 6:44 pm
A much simpler solution to stopping script-kiddies from trying to brute force ssh is to not run ssh on port 22.
If you do that, feel free to continue to use fail2ban, but you will find that it won’t be blocking anyone anymore.
comment by debianATonerussian.com: indeed it is a fine solution for a desktop box especially since you can define ports per host/network under .ssh/config… actually myself I am using knocking to get access to ssh and proxy for my own use. BUT such solution does not work for common services such as SMTP/HTTP/etc whenever port assignment is fixed and assumed by the others — and fail2ban is meant to be used not only for ssh ;-)
May 1st, 2007 at 11:22 pm
donalde: Smart-cards, for something on the Internet? Huh? My keyboard has a smart-card reader, but how many users have that at home? I would personally use two-factor authentication, which will solve this issue completely.
Small environments can use strong password rules. Even the best type of authentication (i.e. Kerberos) doesn’t work with weak passwords.
August 8th, 2007 at 9:51 pm
but why let ssh open to the wild, when openvpn is so easy to set up?
fail2ban is however extremely useful for apache/web app attacks (if you have a web server open to the wild you know what I mean ;) )
September 17th, 2007 at 3:59 am
@donalde - and what about all the other internet services, passworded or not, which invite zillions of attempted connections? I for one will probably implement this for my mail gateway as another method of blocking zombie spammers (without it getting as far as Postfix to deal with).