Search

logcheck: brilliantly simple log monitoring

July 19th, 2009 edited by Tincho

Article submitted by JP Vossen. DebADay needs you more than ever! Please submit good articles about software you like!

Logcheck is a simple yet great idea, an almost set-it-and-forget-it way to monitor your server logs for problems of all kinds. You create three pattern (grep regex) lists:

  • Known bad stuff
  • Looks bad but isn’t
  • Known good stuff

Logcheck periodically checks various syslog (or other) log files and picks up where it left off the last time. During each run it takes the new messages and looks for “known bad” things but first removes stuff that “looks bad but isn’t” and saves the messages as “this is known to be bad.” Then it rewinds, removes the known bad it just collected, removes the “known good” and stuff that “looks bad but isn’t” and saves whatever is left as “unknown.” Then it emails you the results.

Over time, as you tune your files, you end up only being alerted to known bad or new (not yet classified) stuff. Brilliant. I even did a (cheesy) Windows port of it.

Originally written by Marcus J. Ranum and Fred Avolio as frequentcheck.sh for the TIS Gauntlet firewall toolkit, it was adapted by Craig Rowland and applied to system logs. It spent some time as logsentry as part of Psionic’s Abacus/Sentry tools until they were bought by Cisco and the tools moved to SourceForge. The version in Debian is a re-write which was then inherited by Ubuntu.

But the best about the Debian/Ubuntu implementation is that almost all of the patterns you need are already Just There™. I usually only have to add a handful to work around odd things I’m doing or minor bugs. See the example at the bottom.

If you run a server you should be using fcheck and logcheck. And probably tmpreaper, etckeeper and maybe monit too. Articles about all these tools will be published soon, stay tuned!

As drawbacks, it should be noted that it may require some tuning, especially on a workstation or newer distro versions, and that may not be scalable for a lot of servers.

Is also worth mentioning that there are a variety of commercial and Managed Security Monitoring solutions that will scale and provide more information about events, but none are this easy.

Availability

The logcheck package is available in Debian since at least etch, and in Ubuntu since at least Dapper. See also the logcheck-database package.

Sample configuration

/etc/logcheck/ignore.d.server/LOCAL.ignore (lines wrapped for readability)

# /usr/sbin/logcheck automatically removes blank lines and comments.
# See 'man run-ports' for file name restrictions.

# For testing, create a sample log file and:
# su -s /bin/bash -c "/usr/sbin/logcheck -tsol sample" logcheck
# e.g.: su -s /bin/bash -c "/usr/sbin/logcheck -tsol /tmp/mylog" logcheck

# # DHCP Client lease renewals
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ dhclient: New
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ dhclient: DHCP(REQUEST|ACK)
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ NetworkManager:   DHCP daemon state
is now 3 \(renew\) for  interface

# # NTP, usually: 4001/0001
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync status
change [0-9]+

# # Syslog restarts (morning or all)
# ^\w{3} [ 0-9]{2} 07:[45][:0-9]{4} [._[:alnum:]-]+ syslogd
1\.5\.0#[0-9]ubuntu[0-9]: restart\.
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ syslogd 1.5.0#[0-9]ubuntu[0-9]: restart\.

# # fcheck
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ fcheck: “INFO: Rebuild of the fcheck
database  /var/lib/fcheck/fcheck\.dbf begun for [._[:alnum:]-]+ using config
file /etc/fcheck/fcheck\.cfg”

# # lm-sensors (normal)
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: \[[0-9. ]+\] CPU[01]:
Temperature/speed normal
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: \[[0-9. ]+\] Machine check events
logged

# # Wireless
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ NetworkManager:   \(eth1\):
supplicant connection state:

Posted in Debian, Ubuntu |

7 Responses

  1. Olivier Says:

    How does this compare to the logwatch package?

  2. martin f. krafft Says:

    Note that logcheck is in desperate need for helpers. It’s a great (because simple) tool, but the rule database has grown a bit out of control. If you feel you have the time and want to hack on this, don’t hesitate to write to logcheck-devel@lists.alioth.debian.org. We’ll gladly point you in the right direction and support you on the way.

  3. Sam Rowe Says:

    Tenshi > logcheck.

    Logcheck is a great way to get spammed to death with every single log entry generating its own mail. Tenshi has smart queuing and a whole host of better options.

    http://dev.inversepath.com/trac/tenshi

  4. Daniel Lowe Says:

    That’s a misrepresentation of logcheck, Sam. You can easily configure it to send out emails as often as you’d like.

    Martin’s right, though. It doesn’t really “just work” on many things. It’s particularly bad at filtering out amavis log messages, which don’t seem to follow much of a structure.

  5. leftystrat Says:

    After all this time, we’re still short a syslog GUI app that `just works’.

    I despise Vista and even KDE4 due to excessive bling but can’t we meet in the middle?

    Do I really need to learn regex?
    Is it just me?

    If I had any coding skills at all, I’d be all over this. Instead I’ll just bitch :)

  6. Mahmoud Says:

    Nice tool. A couple of extra tips for new users like me.

    Rule files with dots in their names will be ignored.

    You can test your rules by grepping syslog. The “README.logcheck-database.gz” file recommends doing so like this:

    sed -e ’s/[[:space:]]*$//’ /var/log/syslog | egrep ‘^\w{3} [ :0-9]{11} oempc wwwoffled\[[0-9]+\]: WWWOFFLE (On|Off)line\.$’

    Also it’s a good idea to test the whole “LOCAL” file as well:

    egrep -f /etc/logcheck/ignore.d.server/LOCAL /var/log/syslog

  7. zerodeux Says:

    I’ve tried to embrace logcheck early in 2004 (on Debian of course) on 4 servers, and finally got mad 4 months later. Rulesets were already long, hairy and mainly painful to debug and tune.

    I wrote a very simple tool which only list stuff you want to ignore. Those list I found to be 1/ subjective (depends on policy, usage, mood, etc), 2/ in dire need for easy and frequent maintenance. Having a black-and-white policy and roughly one file per service is a good start.

    http://home.gna.org/logalert/ is trivial to configure and maintain and comes with integrated options to debug your rulesets (answer in a few seconds to: which one is maching or NOT matching this message ? Re-run easily on whole logs or last batch to evaluate the noise level on different time span of your ruleset)

    There’s more doc, packaging and stuff than code (320 lines of Perl), good sign IMHO. I don’t maintain it much, I don’t use it anymore. I found out that log watching does not scale to lots of servers and is far less relevant than on-demand log analysis (and other good practices security-wise).

    The bundled ruleset of logalert is of the Sarge area, it’s just a useless example right now. And some project did reuse the name (and is hosted on SF), it’s not related though.