Debian Package of the Day (static archived copy)

Logcheck is a simple yet great idea, an almost set-it-and-forget-it way to monitor your server logs for problems of all kinds. You create three pattern (grep regex) lists:

• Known bad stuff
• Looks bad but isn’t
• Known good stuff

Logcheck periodically checks various syslog (or other) log files and picks up where it left off the last time. During each run it takes the new messages and looks for “known bad” things but first removes stuff that “looks bad but isn’t” and saves the messages as “this is known to be bad.” Then it rewinds, removes the known bad it just collected, removes the “known good” and stuff that “looks bad but isn’t” and saves whatever is left as “unknown.” Then it emails you the results.

Over time, as you tune your files, you end up only being alerted to known bad or new (not yet classified) stuff. Brilliant. I even did a (cheesy) Windows port of it.

Originally written by Marcus J. Ranum and Fred Avolio as frequentcheck.sh for the TIS Gauntlet firewall toolkit, it was adapted by Craig Rowland and applied to system logs. It spent some time as logsentry as part of Psionic’s Abacus/Sentry tools until they were bought by Cisco and the tools moved to SourceForge. The version in Debian is a re-write which was then inherited by Ubuntu.

But the best about the Debian/Ubuntu implementation is that almost all of the patterns you need are already Just There™. I usually only have to add a handful to work around odd things I’m doing or minor bugs. See the example at the bottom.

If you run a server you should be using fcheck and logcheck. And probably tmpreaper, etckeeper and maybe monit too. Articles about all these tools will be published soon, stay tuned!

As drawbacks, it should be noted that it may require some tuning, especially on a workstation or newer distro versions, and that may not be scalable for a lot of servers.

Is also worth mentioning that there are a variety of commercial and Managed Security Monitoring solutions that will scale and provide more information about events, but none are this easy.

Availability

The logcheck package is available in Debian since at least etch, and in Ubuntu since at least Dapper. See also the logcheck-database package.

Sample configuration

/etc/logcheck/ignore.d.server/LOCAL.ignore (lines wrapped for readability)

# /usr/sbin/logcheck automatically removes blank lines and comments.
# See 'man run-ports' for file name restrictions.

# For testing, create a sample log file and:
# su -s /bin/bash -c "/usr/sbin/logcheck -tsol sample" logcheck
# e.g.: su -s /bin/bash -c "/usr/sbin/logcheck -tsol /tmp/mylog" logcheck

# # DHCP Client lease renewals
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ dhclient: New
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ dhclient: DHCP(REQUEST|ACK)
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ NetworkManager:   DHCP daemon state
is now 3 \(renew\) for  interface

# # NTP, usually: 4001/0001
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync status
change [0-9]+

# # Syslog restarts (morning or all)
# ^\w{3} [ 0-9]{2} 07:[45][:0-9]{4} [._[:alnum:]-]+ syslogd
1\.5\.0#[0-9]ubuntu[0-9]: restart\.
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ syslogd 1.5.0#[0-9]ubuntu[0-9]: restart\.

# # fcheck
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ fcheck: “INFO: Rebuild of the fcheck
database  /var/lib/fcheck/fcheck\.dbf begun for [._[:alnum:]-]+ using config
file /etc/fcheck/fcheck\.cfg”

# # lm-sensors (normal)
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: \[[0-9. ]+\] CPU[01]:
Temperature/speed normal
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: \[[0-9. ]+\] Machine check events
logged

# # Wireless
# ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ NetworkManager:   \(eth1\):
supplicant connection state: