Syslog Analysis

by Harry Hoffman
(hhoffman@ip-solutions.net)
Copyright (C) 2002


Update! Russell has just finished with a new version of sl2, now called sl3 and also the patterns file log.patterns .






Introduction:

In April of 2002 I went to work for the University of Auckland in New Zealand.  One of my first jobs was to create a syslog infrastructure. After examining many different products we decided to select syslog-ng . The primary reason for syslog-ng was that it allowed us to create directories of hosts/log facility that would create new directories based on the date. Our directory structure looks something like:

/var/log/HOSTS/host1.auckland.ac.nz/2002/08/10/kern/kern20020810


This worked great because it allowed us to perform recursive greps through dates on a particular host or hosts.
Also, we didn't need to worry about rotating logs because syslog-ng does it for us automatically based on a new day. Our syslog-ng.conf file can be found here . While this is a very simple example of a configuration file for syslog-ng it is also very powerful.
We have found one problem: when the system is rebooted or some of the substems log (i.e. -> SCSI) we get strange directories created based on the messages delivered to syslog. If you've figured a way around this I would appreciate any information you can provide in fixing the problem.
A good site to visit is Campin.net . Nate Campi, the owner of the site, maintains the syslog-ng faq there and is quite a good person about answering questions about syslog-ng.
 

Since Windows doesn't have a native syslog tool built in we chose to use NTSyslog which is a Open Source syslog tool.



Log Analysis Philosophy:

When analyzing logs I like to start off viewing everything that is reported. While this can be a very time consuming way of doing things there are benefits from this approach.

Even though I log all items to a centralized server I also keep the logs locally on the client. This way I can compare the files with diff to make sure that the files are the same in content.
I always keep my log data on seperate disks than the operating system and its applications, that way if the disks get filled up it won't grind the syslog server to a halt.
If you are running a webserver you should also have its error logs sent to the syslog server. In Apache you can accomplish this with the following statement in your httpd.conf file:
ErrorLog "|/bin/logger -p local0.info -t APACHE_ERR_LOG"

The only services that I run on the log server are syslog, stunnel, and ssh. The server is always kept up to date with the latest patches and is not granted access to/from the outside world.
 

Analyzing System Logs:

Once we had the logging system up and running we decided that an automated way of alerting based off of a pre-defined configuration file would be much more appropriate than scanning the vast amount of logs.  The log.patterns file can be used to define the following: services, hosts, email accounts, ignore statements, alert/attack staments, and warning statements.
I've always used logsentry (formally logcheck) for this sort of task before. However logcheck involves shell, perl, and a c program to get the job done. Also, I wanted to be able to return reports to the individual system admins regarding their systems. And finally logcheck isn't really written to recursively move through directories and base it's reporting on many different hosts.
I started to write a perl program to incorporate all of logcheck into one perl script. However time is a premium so I sought the help of Russell Fulton, who is the security officer from the University of Auckland, in getting the perl script written. Russell and I decided on the ways that we would like to analyze the data and he quickly wrote up sl2 . SL2 does exactly what logcheck did but with a few great features including the following:


It shares a common configuration file called log.patterns SL2 is still a work in progress but it does the job well for us. We use it in combination with swatch to alert via pager/cell phone when something really goes wrong. Here is a link to our swatchrc

If you use sl2 please remember that we can offer no warranty whatsoever and if it breaks your system it's not my fault. It's works pretty well for us though.