Syslog Analysis
by Harry Hoffman
(hhoffman@ip-solutions.net)
Copyright (C) 2002
Update! Russell has just finished with a new version of
sl2, now called sl3 and also the patterns file
log.patterns .
Introduction:
In April of 2002 I went to work for the University of
Auckland in New Zealand. One of my first jobs was to create a syslog
infrastructure. After examining many different products we decided to select
syslog-ng . The primary reason for syslog-ng was that it allowed us to
create directories of hosts/log facility that would create new directories
based on the date. Our directory structure looks something like:
/var/log/HOSTS/host1.auckland.ac.nz/2002/08/10/kern/kern20020810
This worked great because it allowed us to perform recursive
greps through dates on a particular host or hosts.
Also, we didn't need to worry about rotating logs because
syslog-ng does it for us automatically based on a new day. Our syslog-ng.conf
file can be found here
. While this is a very simple example of a configuration file for syslog-ng
it is also very powerful.
We have found one problem: when the system is rebooted
or some of the substems log (i.e. -> SCSI) we get strange directories created
based on the messages delivered to syslog. If you've figured a way around
this I would appreciate any information you can provide in fixing the problem.
A good site to visit is Campin.net
. Nate Campi, the owner of the site, maintains the syslog-ng faq there
and is quite a good person about answering questions about syslog-ng.
Since Windows doesn't have a native syslog tool built in we chose to use
NTSyslog which is a Open Source
syslog tool.
Log Analysis Philosophy:
When analyzing logs I like to start off viewing everything
that is reported. While this can be a very time consuming way of doing
things there are benefits from this approach.
-
You'll learn alot about all of your systems and how they
report on things in a day to day manner
-
Having no ignore statements to begin with means you will
have to think about what you don't want to see
-
Pareto's law applies to ignore statements (20% of ignore
statements gets rid of 80% of the results)
Even though I log all items to a centralized server I also
keep the logs locally on the client. This way I can compare the files with
diff to make sure that the files are the same in content.
I always keep my log data on seperate disks than the
operating system and its applications, that way if the disks get filled
up it won't grind the syslog server to a halt.
If you are running a webserver you should also have its error logs sent to
the syslog server. In Apache you can accomplish this with the following
statement in your httpd.conf file:
ErrorLog "|/bin/logger -p local0.info -t APACHE_ERR_LOG"
The only services that I run on the log server are syslog,
stunnel, and ssh. The server is always kept up to date with the latest
patches and is not granted access to/from the outside world.
Analyzing System Logs:
Once we had the logging system up and running we decided
that an automated way of alerting based off of a pre-defined configuration
file would be much more appropriate than scanning the vast amount of logs.
The log.patterns file can be used to define the following: services, hosts,
email accounts, ignore statements, alert/attack staments, and warning statements.
I've always used logsentry
(formally logcheck) for this sort of task before. However logcheck involves
shell, perl, and a c program to get the job done. Also, I wanted to be
able to return reports to the individual system admins regarding their
systems. And finally logcheck isn't really written to recursively move
through directories and base it's reporting on many different hosts.
I started to write a perl program to incorporate all
of logcheck into one perl script. However time is a premium so I sought
the help of Russell Fulton, who is the security officer from the University
of Auckland, in getting the perl script written. Russell and I decided
on the ways that we would like to analyze the data and he quickly wrote
up sl2 . SL2 does
exactly what logcheck did but with a few great features including the following:
-
ability to recurse through directories
-
precompiled perl re's for faster pattern matching
-
configuration based on services and hosts
-
multiple runs of sl2 without updating the offset files (useful
for testing the log.patterns file)
-
email specific administrators based from configuration files
-
ability to utilize multiple configuration files
-
easily run sl2 on past days by specifying "x" days ago
-
send summary emails detailing each host and the number of
ignores, warnings, and alerts (Great for profiling of servers)
It shares a common configuration file called log.patterns
SL2 is still a work in progress but it does the job well for us. We use
it in combination with swatch to alert via pager/cell phone when something
really goes wrong. Here is a link to our swatchrc .
If you use sl2 please remember that we can offer no warranty
whatsoever and if it breaks your system it's not my fault. It's works pretty
well for us though.