This software serves two purposes.  First, it gives the ability to decode the syslog entries into a database which can be sorted, searched, and indexed in a more useful manner.  Second, it adds the ability to take custom data which is embedded within the Syslog message field and decode it into separate fields, each of which can be sortable.

The software consists of two pieces, although only the first one has been written to date:

•  A daemon which runs at system startup, and monitors all the files managed by syslog on the local machine (as defined in /etc/syslogd.conf), recording these records to the MySQL database(s).  Optionally, old data is deleted from the database(s) to prevent infinite growth.  Because of the way syslog-filter queries the log files, files are switched as appropriate when log rotation is performed by the system.
• A GUI program to display, constrain, and sort the data in the database.  Until this is written, any generic GUI program which supports MySQL can be used, although it might be a little difficult looking at the custom data fields.

As provided, this program supports the standard syslog and two custom example programs with a total of three custom data formats.

At the simplest, a database can be thought of as a series of rows of data, each of which contain columns of information of the same meaning.  Using this definition, the main table in our database, syslog, consists of rows of data (one row per syslog entry in the various text based syslog files), broken into the following fields:
 

Field Name	Field Description
File	The name of the textual log file where this message was found
Added	Date/Timestamp of when the record was put into the text file
User	The user name or system name of the creator of the log message
Program	The program that caused the message
Pid	The process ID number of the task which caused the message (if provided)
Message	The textual message from the log file if it was not a custom formatted message
SeqNum	A sequential number used to guarantee unique indexing and to associate this record with child records if there is custom format data
System	A flag that is set if the User field does not contain a local user (as verified by the password file)

Custom formats are represented by two tables.  The two tables have similar column names.  The first table will contain one row per custom data message from the syslog text files.  The second table contains one row, and contains textual labels of the columns for use in a GUI display program.

A final table contains one row per custom format, and has two columns: the names of the configuration and data tables.

For displaying the data, the primary table can be "joined" with subsequent custom data tables based upon the SeqNum field that ties the two tables together.

SUPPORTED PLATFORMS
 

Syslog-filter is being developed under Redhat 6.2 Linux on Intel x86 hardware.  Any platform which supports the same format /etc/syslogd.conf file, a Linux compatible syslog(3) API, and MySQL should be able to use this software without significant modifications.

DOWNLOADING AND BUILDING
 

Click here to download a released version of the software.

eventual FTP: ftp://download.sourceforge.net/pub/sourceforge/syslog-filter
eventual HTTP: http://download.sourceforge.net/syslog-filter

Only source releases are being provided, and are of the form syslog-filter-release.tgz, where release is the release number.

Create a directory, change to that directory, and extract the tarball.  The commands to extract the files from the bundle are:

mkdir syslog-filter-release
cd syslog-filter-release
gzip -d < ../syslog-filter-release.tgz | tar xpf -

Make sure that MySQL (the development environment) is installed on your machine.  You will need a database, the header files, and the libraries.  Refer to the LINK section below for a link to the MySQL site.

The file INSTALL within that tree describes how to build the software.  Essentially, you need to do the following steps to build the software:

Modify the root level Makefile as appropriate (there is a block towards the top of the Makefile for customizations).
make depend
make
Become root (if you aren't already)
make install
edit the /etc/syslog-filterd.conf file as appropriate per the instructions in the INSTALL file and for your connectivity information to the MySQL database.
Run the mysql command (as described in the INSTALL file) to create an empty database for the syslog-filter daemon. Do this for each database you are going to store the data in.
Modify the RC files on your machine to start /usr/sbin/syslog-filterd on system startup.

CVS
 

The CVS archive is being maintained at SourceForge, and can be accessed using the intructions provided at the following link: SourceForge syslog-filter CVS page

KNOWN BUGS AND LIMITATIONS
 

This is alpha level software.  As such, it will have some problems.  There are no currently known bugs.

Currently, the following limitations exist:

• The error output is inconsistent (this will be fixed in the 0.2 release).
• MySQL is the only database supported.
• Schema changes after creation of the database are not supported.
• The syslog.conf file supported is limited to the same format that is part of Redhat 6.2 Linux with the format restrictions as described in the README.TXT file as part of the syslog-filter distribution.  Only the syslog file as distributed in the Redhat distribution has been tested, with the addition of testing the machine redirection capability of syslog.
• Syslog-filterd and the MySQL engine have only been tested when they are on the same machine.
• Currently, output will only go to one database, eventually, support for output to multiple servers should be added (release 0.2 will support multiple databases with a slightly more powerful /etc/syslog-filterd.conf file format).
• When run, syslog-filterd backgrounds itself.  If the -d option is provided on the command line, it runs in the foreground.
• Only SIGINT and SIGTERM are managed (version 0.2 catches all catchable signals and logs errors for any received signals other than SIGINT and SIGTERM).
• No GUI program for display of data currently exists.
• The only system that the software has had any building or testing operations performed on is Intel x86 Redhat Linux 6.2.
• All program messages are US English only.
• The make files, although flexible, aren't neccesarily portable to other platforms without modification.  Autoconf should be used.
• Documentation is sparce, all information is currently in README.TXT within the distribution (0.2 will include manual pages for the configuration file and for the daemon itself).

ADDING NEW CUSTOM SYSLOG FORMATS
 
 

RELATED SITES
 

• General project info.
• Redhat
• Linux
• www.kernel.org
• MySQL
• SourceForge

LICENSE CHOICES
 

Some people in the open software community may question the use of the BSD License over the GNU License.  The BSD license was chosen over the GNU license for this project because of the need for people to be able to modify tables.c (and add custom source files for decoding) when adding custom log formats.  For commercial products to embed custom log messages in their diagnostics requires custom changes to this progam.  Since some companies won't necessarily want to have to open source their decoding algorithms, a GNU license would prevent use of this software in those environments.

WHY
 

Why write this software?  Why not use some other software which takes syslog data and shove it to a database?  Why not write a series of shell or perl scripts to manipulate syslog data directly from the flat file?  All valid questions.

At the last company I worked for, I had a need for some programs to log data which can be filtered upon and reported on.  Using syslog would have been perfect, except for the fact that there was no way to parse data embedded in the message field, without doing a lot of work.  Querying on the data also would have proven difficult.  Over the last 5 years, I have interviewed at a number of other companies doing systems or software on top of Linux or FreeBSD, all of which wanted to do the same thing.  Probably, in the future it would come up again, so, I figured, why not just write it and be done with it.

Using some other software would have addressed all the issues for sorting, and querying.  I was unable to find another program that provides a custom decode capability. short of modifying the /etc/syslogd.conf file to put each programs' data into it's own file.

Dave Gotwisner