I’m attending a fantastic session at LinuxWorld today titled “Working with Log Files.” The session is being run by Mark Cohen (Quote.com, LookSmart, Penquin Computing) and Patrick McGovern (SourceForge.net and Splunk). The lab is designed for system administrators, developers and support people that want to learn more about how to use log files to troubleshoot IT infrastructures.In a three hour session Mark and Patrick are providing a series of hands on labs, each lasting about 15 minutes, ranging from LAMP stacks, syslog, Linux, VOIP and J2EE. The lab participants ssh over wireless to a central server in the room running 50 Linux sessions using VMWare. We’re using a variety of tools including Logwatcher, Swatch and Splunk Server to figure out what’s going on with each of our Linux sessions use the log files.The first lab involved ssh access to the VMWare server to get a virtual machine session and setting up and configuring syslog and syslogng. Syslog is the network logging workhorse of Linux and Unix systems. It is both a communications protocol as well as a set of actual programs and libraries. Syslog as opposed to syslogng is included in all Linux and Unix distros. Syslogng (next generation) is a newer and improved version of syslog adding better message filtering, better forwarding, message integrity and encryption and remote logging over TCP and UDP. – Syslog can take messages generated by operating systems, processes and applications and save the messages to a log file or other syslog server over the network. – Syslog can organize messages into categories according to facility and priority.– Syslog can display messages directly to the console. Syslog can be used for troubleshooting:– Routers, firewalls and devices during installation or in problem situations.– Intrusion detection. – Operations management.– Auditing.– Tracking user and administrative activities. One of the important things the lab reinforced is that setting up and deploying syslog requires some good thinking into your particular needs. Syslog can be set-up on specific services and devices in a number of different modes including:Device – generates messages (may be a program).Collector – receive and optionally store messages. Relay – receives and forwards messages.Sender – any program or device sending messages (device & relay).Receiver – any syslog program running who receives syslog messages (relay & collector). A typical syslog set-up involves one or multiple senders sending dta to a single, central syslog server. The server stores the received data usually all on one network. A larger syslog set-up can involves senders sending data to intermediary syslog servers. The intermediary servers receive, optionally filter and forward messages to a final syslog destination.Syslog has a number of fields that you should be familiar with including facility, severity, timestamp, host, tag and message. Not all of them are always present depending upon the implementation. The configuration can be controlled with syslog.conf file in /etc.Facility is a numerical indicator ranging from 0 to 23 indicating the sender of the message. 0 is traditionally used for kernel messages, 2 for email subsystem messages and 16-23 are reserved for your own customization. Severity is a numerical code from 0 to 7 with 0 indicating the level of the message. The lower the number the more important the message is deemed to be (0 is “emergency” and 7 is “debug”). We also learned in the lab that syslog timestamps are funny, containing no time zone, year or greater than 1 second resolution. Often too times in syslog can be out of sync unless your servers are syncing with a network time protocol like (NTP).The syslog host is the name or IP address of the sender of the messages. The intention is to provide the name of the original sender when passing through syslog relays. The syslog tag is a short ID meant to identify the process id of the sender. The syslog message contains the actual content of the message and can be highly non-structured data.Our presenters were informed syslog users and happily pointed out the short-comings. – Syslog is not the most reliable as it runs over UDP.– Syslog is not terribly secure as the sender can be faked and it is open to relay attacks.– The content is not standardized and there are a large variety of unstructured message contents. Words of wisdom are if you deploy syslog, spend some time thinking about how to do it for your environment. Syslogng offers a number of improvements over syslog and I’ll be talking in a next post about Part 2 of the lab covering the benefits of syslogng and some hands on experience with it.If you have questions or thoughts about syslog or syslogng email me at thebaum@splunk.com. Technology Industry