InfoWorld review: Better network security, compliance with log management

Log management is one of those necessary tasks that every company should do, but that few companies do consistently well. Collecting and analyzing computer and device logs can pay off in many areas, including information security, operations management, application monitoring, system troubleshooting, and compliance auditing. A good log management solution can help with any — or all — of these efforts.

Security auditing may be the No. 1 reason why many companies first investigate log management tools. Verizon’s “2008 Data Breach Investigations Report” [PDF], which is quickly becoming one of the most respected sources on computer crime statistics, said it best: “Evidence of events leading up to 82 percent of data breaches was available to the organization prior to actual compromise. Regardless of the particular type of event monitoring in use, the result was the same: Information regarding the attack was neither noticed nor acted upon.”

This review covers seven different hardware and software solutions for log management: ArcSight Logger 4.0, GFI EventsManager v.8.2, LogLogic MX3020 v.4.9.1, LogRhythm LR2000-XM v.5.0, NitroSecurity NitroView ESM and ELM v.8.4, Splunk 4.1.2, and Trustwave SIEM.

The goal of this review is to expose readers to a general cross-section of log management features and functionality, including what features set the different solutions apart. It’s important to note that while we rank each product across a common set of evaluation criteria (on a scale of 1 to 10, 10 being the highest), the products are often dissimilar to one another — they are often different classes of products.

For example, ArcSight’s single-appliance Logger is strictly a log management solution and therefore lacks a number of features found in NitroSecurity’s two-appliance SIEM (security information and event management) solution. My evaluation of both products — and all the others in this review — focused only on log management capabilities, and the product scorecards reflect only their log management features. I did not evaluate real-time event correlation — the hallmark of the SIEM solution — though I do note in the reviews and the product comparison table where those features are present. It’s usually a good thing when a solution offers more capabilities at a given price point.

The product features and functions I did evaluate are those related to collecting, storing, and reviewing the wide variety of event logs a company might want to watch closely. While you won’t need a complete and detailed understanding of log management to follow this product review, you might keep in mind the several distinct phases of the log management lifecycle: policy definition, configuration, collection, normalization, indexing, storage, correlation, baselining, alerting, and reporting. (You’ll find summaries of these phases in the sidebar, “Living the log management lifecycle,” and a more thorough treatment in my downloadable report, “Log Analysis Deep Dive: Finding Gold in Log Files.”) The specific product features I examined, and the most important differences among products in this category, are explored in the remainder of this article.

Testing was done in a small private lab with 15 to 20 computers (some physical, some virtual), mimicking a small-business network with Windows, Linux, BSD, routers, and wireless clients. At times, some of the functionality was viewed when the product was running on larger, real production networks or on a remote lab created by the vendor, when more clients better demonstrated a particular feature.

I did not test vendor performance or compression reports, both of which are often exaggerated. Some vendors felt this was unfortunate because one of their strongest claims of competitive advantage was in dealing quickly with huge amounts of data. We recommend testing real-life performance before buying any log management product. This author has seen many log management products perform well when handling a few hundred machines but slow to a crawl when handling a few thousand computers.

In a pleasant turn of events (excuse the pun), I felt all of the reviewed items were solid products ready to be deployed on any company’s network. Not one of the products tested would fail to provide value, although of course some would provide more value than others. Every reviewed product worked as advertised, had a myriad of useful features, and was mature enough to be used in a production environment. The top goal of this review was to highlight the features that made each product competitively distinct so that readers can decide which ones might make sense for testing in their environment.

Log management evaluation guide This section will discuss the various features available in each of the log management products tested and should help provide a framework for evaluating any other log management solution. For the seven products reviewed, the table below compares the key features to help in your evaluation.

One of the first decisions to be made is whether to use an all-inclusive appliance or a software-based product. Most log management products come as appliances simply because appliances typically handle the performance and storage requirements more easily than a software product running on a general-purpose operating system. Yes, it is true that administrators could configure and optimize a software product’s underlying host OS to be as efficient as an appliance — after all, an appliance is just an operating system host running log management software. With appliances, however, the hard configuration and optimization work is already done.

The downside of appliances is that they tend to be limited to a few off-the-shelf configurations and disk capacities, and their underlying operating system — often a Linux distro or Microsoft Windows — may be harder to patch. Although most of the appliance vendors in this review claimed to keep the underlying host patched and up-to-date as a part of their normal product upgrades (which are often automated), I found many products running older versions of code, such as the Apache Web server, with known vulnerabilities for which patches are available. If you decide to use an appliance, ask the vendor whether they update the underlying OS quickly when patches are available; if allowed under the licensing agreement, also consider testing the product for vulnerabilities before buying.

Workload distribution. Most of the products tested provided all-in-one functionality, meaning their product would act as management console, data collector, storage device, indexer (for search queries and filters), and report generator. In addition, most products could be configured to serve in just one role or multiple roles without performing all roles.

Workload distribution is incredibly important if you plan to collect log messages from more than a few hundred clients. Not that the log management tool itself poses a bottleneck — if an appliance, it will usually have four or more Gigabit Ethernet interfaces — but a network can sustain only so much additional traffic without causing application and operation performance issues. Sending log messages from 1,000 computers to a single log manager can bring any network to its knees.

Work with the vendor to figure out log management instance roles and distribution to maximize performance in your environment. Every product in this review can act as a store-and-forward collector for its own products, meaning you can have one instance collect all the local traffic before forwarding the data, often compressed, to a centralized, “master” log management instance. Many of the products could also forward events to other products, especially those that support syslog and SNMP. Several products, including both software and appliances, could act as collectors only or as indexers, which tend to be the two most CPU-intensive operations.

Give the vendor your network’s dimensions (network bandwidth, available capacity, and number of clients that will be monitored) and your log management plans. Then let the vendor respond with their recommended distributed configurations. With appliances, this will often result in different hardware models in different locations.

Performance is not only important in avoiding network congestion issues, but also when analyzing real-time or historical data, printing reports, and doing more involved forensic analysis. When you have tens of millions to billions of event messages to work with, you don’t want to be waiting 10 minutes for a simple query to return. If your solution involves multiple log management nodes, make sure that queries and reports can work across “peers,” meaning that one click in a management console will execute searches and reporting across all product instances. And test the performance differences and features when crossing peers. A few of the products have more limited features when searching across peers. All of the products tested are fairly flexible regarding workload distribution, with the lone exception being GFI EventsManager.

Most vendors will claim that they can work with environments of any size. And many vendors claim to have installed solutions handling tens of billions of messages per day, without client complaint. Ask for customer references, get any performance guarantees in writing, and test thoroughly before committing to a big-dollar purchase.

Management dashboard. Every log management product has management console dashboard that displays crucial real-time and short-term summary statistics about the log management system itself and the monitored events. Most dashboards include event messaging counts, local CPU performance, and notification about any critical events.

Almost all allow customization of the dashboard and let you configure what is shown by user or role. In most cases, but not all, dashboard displays are context-sensitive. You can click on a displayed graph to get a more detailed drill-down. A few, like NitroSecurity, allow extensive modifications where almost any metric, graph, or alert can be shown.

User roles are important, as most products allow administrators (which have full privileges) to set more limited roles. For example, some products allow limited admins to be defined, in cases where administrator-level privileges are needed but only regarding a predefined set of clients: all Windows machines, all Cisco routers, and so on. Most products have a read-only role where none of the configuration settings can be modified, but the users can run reports and see predefined graphs and metrics. Most of the products allow only two to four roles to be defined, and only allow administrators to define which screens are displayed. A few others, including Splunk, NitroSecurity, and LogLogic, allow extensive role definitions where each attribute and field on a screen can be defined per role.

Log collection. Collecting the log information from all the various monitored clients is the backbone of any log management product. Most products have both agentless methods and client agents to collect logs. Not having an agent means administrators don’t need to distribute, install, and configure additional software to every client. However, agentless log collection still requires planning and work. Most products collect logs using syslog forwards, WMI queries, or other remote methods (the last two usually require client administrative passwords). All require the necessary rule modifications if firewalls are involved. Whatever you do, don’t think of agentless as no work or it will surprise you.

Client agents have benefits that agentless collection methods have a hard time meeting. Most agents have multiple configuration options that allow administrators to have finer-grained control over what events are collected and how. For instance, instead of sending every log message to the centralized server, an agent can just send critical events, and store the rest locally for later retrieval if needed. Client agents can often offer transmission compression, allowing more events to be sent in less time and with smaller network bandwidth utilization, although it’s doubtful you’ll get the superior compression statistics that each vendor advertises in real-life scenarios.

Monitored clients can be added one at a time (usually via IP address or domain name), using mass importing (to add multiple devices at once), or using some sort of initiated querying process (usually through Active Directory browsing or IP address scanning). Most products allowed “device groups” to be created, to collect one or more monitored clients under a given group name as determined by some attribute — for example, by device type, IP address, or name. Device groups can then be monitored as a single entity making alerting and reports easier to accomplish when trying to focus on a particular device class.

Client agents can also be used to store events, in case the centralized log management tool is offline. One of the best features of the most sophisticated agents is in measuring network and/or local CPU utilization and throttling back the message send rate until the congestion clears up. Lastly, many agents have a “heartbeat” feature that will send warnings if the client has not transmitted messages in a certain time period, although this can be mimicked identically with “zero baseline” alerts as well server-side only. Not surprisingly, ArcSight, a longtime SIEM leader, has more client agents than any other competitor.

As covered above, the more parsed data a log management product has, the faster and more efficient the product can be when sifting through large amounts of data looking for a particular data interest. A big differentiator of products is how many parsers the product comes defined with. Some of the leaders, like ArcSight, are bundled with well over 100 data collectors defined. On the lower end, some products only have a few dozen parsers or will claim that their generic parsers are identical in efficiency. But in general, the closer the parsers mimic your environment, the better (but don’t let this be the sole decision point). Some log management products allow administrators to create their own parsers, which could prove very useful in many environments.

One relevant additional note: Most products claim to have Windows event log collection agents. However, many of these agents were made prior to Microsoft’s latest Windows versions and don’t have a good understanding and parsing of the more granular logs and views in these later operating systems. Many of the parsers and agents understand the three conventional default logs — Application, Security, and System — but cannot allow the administrator to choose from among the 100-plus built-in views that Windows Vista, Windows 7, and Windows Server 2008 provide.

Splunk is one tool that understands the new Windows log formats. However, I didn’t find a tool that works easily with the newer Windows’ built-in, event forwarding technologies (even if the product was hosted on a Windows OS and easily could use the newer technologies). Windows’ own event forwarding could be used in place of all the other agent and agentless methods. As with most product categories, log management hasn’t kept up with the latest client changes.

Log storage. Storing tens of millions to billions of messages takes a lot of disk space. Most appliances come with terabytes of disk storage in RAID configurations. Both software and hardware products claim to do some sort of storage compression, but as with transmission compression claims, take the vendors’ figures with a grain of salt. Their storage compression statistics are often based upon the smallest event log messages and the highest compression values, and don’t reflect real-world results.

Still, it’s important to find out from the vendor whether the product is software or appliance, what is the maximum disk space (or file size) the product supports, and in what configurations. What RAID arrays are supported? Different RAID configurations have different performance characteristics — that is, some are faster at writing and some are faster at reading — so flexibility is a plus. Does the vendor support digital signing of collected log data for attestation needs?

Most products have a maximum log size as well, having to do with limitations of the underlying host OS. If the product is an appliance, can the data be stored to external drive arrays? How much data can actively be indexed and easily retrievable? Every product allows data to be exported or archived. Exported data typically is kept offline and must be imported en masse to be searchable. A few solutions handle this more flexibly. For example, LogRhythm allows administrators to define a filter to import only the needed data instead of everything.

A few products have what is known as “storage groups,” which are individually defined logical partitions devoted to a particular task, such as PCI compliance, or a particular grouping of devices — for example, Cisco wireless routers. In addition to organizing a certain class of data for reporting purposes, storage groups can be used to make sure that a particular application has enough disk space to serve a particular policy requirement — for instance, save data for two years. ArcSight is especially strong in this area, with sizing parameters and CPU prioritization available.

Lastly, you’ll want to determine whether event log data is stored or archived in the vendor’s proprietary format, in its raw (unfiltered and unstructured) form, or in both? Most products store active data in a proprietary format, but archived or exported data remains in a raw format. This means that re-imported data will have to be parsed and indexed again to be useful, but it’s also easier to prove chain-of-custody concerns if that raw data (assuming it is also digitally signed) is later needed for legal reasons.

Real-time viewing. Most products allow real-time viewing of incoming data and show some top trends — often called tailing. If you have a system of any moderate size, with hundreds to thousands of messages coming in every second, real-time viewing of all data quickly loses its allure. All products allow real-time data to be filtered to show only relevant events for a particular interest. Often these filters can be saved to search historical data and produce related reports.

The best real-time viewers allow users to click on specific data fields to pivot to new views. For instance, maybe you’re viewing incoming data about a particular workstation and you see a suspicious TCP port. In some products, clicking on the port value could switch the current real-time view to show all workstations using the same port. Other products can only do this on historical data or require that you switch views into an “investigator” mode. All of the products tested provide pretty flexible viewing, though LogLogic and LogRhythm were strongest.

Searching stored data. Searching stored data for interesting patterns and events is an important part of log management, and an area where vendors strive to differentiate their products. Vendors will often tout how quickly their filtered searches work across very large amounts of data (although none of these claims were tested in this review). Most offer searches based upon keywords, English phrases, and Boolean logic. Some vendors force the user to type in all search expressions, while others also provide a graphical, pick-and-choose, “build a query” interface. Building a query click by click is helpful in teaching new administrators, although experienced admins almost always prefer the quickness and flexibility of a typed query.

If your organization needs to search a lot of raw, unstructured event logs, ask the vendor if they support search filters across non-normalized data. And if they do, how exactly can you search it, and how do searches of unstructured data differ from searches of structured data? Many vendors only allow keyword searches of raw data, whereas others allow Boolean logic.

Can searches be performed across peers? Among the products reviewed, only ArcSight, LogLogic, LogRhythm, and Splunk can execute searches across multiple nodes. All the products allow search filters to be saved. But the better products allow them to easily be turned into reports and saved for later use. Some products allow search filters to be sent to others and shared, which is particularly helpful in very large environments with many log reviewers.

It’s also good to have plenty of built-in, predefined search filters. Some products come with none or just a small sampling. The best products come with dozens of predefined, interesting queries, typically tied to one or more compliance objectives. The most common are for failed logons. A few products, including LogLogic, include “near context” queries that will show 10 or so events before and after a particular message you are interested in.

Alerting. Alerting is an important feature of log management and even more essential for SIEM. The vendor should support several different methods of alerting. All the products reviewed have email alerting, and most allow SNMP forwarding. Surprisingly, only a few have SMS alerting or allow analog modem dialing for pagers that lack an Internet interface. Some, including NitroSecurity, interface with common help desk software (usually Remedy) or have their own “help desk” function to help with responses. Most products allow unlimited alerting, but some, notably ArcSight Logger, only allow a limited number of active alerts — five in the case of Logger. ArcSight’s SIEM product has no such limitation.

Alerting comes in several forms. At the very least, alerting allows a notification to be sent if a particular log event is detected, and all products allow alerts to be based upon a certain number of events in a particular time period. One of my favorite alert types is the baseline alerting, in which the product itself determines the “normal” event patterns for the environment, while the admin determines the percentage of deviation to alert on. NitroSecurity supports baselining on every message type, whereas LogLogic’s baselining is limited to all messages coming from a particular device or set of devices.

Whatever log management product you choose, make sure it has the ability to throttle alert messaging. Nothing is worse than getting 100 alerts from a single event in the middle of the night.

Reports. All products come with built-in reports and allow reports to be customized or created. The best products come with hundreds of built-in reports, either free or for additional charge, relating to particular security or compliance needs: NERC, PCI, SOX, FISMA, and so on. Reports can usually be saved to a variety of formats — CSV, HTML, XLS, TXT, and sometimes PDF — run ad hoc, scheduled, and published to predefined file shares. The more built-in reports you have to work with, the better.

Be sure to test the reporting differences regarding structured versus unstructured data. Most vendors cannot easily handle unstructured data in reports or cannot provide the same summaries and counts as are normally available for structured data. Some vendors can incorporate unstructured data into reports only by including the complete raw message detail or very minimal summary counts.

In addition to middle and upper management reports regarding particular compliance initiatives, look for detailed reports that support technical troubleshooting. The products with the best reporting functionality, including ArcSight, LogRhythm, and NitroSecurity, meet both of these needs. Some vendors are operating on workflow processes where compliance reports can be sent up the chain of command and signed off by the necessary responsible parties. My advice is to find out what reports come built-in, what reports are available at additional cost, and to review all of them to see if they fit your compliance needs.

All seven products reviewed contained hundreds of features and proved immensely configurable, and every one represents a solid, well-thought-out solution to log management. I found myself really liking each product reviewed, only to be further impressed with the next product I tested. Read the accompanying product reviews, which highlight the significant differences, to find out which product most closely fits your environment. Then give it a detailed test-drive to measure suitability and performance.

If you aren’t using a comprehensive, enterprise-wide log management solution already, you have a number of excellent products to choose from. The best solutions give you only the alerts you require, filter out the noise, and provide useful dashboards and reports that you can tailor to your specific needs. The better you become at log management, the better equipped you’ll be to serve your company’s information technology needs, whether those relate to security, compliance, operations management, or virtually any other area of IT.

This article, “InfoWorld review: Meeting the network security and compliance challenge,” was originally published at InfoWorld.com. Follow the latest developments in information management and security at InfoWorld.com.