If there's gold in log files, Splunk will help you to find it. Splunk bridges the gap between simple log management and security information and event management products from vendors such as ArcSight, RSA, Q1 Labs and Symantec.

Splunk lets you gather log data from systems and devices, and run queries on that data to find issues and debug problems. Splunk's capabilities also include reporting and alerting, pushing it ever-so-slightly into the world of SIEM.

What separates out Splunk from the world of Syslog servers and SIEM tools is Splunk Apps, a library of nearly 200 addons that make Splunk smarter about particular types of log information, change its look-and-feel or add new types of analysis.

Getting started with Splunk

There's a free version of Splunk for small and midsized deployments, so if your log files don't add up to 500MB each day, Splunk can be yours for the cost of the server you run it on. Some features, such as alerting, role-based access control and distributed searching are not available in the free version. You also can't run premium applications on top of the free version.

But Splunk is designed to scale up, way, way, up. With distributed search databases, role-based access control and the ability to eat terabytes of log data each day, Splunk is aimed at the large enterprise.

Splunk wants to be fed everything, including system, web, security and every other type of log or performance data you can find. We didn't want to go quite that large, so we tested using Splunk on our own small data centre, using live data.

Getting data into Splunk follows the same paths as any log management solution. We set up Splunk on a Linux system (Windows and other Unix flavours are also supported), a simple matter of an RPM installation, and had it listen for data sent to it with Syslog, probably the most common way to get your log data off systems and into an analysis tool.

For Windows systems, Splunk provides their "universal forwarder," an application that will pull Windows WMI data and forward it off to a Splunk server. The Universal Forwarder can also monitor file systems for changes and forward data from remote systems back to a central Splunk installation. We only used it to pull Windows event log information.

Splunk isn't too particular about where and how it gets data, with options for scripting and other network input sources.

Our initial contact with Splunk's input system, however, gave us a pretty good feel for Splunk's operational style. Splunk is not a do-it-yourself piece of open source software, but it also doesn't have the smooth polish we have seen from other commercial products. Splunk has an internal complexity that the Splunk team is happy to share with everyone through an extensive on-line documentation system.

If you want to make Splunk work, you've got to be ready to abandon the slick GUI and dive deep into difficult technical configuration, editing configuration files, writing regular expressions, and taking the time to understand where your data are coming from and how Splunk will see them.

We got Splunk working very smoothly in our multi-vendor environment, but only after investing serious effort in understanding how Splunk collects and indexes data. Our installation is slightly unusual, because we already have a central Syslog server and simply sent a copy of the data over to Splunk for indexing. But in these days of compliance and audits, having centralized Syslog for archiving purposes doesn't seem that unusual.

Overall, getting data into Splunk is much more of your typical open source experience, with a confusing maze of pointers, wikis, product tech notes and documentation, but backed up by Splunk's technical support staff. Plan on spending more than a few moments getting started.

Getting information out of Splunk

If getting information into Splunk takes a while, getting information out of Splunk is a breeze, and can be fun to boot. Splunk has intentionally copied the Google minimalist search bar, and to find information you just start typing into a large box, selecting a time range, and clicking the green "go" button.

Immediately, log entries that have the text you typed begin showing up, while the query continues to run in the background if you selected a particularly wide time range.

But this isn't just your normal text search. Start typing in the search bar and pause for a moment. Splunk creates a drop-down with the most frequently found terms in your logs that contain what you've typed so far, along with the frequency counts.

The other thing that Splunk has copied from Google is speed. This log search is fast. We tested Splunk for two months on a very modest hardware platform: a single core, 2.3 GHz speed virtual machine with 1GB of memory, dropping in about 30 million log entries. Every standard search, even using regular expressions, returned the first screen of data within one or two seconds. If you're looking for something, Splunk is not going to get in the way of finding it.

Not every search is lightning fast, some are merely speedy. For example, we ran a search asking for the most common access point names coming out of our Aruba wireless controller. That search took 19 seconds to summarise the 31,942 records from the Aruba controller, giving us the most common values sorted by frequency.