Relatively few organisations are making good use of gobs of log data they collect for purposes such as identifying attackers, according to a survey of 600-plus IT professionals by security outfit SANS.

According to the SANS Analyst Program survey on log and event management, "Sorting through the Noise," 22% of respondents use a security information and event manager (SIEM) to collect and analyze data, while 58% use log-management systems, and the remainder rely on other means. Most respondents said one of the main reasons to collect logs is for the purpose of regulatory compliance, though 9% discounted the importance of that.

As in previous years that SANS has done this type of survey, virtually all the respondents said that "detecting and tracking suspicious behaviour was important." But according to SANS, there's evidence that insufficient time is being spent in actually analyzing the collected log data.

"The data suggests that respondents are having difficulty separating normal traffic from suspicious traffic," said Jerry Shenk, author of the SANS report "They need advanced correlation and analysis capabilities to shut out the noise and get the actionable information they need. But first they need to get more familiar with their logs and baseline what is normal."

The key issue in log analysis was cited to be "indication of key events from normal background activity" and "correlation of information from multiple sources." According to the survey, organisations are typically collecting log data from Windows and Unix-type servers, security devices, network equipment such as switches and routers, intrusion-detection systems and anti-virus and other security applications, and virtualised servers and hypervisors, as well as desktops and laptops.

Organisations want to detect suspicious activity but when the IT professionals were asked how much time they normally spend on log-data analysis, the largest group (35%) replied, "none to a few hours per week." As for the rest, 18% didn't know, 11% said one day per week, 2% outsourced this task to a managed security service provider, and 24% defined it as "integrated into normal workflow." The SANS survey report, which notes analysis time overall actually seems down from last year, noted that about 50% of the smaller organisations spent zero to just a few hours analising logs.

Overall, "that is really not very much time spent getting familiar with logs," the SANS report states. "Given the advanced threats they are struggling with, we would have expected the time organisations spend on log analysis to increase, not decrease. We cannot stress enough that the best way for organisations to quickly detect abnormalities is to gain understanding of their baseline or 'normal' activity by reviewing/analyzing log data on a regular basis."

The SANS report points out that "SIEM-type tools, including log management tools with analysis and reporting options, will help organise and identify patterns and activities that are generally recognised as indicators of problems. Yet, 58% of organisations are not anywhere close to that level of automation."

At the same time, the SANS report emphasised that automated tools cannot be viewed as a complete substitute for the people who are log analysts who develop a "sixth sense" about traffic anomalies and security because they spend some time every day looking at log data.

When it came to defining difficulties, trying to detect so-called "advanced persistent threat" attacks—APTS being the term to describe stealthy intrusions into the network to steal sensitive information---ranked as the toughest problem, according to the survey, with 85% this year reporting this as an issue in comparison with 65% last year.