The majority of IT and business professionals in large companies are no more than somewhat confident their security systems can detect a threat before it becomes a real problem, a study shows.

About one in 10 of the 200 pros in Enterprise Management Associates' survey were neither confident nor doubtful that threats could be detected, while 7% reported more doubt than confidence. More than half were only "somewhat" confident at best.

The survey, released in May, found that most security pros in large companies were struggling to make sense of the log data gathered from security-related systems. A third of the respondents, all of who worked in companies with 1,000 employees or more worldwide, found it too difficult to distinguish legitimate from malicious activity, while almost three in 10 were equally successful as unsuccessful in correlating security data to business impact. Worse, 4% of the pros said they were mostly unsuccessful.

More than twice a month, almost six in 10 of the respondents have to devote unplanned time to respond to security incidents that occur outside normal investigative activities. A third are doing additional work at least every week, and about one in eight everyday.

The findings point to organisations being overwhelmed with the security data they currently collect. Almost 60% of the respondents knowledgeable about security log and event data management said they collect 50 gigabytes or more of data from routers, firewalls, gateways and other security-related systems each day. This translates into more than 166 million events daily, EMA said.

While this is already an overwhelming amount of information, almost three-quarters of the respondents said they would collect even more security-related data, or a wider variety of data, if they could make use of it.

So why would organisations want to build an even higher mountain of data? Study author and EMA researcher Scott Crawford said part of the reason is fear of the growing likelihood of an attacker's success, given the improvements in technology used to sniff out system vulnerabilities. "Organisations are recognising that attackers may be far more successful than we have been openly acknowledging," Crawford said.

A second factor is organisations know they have to do better with the data they collect, so they are exploring higher-performing analytical tools and techniques, which can process more information.

Finally, security management practices have been based in part on fear, uncertainty and doubt, which have left overtaxed organisations feeling like they need to do more. "Strategists would like to get a more objective handle on their (data) management priorities," Crawford said.

To shore up defences, roughly four in 10 of the respondents said they were spending more money on better security data management and analytic technologies. An additional 40% said they would spend more on similar technologies in the next one to three years.

The most popular technology was security information and event management, which was used by nearly eight in 10 of the respondents. Half said they were using data warehouses to store and analyse information, while nearly the same percentage was using analytical databases.

Business intelligence was used by 55% of the respondents, making it the most popular analytic tool. Other popular tools included analytic platforms, risk analysis or modeling and data mining.

Given the large amounts of data that has to be continuously analysed, almost four in 10 of the respondents used service providers for security data management.