Dozens of the UK’s leading banks are today taking part in Operation Waking Shark 2, the largest ever stress test of the sector’s ability to cope with a major cyber-attack on markets and the networks that support them.

The all-day test, conducted from a single operations centre under the gaze of the Bank of England’s Prudential Regulation Authority (PRA) and the Financial Conduct Authority (FCA), follows on from a much smaller Waking Shark 1 of March 2011. That reportedly uncovered widespread confusion about processes and procedures before it had even considered the much thornier issue of the resilience of bank networks.

Part war-gaming penetration test, part a hard look at processes, everyone agrees that Waking Shark 2 is a step in the right direction. But is it enough of a step and is it the right kind of test? Industry opinion is surprisingly lukewarm.

“What will it practically achieve? What can really be achieved in 24 hours?,” asks a mildly sceptical Adrian Culley, a senior technical consultant for security appliance vendor Damballa.

An ex-detective at Scotland Yard’s old Computer Crime Unit, according to Culley Waking Shark’s biggest problem might be that it’s simply not big enough. Simply looking at banks was assessing only one part of the chain of vulnerability, he said.

Should it ever happen, a Waking Shark 3 should expand the stress-testing to ISPs and service providers, he believed.

“It’s a great start but what’s the next step? It is something that must be an ongoing programme because you can’t compress it all into 24 hours.”

Regulators, bank chiefs and government should also consider the everyday attacks that hit banks and their customers. Today, these were often ignored despite the threat they posed should they suddenlly escalate.

Another concern is that high-level stress tests ignore that future attacks might unfold in ways that take a long time to process. That the system is under attack might not be obvious from a the sort of high-level view taken in Waking Shark 2.

“In our experience, the majority of organisations that suffer a breach do not realise for some time that they have been hit, let alone where the attack originated from, and how it works,” commented Trustwave EMEA director, John Yeo.

“The more important issue is what are they communicating about, and what happens when an attack is more subversive, and not immediately obvious when it strikes.”

David Harley of software security firm ESET also worried about box ticking.

“There’s the risk that a simulation will play to strengths rather than weaknesses: after all, there can be a desire to demonstrate how effective your defences are, rather than display failure.

"What really tests an organisation’s security is a breach that couldn’t have been anticipated, the sort of attack that demonstrates how well (or badly) it can expect the unexpected,” he said.

Harley is right to point to outliers, and some commentators have pointed to completely unanticipated criminal attacks such as the foiled KVM raids of earlier this year as the sort of event that could slip under the radar.

However, those are small-scale compared to the sort of attacks predicted by KPMG almost as an aside in its recent report on the sector. That referenced the dramatic but under-reported ATM heist from last year in which large cyber-thieves in 27 countries were able to co-ordinate the theft of $45 million (£30 million) from cash machines.