On any day of the year, cyberattacks beyond counting are directed against an alarmingly large number of targets, be they individuals, large enterprises, or nation states. It’s a world we know almost nothing about unless a successful attack is detected after the event by which time response becomes a matter of damage limitation.

The popular way to get a handle on these 'unknowns' is to put up defences that are periodically stressed using penetration testing to approximate the way an attacker looks at a company’s systems from the outside. The limitation of this approach is that it is at best a snapshot in time and offers insight on little more than abstract vulnerabilities.

Hacker world

But what if the ‘intelligence gap’ between attacker and defender could be bridged using real-time data before an attack unfolds?

It sounds too good to be true on first hearing but that is precisely the concept small British startup Digital Shadows has pioneered since its founding by CEO Alastair Paterson and CTO James Chappell in 2011. The firm’s platform, SearchLight, is a database-driven ‘awareness’ system that searches 100 million Internet sources in 27 languages including social media, crime forums, GitHub, and even encrypted ‘dark’ nets such as Tor and I2P.

This chatter is gathered in an automated way and fine turned with the help of analysts into reports that build a picture of possible targeting at any point in time, be that hours, days or weeks in the future. It can also be used to uncover evidence of undetected past attacks when breached data is passed around within criminals circles.

The idea of trawling around the Internet and dark web looking for scraps of data isn’t new. Many security professionals will undertake this sort of research on their own initiative from time to time. However, SearchLight is a platform that removes both the effort and risks of such a task and does so in a way that will be more methodical and comprehensive than a manual search.

More typically, this sort of data is ignored by large enterprises that might be targeted because it’s too time-consuming to find and process assuming you even know where to look.

It’s a platform that could help re-define how organisations understand security intelligence gathering. When you reduce the concept to its bare essentials it sounds pretty extraordinary. Where digital forensics is a method for understanding an event after the fact in order to fine-tune future response or for compliance, the hunt for a ‘digital shadow’ is about looking for actionable intel.

In a sense, Digital Shadows is about getting ahead of the game. It stands or falls on a simple formula – if someone out there wants to attack a firm it is possible to get some pre-warning of that event before it happens and then understand what happened in more detail afterwards.

In the world of mathematics, differential equations fed enough variables will predict the future, up to a point. A digital shadow is something more analogue than that, more a hint or a connection that a particular type of attack is being undertaken against a sector, a country or, occasionally, a specific organisation or its executives.

Next: IG Group

Digital Shadows - IG Group

“It gives me visibility on hit words that I’m interested in. It sends alerts to my team in real time,” confirms Stefan Treloar, head of Information Security at spread betting company, IG Group, a Digital Shadows customer.

After using the system at his previous job at National Lottery firm Camelot, Treolar saw the relevance for IG Group, where he uses it to monitor groups or threat types he’s interested in. Every morning he and his team can study the dashboards they have set up, receiving an immediate alert if a particular type of threat against the company or the sector is detected.

This is a hugely complex task at some levels and includes the need to translate from languages other than English so that Treolar can make sense of what he is being told.

“It is giving me visibility into a world that is outside of my control. These types of solutions help you make informed decisions. There is quite a lot of chatter about financial institutions,” he says.

Treolar had been able to keep tabs on specific threat actors, fulfilling his belief in the importance to “know your enemy.”

“If someone was talking about us we are now in the best position we could be without us finding out about it through the Daily Telegraph.”

Next: In the beginning

Digital Shadows – in the beginning

Now up to a headcount of around 35 in Canary Wharf offices, the company was founded by Paterson and Chappell four years ago after packing in their day jobs to offer bespoke threat intelligence and risk reports for financial services. What has become the NoSQL-based SearchLight system emerged from expanding ambitions as the number of sources they were keeping tabs ballooned.

“I’ve always been interested in the idea of a digital footprint, of the traces we leave behind us when we use the Internet,” says Chappell.

“If you know what’s going on around you, if you can see how your brand is being affected by security actors and how they are being targeted then you’re going to be in a better position to defend yourself.”

Two rounds of seed funding were forthcoming 2011 and 2013 totalling $2 million (£1.3 million) – a good performance for the time - with a larger $8 million injection led by VC Storm Ventures earlier in 2015.

Digital Shadows’ involvement with Accenture’s FinTech Innovation Lab programme was particularly helpful, including getting them an office with a view in Canary Wharf. It gave them enough connections to find important test-beds with prospective customers.

“What was great about that was we got to work conjunction with some of those institutions and try out our ideas with them. We got a lot out of the process,” says Chappell.

The company has recently won a clutch of awards, including making it on to Informilo's Top 25 Hottest London Start-ups for 2015, and Best Cybersecurity Startup at the Tech Startup Awards. Numerous magazines have covered the firm in 2015 alone, a sign of growing attention but also rising expectations.

Those expectations include expanding in the US, extending the firm’s reputation among the UK arms of global financial services operations; in the wake of the cash injection Digital Shadows now has a second office in San Francisco.

That means taking on a host of other US-based security startups in an industry suddenly awash with cash and burn on a historically unprecedented scale. Despite the appearance of industry, security remains difficult, fragmented, competitive.

“Threat intel can mean a lot of things to different people. It tends to be one size fits all. That’s fine but cyber-situational awareness differs is it is about that specific organisations and assets and how weaknesses might be revealed in their security stance,” says Chappell.

Customers such as banks can find up to 60,000 ‘shadows’ per day according to Chappell so filtering these is key so that the customer isn’t over-burdened with noise.

The current state of development is to introduce even more automation and to scale the platform, adding new reporting features.

In a market becoming saturated with new methods of defence which can end up sounding increasingly like the old ones, Digital Shadows looks like a fascinating way of re-inventing the problem – the people who launch attacks know plenty about their targets so isn’t it time the defenders used the same outlook?

“You could argue we are in an asymmetric situation. For defenders, the odds are stacked against them,” muses Chappell. “It’s like shining a torch inside a cave. We are making those beams of light much broader and brighter.”