A number of commentators, including Accenture, rightly point out that the advent of so-called "Big Data" technologies doesn’t change the need for our trusty relational databases and our well-curated data warehouses.
Quite the contrary - the increasing availability and usability of unstructured (sensor, text, audio, image, etc.) data makes the enterprise’s existing investments in managing its structured, typically transactional data even more valuable and will drive continued investments in those platforms.
Now that we all agree that enterprise structured data and newer sources of data, that are often less structured, will coexist and be integrated, we can focus on the real disruptive innovation that these big data technologies share: linear scalability.
While most of the talk around big data technologies focuses on the "big", that focus distracts us from the key disruptive innovation: linear scalability. With Hadoop, Cassandra, recently Amazon’s DynamoDB, and many other big data technologies, the first terabyte of storage and processing power costs the same as the last. Therefore, the cost of storing and using data increases as a linear function of the quantity and the number of uses of the data.
Most (but importantly, not all) relational databases scale geometrically, which is to say the last terabyte of processing and storage capacity is exponentially more expensive than the first. Vendors like Greenplum and Asterdata are bringing linear scalability to the relational world, and Microsoft is bringing it to SQL Server on Azure.
But the true disruption really isn’t about database vendors’ market share - the pie is getting bigger, people. From a data management perspective, a linear scaling world couldn’t be more different than a geometrically scaling one. In a geometrically scaling world, you have to prioritise the use of your data resources and only the most valuable applications and reports get access to the operational and enterprise data architectures that support them. If you have ever witnessed (or been) an analytics person asking a DBA about accessing data, you understand the implication of this - ad hoc exploration of data to inform strategic decisions is painful because direct access is impossible and getting exports into analytical data marts is slow and discouraging.
Linear scalability turns data management from an art of managing scarcity to a science of maximising use. The linear-scaling world lets you encourage the use of data. Think about that in the context of our typical conversation with the DBA, and you will begin to realise how revolutionary the notion of promoting data’s use in decision making really is.
Additional use of data - by granting additional users’ access, integrating additional applications, or providing value-added data services - can be accommodated by simply adding proportionally more commodity nodes. The incremental cost of the additional use doesn’t increase, so you can keep adding users, and as long as they meet a minimal threshold of value, they are worthwhile.
That means every organisation now has the technical opportunity and business imperative to embark on a continual journey of implementing business processes, learning about them through exploration and analysis of the data, and achieving new and better outcomes by exploiting and using that insight. Enterprises will move to linear-scaling data platforms for both structured and unstructured data to support the continued growth of data assets and data use.
These new uses are not limited to the internal (data-driven) organisation. Companies can now seek opportunities to monetise their data by providing it to partners and customers, because syndicating outside uses of it also enjoys linear cost to deliver increasing scale. What if our health plans and grocery loyalty programs started sharing data?
On the bright side, personalised dietary modification suggestions are probably not far behind. On the other hand, do I really want the doctor disapprovingly telling me how many pints of Ben and Jerry’s I’ve had in the last decade as I’m in recovery from an emergency heart bypass?
By John Akred, senior manager, Accenture Technology Labs