CARFAX was a technology pioneer, but as it grew, the technology couldn't kep pace. "We had a hard time scaling it and finding people to work on OpenVMS," says Jai Hirsch, senior systems architect for data technologies at CARFAX. The company needed a new database.
For years the default enterprise databases have been based on SQL, a programming language that databases from Oracle, Microsoft, SAP and many other companies predominantly use. But increasingly, SQL databases aren't an ideal fit for companies like CARFAX. Traditionally, SQL databases are based on rows and columns; CARFAX has 13.6 billion records associated with 700 million vehicles. A column-based system would have required thousands of columns and tabs, but for any given vehicle maybe only a dozen of them would be populated. It just wasn't ideal for CARFAX.
Hirsch and his team began playing with a new breed of databases referred to as NoSQL, which originally meant "Not only" SQL databases. NoSQL is a small but growing segment of the database market, according to 451 Research's Matt Aslett, who predicts it at about 2% of the size of the SQL market. While nascent, it's an important segment, especially for companies with big data needs that don't fit well into traditional SQL databases.
CARFAX is now running a 108-server deployment of MongoDB, one of the leading document NoSQL databases. It holds 10.6TBs of data and 1.5 billion vehicle documents are added a year. The Mongo database processes five times as many records per second as CARFAX's legacy system. Its working great for Hirsch and CARFAX and Aslett says it could be coming to many more enterprises in the near future.
Experts say that despite the rise of NoSQL databases during the past three to five years, NoSQL is not necessarily a replacement for SQL databases. Aslett says NoSQL databases are generally used for new projects and for handling data sets that can strain SQL systems. NoSQL databases have less rigid schemes, allowing more flexibility in how databases are created, used and managed. But SQL databases aren't going away. They're stable, proven and well-known amongst database administrators and they're still very good at transactional data that many legacy systems heavily rely on.
Forrester big data analyst Noel Yuhanna says NoSQL databases are "complementary" tools to SQL that "fills gaps found in traditional database systems."
Much of the movement to NoSQL is being driven by megatrends happening in the enterprise IT market. For example, developers are pushed to produce more applications faster. Having a database that's more flexible, can be configured multiple different ways and changed on the fly (NoSQL systems shine at this) can be an advantage in these faster-paced development environments. Meanwhile, applications being created are often interactive, maybe dealing with social streams and other data that doesn't fit neatly into a SQL database.
There's also a growing preference for cloud-hosted databases that can be spun up and down quickly without the need to own and manage specialized hardware. In a SQL-style database it usually is best for a schema of the data to be set up beforehand. In a NoSQL database, the data can basically just be thrown into the database and organized and search later. "It's for developers who don't want to wait for a DBA to provision a database," Aslett says.
With the increased interest in NoSQL databases, a growing market of vendors has blossomed to offer NoSQL databases either as on-premises software installations or on a public cloud. NoSQL databases can generally be broken into four major buckets: key-value store, document, wide-column and graph databases. There are a range of products across these categories from Amazon Web Services' DynamoDB being a leading key-value database to offerings from MongoDB, Couchbase, MarkLogic and Cloudant (which IBM purchased) specializing in document DBs. Vendors like Datastacks, ObjectRocker (acquired by Rackspace) and FoundationDB have a new breed of SQL-style databases that run on distributed platforms which some have dubbed NewSQL.
As the database market continues to evolve, the lines between database segments are beginning to blur. Oracle, a giant in the SQL database market now has NoSQL offering. Even within the NoSQL market overlap is emerging. AWS's DynamoDB, for example, was a leading key-value store database but recently announced it would support JSON to become a document database as well. The consolidation is only natural in a nascent and growing industry, experts say.
NoSQL offerings are still evolving, with ease of management, installation and building up fault tolerance and reliability being keys that vendors are working on. Backed by more than $230 million in venture financing (including a $150 million round last year), the 7-year old, New York City-based MongoDB (formerly named 10gen) recently announced MMS, a wizard-like tool which allows installation and management of a distributed MongoDB deployment through a slick GUI and automated provisioning engines.
But it's still in its early days. Aslett estimates that in 2012 the NoSQL market collected about $184 million in licensing and support revenue. By 2016 it is expected to collect $1 billion. That's considerable growth but it's still peanuts compared to the $16 billion overall big data market that IDC predicts in 2016.
That hasn't held back companies from adopting it though. At a recent Boston event for MongoDB as part of a traveling roadshow for the company, a senior architect from MetLife described how the company uses the NoSQL system as the back-end database for the company's customer service portal, used by call center employees to provide more detailed response to customer issues. Attendees roaming the halls ranged from a regional medical insurer who was in the earliest stages of testing MongoDB and evaluating it for broader use to a major university IT administrator who already has a MongoDB deployment in production.
Bob Wiederhold, CEO of NoSQL database company Couchbase, says the market started to pick up in 2007/2008 when the first NoSQL products emerged. In the beginning of 2013 he noticed a shift when enterprise buyers began flocking to NoSQL databases as their feature richness and reliability reached an adequate point. "They were stable enough for mission critical," he says. He believes the market is still in the phase of enterprise adoption. He predicts 2014 to be a year of broad deployment of NoSQL databases across many different sectors and types of businesses. That's the CEO of a NoSQL database talking, but experts agree, NoSQL is here to stay.