A new cloud platform has been launched for Twitter users, allowing them to mine more than two years of historical tweets to extract insights and identify trends relating to businesses, brands, financial markets, news and public opinion.
Previously, only 30 days of historical Twitter data have been available for organisations to search, via services like Gnip. However, the new DataSift Historics platform, which leverages a re-syndication partnership with Twitter, enables companies to search tweets back to January 2010.
Historics aggregates public tweets using DataSift's social data platform and enriches them with details including sentiment, topics, web links, location and social media influence. This enrichment process incorporates other data sources such as Lexalytics for sentiment analysis and Klout to measure social influence to allow for precise segmentation.
The tweets are then loaded onto a Hadoop data store and run through a set of algorithms to work out which ones are relevant to each of DataSift's real-time clients. Customers can create filters to extract updates that are relevant to their company or their brands.
“DataSift is focused on simplifying how businesses can extract insights from the petabytes of social 'Big Data' being created,” said DataSift founder and CTO Nick Halstead. “With Historics, we are now democratising the Big Data industry to enable entrepreneurs and enterprises to easily create socially-intelligent applications. No data-scientists required, no Hadoop expertise needed.”
The company suggests a number of possible use cases. For example, Business Intelligence companies can correlate point-of-sale data with social data to identify trends in product sales and market sentiment; or financial organisations can analyse popular sentiment, trends and indicators relating to businesses and economic events.
However, some privacy campaigners have warned that the launch of the service represents a radical shift in the wrong direction.
“Did you think Twitter was just for fun? That your tweets were just harmless irrelevant mutterings into the electronic ether? You're wrong. You're taking part in a massive market-research study,” wrote Graham Cluley on the Naked Security blog.
“If you aren't comfortable with firms being able to mine your past tweets – and potentially gather information about you – you may wish to delete your old postings and think more carefully about what you share publicly on the internet in future.”
DataSift's marketing manager Tim Barker confirmed that deleted tweets will not be stored by the company. He said that DataSift operates in unison with Twitter, so tweets that are publicly available will be aggregated, those that are deleted will also be deleted by DataSift, and private accounts will remain invisible.
Historics is available today as a limited release to existing customers and is currently scheduled to be generally available in April 2012. It will be available either as a corporate subscription or as a pay-as-you-go service, designed to appeal to smaller organisations.