A fair amount of time usually passes between when a technology first emerges and when it reaches a tipping point in terms of mass adoption. New research from Dressner Advisory Services suggests that Big Data in form of Hadoop has finally reached that point.
The survey of IT leaders finds that 41 percent have already deployed Hadoop, which is an open source distributed storage system optimized for massive amounts of data, in production with another 46 percent saying they may use Hadoop in the future. Among those considering Hadoop, 47 percent report that they plan to deploy it in 2017.
Efforts to transform data warehouses are driving most of the adoption of Hadoop. Instead of relying on relational databases that have been the foundation of most data warehouses for decades, IT organizations are shifting to Hadoop. The primary reason for this change is that Hadoop makes it affordable to store massive amounts of data. Rather than making business decisions on the limited amount of data that can be housed in a relational database, organizations are moving to deploy analytics applications on top of Hadoop in the hopes that access to all the raw data an organization has will lead to more informed business decisions.
New opportunities for IT service providers
This modernization of enterprise data warehouses creates a massive opportunity for IT service providers to help companies rethink the role data plays inside their organization. Not only are most of them increasing the sheer volume of data they collect, they also want to be able to compare that data against external sources to gather additional insights. In time, organizations will also start to process analytics in real time to create actionable intelligence at the very moment when a transaction is being processed.
Firms such as Accenture are already moving to create entire practices around embedding analytics into business decision-making processes. Further down the line, those Big Data investments will also provide the foundation around which advanced artificial intelligence (AI) applications based on machine learning algorithms will be built.
The top four commercially supported distributions from Hadoop come from Hortonworks, Cloudera, MapR Technologies, and a cloud service managed by Amazon Web Services (AWS). The single most important decision any organization can make is where to physically place their data warehouse. Thanks to the laws of data gravity a data warehouse that contains massive amounts of data is not likely to ever move from the location where it first gets deployed. Naturally, that decision then has all sorts of implications for not only where analytics applications will reside in the future, but also what stratagies need to be employed to manage and protect all that data.
The reengineering of data warehouses at this scale is an IT event that comes along once in a couple of decades. Rather than thinking about the future of data management in a piecemeal fashion, IT service providers need to develop a comprehensive approach to Big Data spanning everything from the management of the core Hadoop platform to backup and recovery of that data. They say data is the new oil in a digital economy. But oil doesn’t have much value in and of itself until someone goes to the trouble of refining it.