At the Strata + Hadoop conference this week, there will be a lot discussion about the business impact of Hadoop specifically and Big Data in general. While interest in both areas continues to run high, many organizations are discovering that mastering all the relevant technologies associated with these trends is anything but simple.
In fact, Nick Heudecker, an industry analyst at Gartner, predicts that through 2018 a full 70 percent of Hadoop deployments will not meet cost savings and revenue generation objectives due to skills and integration challenges.
While that may be unfortunate for those organizations, that kind of disconnect between interest in a technology and the ability to actually use it usually bodes very well for IT service providers.
Challenges and road blocks
The Big Data challenges IT organizations are wrestling with span everything from provisioning a Hadoop cluster to being able to find the appropriate mix of internal and external data sources that can be used to create a meaningful analytics result. Just figuring out how to ingest massive amounts of data is a major challenge for many IT organizations. Trying to actually analyze that data in anything that approaches real time is still more of an aspiration than a technical reality.
For those reasons, many of the Big Data projects involving Hadoop are being moved to the cloud. It’s easier to make provisioning the Hadoop cluster somebody else’s headache. But even once a project moves to the cloud, a host of issues surrounding the management of Big Data remain. In fact, a new Data Science Report from CrowdFlower, an online exchange for sourcing data scientists, finds that three out of five data scientists spend most of their time cleaning and organizing data. Given the fact that most data scientists make well over six figures, that’s a massive amount of money being spent on what amounts to data plumbing issues.
Faced with those challenges, a new report from Capgemini Consulting suggests that Big Data analytics investments are shifting from customer-facing projects to backend operations where data is both more accessible and reliable. A survey of more than 600 executives from the U.S., Europe, and China finds that over 70 percent of organizations now put more emphasis on operations than on consumer-focused processes for their analytics initiatives.
Obviously, some frustration with the return on investment associated with Big Data is starting to build. For the most part, there’s still a lot of faith that investments in Big Data analytics will yield new insights into the business. The challenge is that it’s starting to look like only companies with deep pockets may actually benefit from investments in Big Data analytics.
Big Data opportunities
In the meantime, IT service providers should be advising customers to distinguish between what is a one-time activity that needs to be accomplished to enable a Big Data analytics project and what's necessary to run the analytics application itself. IT service providers generally have more expertise on hand to complete a data ingestion project. The internal IT organization should be focused on marshalling the domain expertise needed to analyze that data.
Naturally, some organizations may choose to outsource that analytics function altogether. But to one degree or another every business is eventually going to need more access to advanced analytics in order to remain competitive, never mind actually gaining market share at the expense of less-informed rivals.