It’s rare for a technology and concept like Hadoop and Big Data to take off with the speed that they have, more so with a technology and concept as big and complex. Hadoop was an internal project at Yahoo just five years ago. Now, companies are betting their business on Big Data.
The challenge, though, is for the products to mature. And much like those silly cheese commercials on TV, Hadoop needs to mature as well. It’s one thing for techies with Java skills to download the source code from the Apache Foundation and conduct internal experiments. But if that’s all there is, that would have guaranteed that Hadoop never got beyond the propellerheads.
For the union of Hadoop and Big Data to grow and thrive it has to do like every other concept or product and “cross the chasm,” as defined in Geoffrey Moore’s definitive 1991 book on marketing. Crossing the chasm means making the leap from a few early adopters, almost always very technically advanced who can work with the product on their own, to the masses on the other side of the chasm. If you make it, you’re Linux. If you fall short, you end up like CORBA.
As far as Gartner is concerned, Hadoop and Big Data are at the point beyond early adopters and ready to make the leap to the other side, but not just yet. In a survey conducted last month of almost 300 enterprise IT decision makers, 54% said they had no plans to invest in Hadoop technologies in the coming year, but that means 46% are ready to pull the trigger or already have. All things considered, said Merv Adrian, the Gartner research VP who conducted the survey, that’s not bad.
“I’m not surprised, because in terms of maturity of the market, Gartner believes we are just finishing the early adopter stage. And there is a natural pause reflected in a lifecycle when mainstream adopters start to look at what early adopters have already taken in. All kinds of questions come up,” he said.
The hype cycle is about the trigger – some disruptive technology coming into the market. In the early days, the fact that a given technology is rough around the edges and needs new skills is accepted by early adopters willing to put up with it to take advantage of being first. But it needs more to make it to the next level.
Hadoop and Big Data: Rejected?
One reason why the disinterest in Hadoop/Big Data is so high is some companies have tried it and decided they didn’t need it. It's not an uncommon characteristic of early adopters that they get into a tech because it's the bright shiny object and want to see what it's good for and play around with it, but not have any use for it. Then it becomes a technology in search of a solution.
“When we ask people why they are not interested, the reason we heard the most was we haven’t identified something we need to use this for. We don’t need it or we have a technology that's adequate. Or they say no one has presented us with a business case,” said Adrian.
Randall Barnes, senior cloud architect at 2nd Watch, an AWS managed services partner, said that about 25% of his clients tried Hadoop and/or Big Data and didn’t want it, which he said is higher than most other new technologies.
“Hadoop has a much higher rate of customers deciding not to pursue it than others, but the vast majority of customers we work with are pleased and do commit to move into the next phase,” he said.
Aside from the business case, there was also the cost, Barnes added. In looking at what it would cost to retool their developer teams and developer processes, match their existing IP with ETL and other data requirements, some customers felt the cost of a migration shift was higher than the benefits.
Finally, one of the biggest obstacles he’s encountered is just getting familiarity and having a broader audience participate in the evaluation and experience and try to understand how it will benefit to have this new workload or app stack. In other words, Big Data still doesn’t have a perfect pitch. “There is this barrier to getting this [concept] into a form the IT department or decision makers can get their hands around it,” he said.
Hadoop Product Maturity
Adrian said that while 54% of people he surveyed have not made a decision on Hadoop and don’t plan to invest in the next two years, that number is not going to hold through the next two years. “As you move into the trough, you move into the mainstream where people have different assessments and have expectations that early adopters don’t have. They want more fit and finish and easier to use and security,” he said.
What's happening now is buyers are looking for more than just the software download. They are looking for things that show the Hadoop stack and offerings are maturing. He made it clear Gartner is not going to predict their change, it's going to report their change.
And change is coming. If you look at the stuff announced at the recent Hadoop Summit, there were deployment utilities, deployment services, security services, monitoring services and more. “These are the kinds of things mature enterprises want. They don't want to hear a bunch of guys in the Silicon Valley are writing this code and it will be available next week,” said Adrian.
Building the Business of Big Data
Prakash Nanduri, co-founder and CEO of Paxata, a Big Data development and deployment firm, agreed with Adrian’s contention that Big Data and Hadoop are at the chasm crossing phase, but thinks it will go quicker than Gartner does.