Big Data Is Falling Into the Trough of Disillusionment
Svetlana Sicular for Gartner:
Last several weeks show that big data is falling into the trough of disillusionment. I realized it earlier today, when I was describing a recent Elephant Riders meetup to my colleagues at Gartner. MapR, HortonWorks and Cloudera were debating the state of Hadoop. And I heard from the very core of the Hadoop movement that MapReduce has always been Hadoop’s bottleneck or that Hadoop is “primitive and old-fashioned.” […] Meanwhile, my most advanced with Hadoop clients are also getting disillusioned. They do not realize that they are ahead of others and think that someone else is successful while they are struggling. These organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions. Their disappointment applies to more advanced cases of sentiment analysis, which go beyond traditional vendor offerings. Difficulties are also abundant when organizations work on new ideas, which depend on factors that have been traditionally outside of their industry competence, e.g. linking a variety of unstructured data sources. Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging: first, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and, third, answering even one of them reliably. It is hard.
I’ve recently wrote in a post about the state of the Hadoop ecosystem:
The current state of affairs in the Hadoop space is that pretty much everything is possible, but the complexity of getting results varies way too much.
Anyways:
- while Hadoop is one of the most important tool of the Big Data toolchain, Hadoop and Big Data are not equivalent
-
according to the Hype cycle definitions, the Trough of disillusionment is defined as the failure of technologies to meet expectations thus becoming unfashionable. But I think it would be a mistake to generalize such a definition to include unrealistic expectations and stories and promises of silver bullets.
To expect that a technology which is still in its very early stages will be able to not only solve all previously unsolved problems, but do this fast and very simple is unrealistic. I am not aware of any young software technology that has ever done this before.
I’m not in measure to suggest changes to Gartner’s Hype Cycle, but my feeling is that for Big Data and Hadoop the Trough of Disillusionment and Slope of Enlightenment phases may coalesce. I don’t know if there are other historical examples to confirm this hypothesis in the software industry or outside it.
Original title and link: Big Data Is Falling Into the Trough of Disillusionment (©myNoSQL)
