What to Expect from Big Data in 2014
Nick Kolakowski, Slashdot
December 30, 2013
http://slashdot.org/topic/bi/what-to-expect-from-big-data-in-2014/
Hadoop will continue to reign; more clients will demand the latest in encryption; and a lot of analytics firms could fail.
As 2013 stumbles across the finish line, technology publications are busy issuing their inevitable Year’s End lists, in which the Best and Worst of everything is carefully delineated (BlackBerry’s failing! Apple continued to produce shiny things! Ballmer’s leaving Microsoft!). And that’s all well and good, but Big Data is all about the future—and with that in mind, here are some mild predictions for 2014:
Hadoop’s Reign Continues
Apache Hadoop, the open-source framework for running data applications on large hardware clusters, enjoyed considerable momentum in 2013: not only did Hadoop 2.2.0 reach its general-release milestone (improvements included more integration with other open-source projects, as well as YARN, a general-purpose resource management system that makes it easier to work with MapReduce and other frameworks), but a variety of prominent firms released their own distributions of the platform—all but ensuring Hadoop will rumble strong through 2014.
But continuing popularity or no, not every company will use Hadoop effectively. “More Hadoop projects will be swept under the rug as businesses devote major resources to their Big Data projects before doing their due diligence, which results in a costly, disillusioning project failure,” Gary Nakamura, CEO of Concurrent, wrote in a December blog posting. “We may not hear about most of the failures, of course, but the successes will clearly demonstrate the importance of using the right tools.”
For those companies that can accurately assess whether Hadoop is the right tool for their business needs, the improvements to the platform could translate into more effective data-crunching—and better insights. But Hadoop’s continued prominence raises still another issue.
Open-Source vs. Proprietary
Open-source has long remained a double-edged sword for firms that build analytics software. Back in Ye Olden Days of mid-2012, for example, research firm IDC suggested that the availability of open-source solutions could very well hamper the ability of proprietary software platforms to fully capitalize the analytics space. “The Hadoop and MapReduce market will likely develop along the lines established by the development of the Linux ecosystem,” Dan Vesset, vice president of Business Analytics Solutions for IDC, wrote in a statement at the time. “Over the next decade, much of the revenue will be accrued by hardware, applications, and application development and deployment software vendors.”
Open-source’s strength in the analytics space hasn’t dissuaded a broad cross-section of tech giants and startups from building and releasing proprietary software into the ecosystem. But those companies’ solutions will need to push back against open-source—and rival platforms—in order to scratch out any sort of market-share and profit; a good number of them will fail, including several companies that were teetering on the brink in 2013.
Data Encryption
Over the summer, government-contractor-turned-whistleblower Edward Snowden revealed the existence of massive National Security Agency (NSA) programs designed to harvest vast amounts of online data. Those revelations, in turn, have sparked everything from legal battles to promises of government review.
Whether or not the NSA actually rolls back its programs, the newfound awareness of widespread online surveillance will drive more companies to encrypt all their data—and not only in storage, but also in movement from Point A to B. That will spark a rise in costs (encryption is expensive, after all), and perhaps slow some everyday processes down (as encryption adds another layer to most operations)—but with more and more clients demanding the latest in privacy and security, vendors may have no choice but to armor their infrastructure as extensively as possible.
Ease of Use…
Data analytics tools: they’re not just for data scientists anymore. Over the next year, the trend of making those tools easier to use for people with relatively little analytics knowledge will continue, with simplified dashboards and greater integration of multiple types of data.
…But Issues Remain
Facebook and Netflix wouldn’t be operational without custom data infrastructure painstakingly developed by their in-house data scientists; IBM is using its Watson supercomputing platform to assist in everything from retail to healthcare IT; dozens of firms use analytics to improve everything from shipping and logistics to customer service.
But for many firms, the jury’s still out about the ultimate effectiveness of analytics. Back in August, The New York Times asked if Big Data was “an economic big dud,” because U.S. economic productivity had slowed over the past few years despite a massive growth in the amount of data held by individuals and corporations. That might have been overstating things—many analytics tools have only begun to penetrate the market in a meaningful way, suggesting that only a small percentage of firms are leveraging their datasets to fullest effect—but it’s undeniable that, when it comes to various industries, analytics still has something to prove. 2014 might be the year that Big Data actually makes more of a name for itself.