Concurrent’s new Hadoop monitoring tool looks out for big-data snags
Jordan Novet, VentureBeat
February 4, 2014
http://venturebeat.com/2014/02/03/concurrents-new-hadoop-monitoring-tool-looks-out-for-big-data-snags/
Software developers want to know if their applications are running properly on the Internet. Just look at the runaway success of application-performance management companies like New Relic and AppDynamics.
Similarly, engineers who process lots of different kinds of data using the Hadoop ecosystem of open-source software need a monitoring tool to make sure their Hadoop jobs are running right.
Just such a tool is coming out today from Concurrent, the company behind the open-source Cascading framework for implementing Hadoop jobs.
It’s called Driven. And it’s been a long time in the making. Founder Chris Wensel has been talking about it for at least two and a half years.
“The net of it is that we’re going to be able to provide very detailed, high-fidelity [information] about data applications as they’re running on Hadoop,” Wensel said in an interview with VentureBeat. “So things like, you know, your joins and merges and filters — you’ll be able to see them progress … . We’ll notify you of failures, and we’ll take you right to the line of code where the application failed essentially and give you insight to go and fix the issue.”
Concurrent is making Driven available as a free service that runs in the cloud, so Cascading users can get a taste of it while building applications, Gary Nakamura, Concurrent’s chief executive, told VentureBeat. If people want to use Driven in production or in on-premises data centers, they’ll have to pay for an annual license, Nakamura said.
Engineers using the service in an early adopter program have been pleased with Driven’s ability to identify problems, so debugging can take less time, Wensel said.
Driven sounds like the kind of thing that could make more companies comfortable with the idea of experimenting with Hadoop. Hadoop can work in addition to or in the place of more traditional, and typically more expensive, data warehousing technology from companies like IBM and Teradata. Hadoop still isn’t extremely widely adopted, but it’s gotten more popular each year since the Apache Hadoop project began in 2006.
The complexity of Hadoop has inhibited its adoption, along with concerns about security and other issues. If Driven does manage to make Hadoop easier to operate, Concurrent could stand to enjoy a degree of the success that Hadoop companies like Cloudera have. Cloudera is in a position to go public later this year.
San Francisco-based Concurrent started in 2008, and to date it has raised $5 million, including a $4 million round last year.