The Apache Foundation has accepted “Storm” into its incubator program yesterday after a series of +1 votes and no -1 votes. Acceptance to the incubator program serves as the first step for a project to be offered as an open source solution through Apache Foundation.
The voting proposal was put forward on September 12 and after roughly 6 days of voting, Storm was accepted. The main aim of Storm is be the ‘Hadoop of real time processing’. The way Hadoop does for batch processing: queues jobs, send them off to cluster of computers, waits for the results and then pulls the results back in usable form, the same Storm intends to do, but for real time data processing.
Nathan Marz, the project lead of Storm claims that “The lack of a ‘Hadoop of real time’ has become the biggest hole in the data processing ecosystem.”
Storm has been described as a “distributed realtime computation system” on its Github page. “Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use!” it reads further.
According to Marz, Storm provides primitives that allow programmers to write parallel real time computations with great ease. As of now those intending to process big data in real time would need to build network of queues and workers manually. Storm automates this process thereby adding a lot of scalability, which is evident from one of Marz’s claims: “one of Storm’s initial applications processed 1,000,000 messages per second on a 10 node cluster, including hundreds of database calls per second as part of the topology.”
Storm was developed and deployed at BackType back in January 2011. Twitter acquired BackType in July 2011 and open sourced Storm in September 2011. Over the period of two years, the open source project has seen deployments in over 50 organizations and has garnered over 40 active developers from companies including Twitter, Yahoo!, and Alibaba.
As a result of the acceptance Storm will see an influx of developers and supports from all over the globe thereby speeding up its development and deployments. Further, Storm’s entire codebase will be moved to Apache; development process with be integrated with that of Apache’s process; all dependencies will be made compliant with Apache License Version 2.0; and incremental update and releases will be as per Apache guidelines.