Channels ▼   Dataflow Programming: Handling Huge Data Loads
Without Adding Complexity  Today, we speak casually of terabytes
of data the way we used to speak of megabytes. Multi-petabyte
data stores are common and growing. To handle all this data
quickly and efficiently, and to extract useful business insights
from it, we need programming paradigms that combine the scalable
use of multicore processors and many-node clusters with data
processing and analysis. No single programming paradigm can
serve all problem domains, so expect many alternatives and
consolidation. We're entering an era of innovation and the time
is ripe for a robust programming paradigm well suited to big
data processing and analytics. In this article, I'll discuss
dataflow, a technique that manages very large data flows into
and through islands of high-speed computation.  What Is
Dataflow?  As a software architecture, dataflow consists of
nodes in a graph connected by directed dataflow queues. The
graph represents an application or program