A survey of systems for massive stream analytics
---
paper_title: S4: Distributed Stream Computing Platform

paper_content:

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. Keyed data events are routed with affinity to Processing Elements (PEs), which consume the events and do one or both of the following: (1) emit one or more events which may be consumed by other PEs, (2) publish results. The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. In this paper, we outline the S4 architecture in detail, describe various applications, including real-life deployments. Our design is primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware.
---
paper_title: S4: Distributed Stream Computing Platform

paper_content:

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. Keyed data events are routed with affinity to Processing Elements (PEs), which consume the events and do one or both of the following: (1) emit one or more events which may be consumed by other PEs, (2) publish results. The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. In this paper, we outline the S4 architecture in detail, describe various applications, including real-life deployments. Our design is primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware.
---
paper_title: MillWheel: Fault-Tolerant Stream Processing at Internet Scale

paper_content:

MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework's fault-tolerance guarantees. ::: ::: This paper describes MillWheel's programming model as well as its implementation. The case study of a continuous anomaly detector in use at Google serves to motivate how many of MillWheel's features are used. MillWheel's programming model provides a notion of logical time, making it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel's unique combination of scalability, fault tolerance, and a versatile programming model lends itself to a wide variety of problems at Google.
---
paper_title: Scale-Up Strategies for Processing High-Rate Data Streams in System S

paper_content:

High performance stream processing is critical in sense-and-respond application domains – from environmental monitoring to algorithmic trading. In this paper, we focus on language and runtime support for improving the performance of sense-and-respond applications in processing data from high rate streams. The central tenet of this work is the definition of a streaming architectural pattern for these application domains and the programming model and the code generation framework to support it. Using IBM Research's System S middleware and the SPADE language, we demonstrate how to scale up a financial trading application.
---
paper_title: A streaming approach to radio astronomy imaging

paper_content:

The emergence of large-field radio telescopes has generated the need to process huge amounts of data “on the fly” in order to avoid substantial costs in storage and signal processing. In this paper, we describe a streaming software approach to radio astronomy, and in particular, show how the IBM-developed stream computing architecture System S enables seamless “on the fly” radio astronomy imaging. This approach overcomes pertinent issues of memory limitation and latencies related to data processing, and it allows autonomous processor allocation to tasks as the need arises.
---
paper_title: Large-Scale DNA sequence analysis in the cloud: a stream-based approach

paper_content:

Cloud computing technologies have made it possible to analyze big data sets in scalable and cost-effective ways. DNA sequence analysis, where very large data sets are now generated at reduced cost using the Next-Generation Sequencing (NGS) methods, is an area which can greatly benefit from cloud-based infrastructures. Although existing solutions show nearly linear scalability, they pose significant limitations in terms of data transfer latencies and cloud storage costs. In this paper, we propose to tackle the performance problems that arise from having to transfer large amounts of data between clients and the cloud based on a streaming data management architecture. Our approach provides an incremental data processing model which can hide data transfer latencies while maintaining linear scalability. We present an initial implementation and evaluation of this approach for SHRiMP, a well-known software package for NGS read alignment, based on the IBM InfoSphere Streams computing platform deployed on Amazon EC2.
---
paper_title: Novel data stream techniques for real time HF radio weather statistics and forecasting

paper_content:

The HF radio environment often exhibits dramatic changes that span over many different time scales, ranging from milliseconds (burst radio transmissions) to about eleven years (the solar cycle). It is desirable to obtain statistics of this HF "radio weather" in order to make reliable forecasts. We have used real time data from a vector antenna in Uppsala, Sweden, to measure intensity, polarization, and direction of arrival of sources in the 100 kHz to 20 MHz frequency range. The digitized antenna signals are distributed over the internet and the real time processing is performed using a novel streaming data base manager, System S from IBM. The initial results of this research project are presented and its future applications are discussed.
---
paper_title: S4: Distributed Stream Computing Platform

paper_content:

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. Keyed data events are routed with affinity to Processing Elements (PEs), which consume the events and do one or both of the following: (1) emit one or more events which may be consumed by other PEs, (2) publish results. The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. In this paper, we outline the S4 architecture in detail, describe various applications, including real-life deployments. Our design is primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware.
---
paper_title: IBM infosphere streams for scalable, real-time, intelligent transportation services

paper_content:

With the widespread adoption of location tracking technologies like GPS, the domain of intelligent transportation services has seen growing interest in the last few years. Services in this domain make use of real-time location-based data from a variety of sources, combine this data with static location-based data such as maps and points of interest databases, and provide useful information to end-users. Some of the major challenges in this domain include i) scalability, in terms of processing large volumes of real-time and static data; ii) extensibility, in terms of being able to add new kinds of analyses on the data rapidly, and iii) user interaction, in terms of being able to support different kinds of one-time and continuous queries from the end-user. In this paper, we demonstrate the use of IBM InfoSphere Streams, a scalable stream processing platform, for tackling these challenges. We describe a prototype system that generates dynamic, multi-faceted views of transportation information for the city of Stockholm, using real vehicle GPS and road-network data. The system also continuously derives current traffic statistics, and provides useful value-added information such as shortest-time routes from real-time observed and inferred traffic conditions. Our performance experiments illustrate the scalability of the system. For instance, our system can process over 120000 incoming GPS points per second, combine it with a map containing over 600,000 links, continuously generate different kinds of traffic statistics and answer user queries.
---
paper_title: S4: Distributed Stream Computing Platform

paper_content:

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. Keyed data events are routed with affinity to Processing Elements (PEs), which consume the events and do one or both of the following: (1) emit one or more events which may be consumed by other PEs, (2) publish results. The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. In this paper, we outline the S4 architecture in detail, describe various applications, including real-life deployments. Our design is primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware.
---
paper_title: Performance Evaluation of Yahoo! S4: A First Look

paper_content:

Processing large data sets has been dominated recently by the map/reduce programming model [1], originally proposed by Google and widely adopted through the Apache Hadoop1 implementation. Over the years, developers have identified weaknesses of processing data sets in batches as in MapReduce and have proposed alternatives. One such alternative is continuous processing of data streams. This is particularly suitable for applications in online analytics, monitoring, financial data processing and fraud detection that require timely processing of data, making the delay introduced by batch processing highly undesirable. This processing paradigm has led to the development of systems such as Yahoo! S4 [2] and Twitter Storm.2 Yahoo! S4 is a general-purpose, distributed and scalable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. As these frameworks are quite young and new, there is a need to understand their performance for real time applications and find out the existing issues in terms of scalability, execution time and fault tolerance. We did an empirical evaluation of one application on Yahoo! S4 and focused on the performance in terms of scalability, lost events and fault tolerance. Findings of our analyses can be helpful towards understanding the challenges in developing stream-based data intensive computing tools and thus providing a guideline for the future development.
---
paper_title: Smart cities and the future internet: towards cooperation frameworks for open innovation

paper_content:

Cities nowadays face complex challenges to meet objectives regarding socio-economic development and quality of life. The concept of "smart cities" is a response to these challenges. This paper explores "smart cities" as environments of open and user-driven innovation for experimenting and validating Future Internet-enabled services. Based on an analysis of the current landscape of smart city pilot programmes, Future Internet experimentally-driven research and projects in the domain of Living Labs, common resources regarding research and innovation can be identified that can be shared in open innovation environments. Effectively sharing these common resources for the purpose of establishing urban and regional innovation ecosystems requires sustainable partnerships and cooperation strategies among the main stakeholders.
---
paper_title: A streaming approach to radio astronomy imaging

paper_content:

The emergence of large-field radio telescopes has generated the need to process huge amounts of data “on the fly” in order to avoid substantial costs in storage and signal processing. In this paper, we describe a streaming software approach to radio astronomy, and in particular, show how the IBM-developed stream computing architecture System S enables seamless “on the fly” radio astronomy imaging. This approach overcomes pertinent issues of memory limitation and latencies related to data processing, and it allows autonomous processor allocation to tasks as the need arises.
---
paper_title: Foundations for smarter cities

paper_content:

This paper describes the information technology (IT) foundation and principles for Smarter Cities™. Smarter Cities are urban areas that exploit operational data, such as that arising from traffic congestion, power consumption statistics, and public safety events, to optimize the operation of city services. The foundational concepts are instrumented, interconnected, and intelligent. Instrumented refers to sources of near-real-time real-world data from both physical and virtual sensors. Interconnected means the integration of those data into an enterprise computing platform and the communication of such information among the various city services. Intelligent refers to the inclusion of complex analytics, modeling, optimization, and visualization in the operational business processes to make better operational decisions. This approach enables the adaptation of city services to the behavior of the inhabitants, which permits the optimal use of the available physical infrastructure and resources, for example, in sensing and controlling consumption of energy and water, managing waste processing and transportation systems, and applying optimization to achieve new efficiencies among these resources. Additional roles exist in intelligent interaction between the city and its inhabitants and further contribute to operational efficiency while maintaining or enhancing quality of life.
---
paper_title: Smart cities at the forefront of the future internet

paper_content:

Smart cities have been recently pointed out by M2M experts as an emerging market with enormous potential, which is expected to drive the digital economy forward in the coming years. However, most of the current city and urban developments are based on vertical ICT solutions leading to an unsustainable sea of systems and market islands. In this work we discuss how the recent vision of the Future Internet (FI), and its particular components, Internet of Things (IoT) and Internet of Services (IoS), can become building blocks to progress towards a unified urban-scale ICT platform transforming a Smart City into an open innovation platform. Moreover, we present some results of generic implementations based on the ITU-T's Ubiquitous Sensor Network (USN) model. The referenced platform model fulfills basic principles of open, federated and trusted platforms (FOTs) at two different levels: the infrastructure level (IoT to support the complexity of heterogeneous sensors deployed in urban spaces), and at the service level (IoS as a suit of open and standardized enablers to facilitate the composition of interoperable smart city services). We also discuss the need of infrastructures at the European level for a realistic large-scale experimentally-driven research, and present main principles of the unique-in-the-world experimental test facility under development within the SmartSantander EU project.
---
paper_title: Large-Scale DNA sequence analysis in the cloud: a stream-based approach

paper_content:

Cloud computing technologies have made it possible to analyze big data sets in scalable and cost-effective ways. DNA sequence analysis, where very large data sets are now generated at reduced cost using the Next-Generation Sequencing (NGS) methods, is an area which can greatly benefit from cloud-based infrastructures. Although existing solutions show nearly linear scalability, they pose significant limitations in terms of data transfer latencies and cloud storage costs. In this paper, we propose to tackle the performance problems that arise from having to transfer large amounts of data between clients and the cloud based on a streaming data management architecture. Our approach provides an incremental data processing model which can hide data transfer latencies while maintaining linear scalability. We present an initial implementation and evaluation of this approach for SHRiMP, a well-known software package for NGS read alignment, based on the IBM InfoSphere Streams computing platform deployed on Amazon EC2.
---

Title: A Survey of Systems for Massive Stream Analytics

Section 1: INTRODUCTION
Description 1: Introduce the importance and motivation for real-time data processing systems and outline the organization of the paper.

Section 2: STREAMING CONCEPTS
Description 2: Discuss the fundamental differences between streaming data and traditional data handling, along with the requirements and challenges of real-time stream processing.

Section 3: Open Source Streaming Solutions
Description 3: Examine various existing open-source streaming solutions, including Storm, Spark Streaming, and S4.

Section 4: Commercial Streaming Solutions
Description 4: Explore commercial streaming solutions like Google MillWheel, Amazon Kinesis, and InfoSphere Streams.

Section 5: ARCHITECTURE ANALYSIS
Description 5: Analyze the architecture of different stream processing solutions, focusing on processing models, latency, data pipelines, fault tolerance, and data guarantees.

Section 6: APPLICATIONS
Description 6: Present various applications of real-time streaming data processing in different domains such as finance, healthcare, smart cities, and more.

Section 7: CHALLENGES
Description 7: Detail the challenges associated with stream processing solutions at both the application and system levels.

Section 8: CONCLUSIONS
Description 8: Summarize the insights gained from the survey and discuss the future directions and considerations for stream processing paradigms.