Run-time adaptation of a functional stream processing system

Abstract

Extracting value from streams of events generated by sensors and software has become key to the success of many important classes of applications, whether this be sensors for smart cities/buildings, or wearable healthcare devices. However, writing streaming data applications is not easy – developers are confronted with major challenges, including processing events arriving at varying rates from thousands to millions of events per second, distributing processing over a set of heterogeneous platforms including edge devices and cloud servers, and meeting non-functional requirements such as energy, networking, security and performance. The data within these applications can be largely dynamic, and requires the streaming system to adapt to the ever-changing demands. This thesis focuses on one challenge in distributed stream processing: automatically adapting the partitioning of the processing between the edge and the cloud without a loss of service. An example is when the event arrival rate increases and the edge processor can no longer meet performance requirements. Re-partitioning without loss of service involves moving computations between the edge and the cloud while events are still being processed. In this thesis the StrIoT system is introduced – a stream processing system that supports automatic re-partitioning of a streaming application. It is based on a set of functional stream operators, and the thesis describes how the run-time system can automatically adapt applications that use them. Results are presented from the evaluation of StrIoT on a real-world dataset of taxi journey information, using both cloud servers and an edge device, showing that performance can be improved with only a low, temporary impact during adaptation.