Purely-Functional Stream Processing

Abstract

Extracting value from streams of data generated by sensors and software is key to the success for many important problem domains including the Internet of Things (IoT). However there are many non-functional challenges to be overcome in achieving this, including very high data rates, a deployment environment featuring nodes with differing capabilities and limitations, energy and bandwidth constraints, performance requirements, and security guarantees. Most modern stream-processing systems leave addressing these challenges for the application programmer to solve by making manual adjustments to their program. This entanglement of the functional and non-functional aspects of stream processing increases the risk of mistakes and the potential cost of future maintenance. We describe an alternative approach — a declarative architecture (”StrIoT”) based on purely-functional programming. The application programmer provides a pro- gram encoding the functional requirements, a separate description of the operating environment, and the applicable non-functional requirements. StrIoT then derives functionally-equivalent program variants, generates corresponding deployment plans and automatically deploys the best-cost plan. In order to explore the viability of purely-functional programming for this problem domain, we designed and built a proof-of-concept, end-to-end implementation of this architecture using the purely-functional language Haskell, Linux containers and orchestration technology. This thesis focusses on two key components: the Optimiser and Evaluator. The Optimiser leverages purely-functional semantics to apply term rewriting and derive functionally-equivalent variants of the user’s Haskell program. The Evaluator filters and selects deployments plans using modelling techniques including queueing theory, in order to select the plan which will perform best for two non-functional requirements: bandwidth and cost. To evaluate these components and approaches we implemented several solutions to real-world stream-processing problems. In some cases, StrIoT can generate programs which meet applicable non-functional requirements even when the user-supplied program does not.