Banner Banner

MorphStream: Adaptive Scheduling for Scalable Transactional Stream Processing on Multicores

Yancan Mao
Jianjun Zhao
Shuhao Zhang
Haikun Liu
Volker Markl

February 01, 2022

Transactional stream processing engines (TSPEs) differ significantly in their designs, but all rely on non-adaptive scheduling strategies for processing concurrent state transactions. Subsequently, none exploit multicore parallelism to its full potential due to complex workload dependencies. This paper introduces MorphStream, which adopts a novel approach by decomposing scheduling strategies into three dimensions, and then strives to make the right decision along each dimension, based on analyzing the decision trade-offs under varying workload characteristics. Compared to the state-of-the-art, MorphStream achieves up to 3.4 times higher throughput and 69.1% lower processing latency for handling real-world use cases with complex and dynamically changing workload dependencies.