KRAKEN: Joint Optimization of Operator Placement and Communication for Distributed Stream Processing
Internet of Things applications run continuous analytics over geo-distributed event streams and often rely on stateful binary operators, such as window joins and sequences. Efficient decentralized execution of these workloads depends on two tightly coupled decisions: where to execute operators (placement) and how to exchange events between nodes (communication). Most existing planners make these decisions sequentially by first fixing placement and then optimizing communication, which can miss globally better execution plans.
In this paper, we present KRAKEN, an execution planner that jointly optimizes both decisions by evaluating placement candidates under alternative communication schemes (all-push and push-pull) before committing to atomic deployment decisions. To compare different decisions, KRAKEN uses a bi-objective cost model that balances tuple volume and latency. Across diverse synthetic topologies and workloads, KRAKEN reduces tuple volume by 3× on average while lowering latency and planning time compared to the sequential baseline.
