Incremental Stream Query Merging in Action
Abstract: Stream Processing Engines (SPEs) execute long-running queries on unbounded data streams. However, they primarily focus on achieving high throughput and low latency for a single query. In order to execute multiple queries, users run each query in isolation on independent compute clusters. As a result, SPEs overlook potential data and computation-sharing opportunities among several long-running queries.Streaming queries are generally continuous and long-running; identifying sharing opportunities among these existing queries can improve resource utilization. In this demonstration, we present Incremental Stream Query Merging (ISQM), a generic end-to-end framework to identify and maintain sharing among stream queries. We implement and demonstrate a total of six different types of sharing identification techniques in ISQM and their impact on query optimization and execution time.
