Optimizing Continuous Queries in the Edge-Cloud Continuum

Sensor and edge nodes with data processing capabilities produce vast amounts of data outside the cloud. A new generation of data management systems has emerged to address the growing data volume by seamlessly integrating sensors, edge, and cloud infrastructures for efficient processing. These systems enhance latency and resource efficiency by processing data near its sourceon sensor and edge nodestransmitting only relevant information to the cloud for further analysis. However, a significant challenge for these systems is maintaining high resource efficiency while processing thousands of stream queries in a dynamic environment. In this thesis, we propose IMPACT (Incremental Stream Query Merging, Placement and Deployment), an end-to-end framework for computing resource-efficient plans, performing adaptive placements, and deploying query plans on a dynamic infrastructure. First, IMPACT uses incremental stream query merging (ISQM) to identify and manage sharing opportunities among thousands of queries under continuous operations. In particular, ISQM captures the semantic information of stream queries to enable merging even in the presence of syntactic differences. Our evaluation shows that ISQM exploits up to 65Πmore sharing opportunities than the naive baseline using hash-based signatures, scales linearly for thousands of queries, and saves a significant amount of resources compared to state-of-the-art approaches. Second, IMPACT utilizes incremental stream query placement (ISQP) to maintain valid operator placements under continuous query and infrastructure changes. ISQP performs a f ine-grained identification of invalid operator placements and takes concurrent, incremental placement decisions. This enables our framework to minimize the optimization overhead required to maintain valid placements, even in the face of continuous query and infrastructure changes. Our evaluations show that ISQP reduces the optimization overhead by one order of magnitude compared to the baseline. Lastly, IMPACT utilizes incremental stream query deployment (ISQD) to maintain the validity of query deployments on a dynamic infrastructure. ISQD uses a greedy strategy to concurrently select and redeploy only the operators affected by topology changes. Additionally, ISQD removes the need for an external state transfer component by using ad-hoc queries to perform state migration in a hierarchical infrastructure. Our evaluations show that ISQD incurs up to 7.5Πlower deployment latency and up to 39Πlower event time latency compared to the strongest baseline, while efficiently handling high-frequency topology changes. In conclusion, this thesis introduces a framework designed to generate resource-efficient plans and ensure the continuous execution of queries in a dynamic sensor-edge-cloud infrastructure. Our framework serves as a foundational component of NebulaStream, a next generation IoT data management platform, enabling the efficient management of massive amounts of workloads in an ever-evolving environment.