Heterogeneity-Aware Data Management for Data-Centric Machine Learning Pipelines
Machine learning (ML) has emerged as one of the most consequential technologies of the past decade, driving progress across many domains. Beyond improvements in model architectures and hardware, the effectiveness of ML critically depends on data management processes such as data discovery, integration, and loading. As data volume and diversity increase, these processes become a major source of complexity and cost in data-centric ML pipelines. Prior work on data management for ML largely assumes centrally governed and homogeneous environments. In practice, however, ML pipelines operate under heterogeneous conditions. Data are frequently collected and maintained by autonomous providers using diverse schemas and formats, complicating data discovery. Furthermore, different data representations produced by independent repositories are not directly comparable, limiting cross-repository data integration. Meanwhile, data loading must adapt to varying hardware characteristics and encoding choices for large-scale data modalities, such as images. This heterogeneity limits the applicability of existing approaches and increases both human effort and computational cost. In this thesis, we address data management challenges of data-centric ML pipelines in heterogeneous environments and make three main contributions. First, we propose Fainder, an index-for-distribution-aware dataset search over decentralized data repositories. It introduces percentile predicates to express distributional constraints and operates on heterogeneous histogram collections without requiring access to raw data. Second, we present FedAugment, a framework for table augmentation search in decentralized settings. State-of-the-art augmentation search approaches perform semantic similarity search over table embeddings, but assume homogeneous embeddings generated by a single pipeline. FedAugment enables joinable and unionable table discovery across independently generated embeddings by projecting local representations into a shared semantic space. Third, we position lossy image compression as a domain-aware optimization technique for data loading under heterogeneous hardware constraints. Through an extensive experimental study, we show how codec and quality choices tune data representations to improve training throughput and, under resource constraints, even improve model accuracy. Overall, this thesis advances data management for data-centric ML pipelines by introducing mechanisms for aligning, indexing, and tuning heterogeneous data representations. Our contributions enable advanced dataset search without raw-data centralization and improve resource utilization during model training.
