APEX-DAG: Library and Language independent Pipeline EXtraction
Modern data-driven systems often rely on complex pipelines to process and transform data for downstream machine learning (ML) tasks. Extracting these pipelines and understanding their struc ture is critical for ensuring transparency, performance optimiza tion, and maintainability, especially in large-scale projects. In this work,weintroduceanovelsystem,APEX-DAG(AutomatingPipeline EXtraction with Dataflow, Static Code Analysis, and Graph Atten tion Networks), which automates the extraction of data pipelines from computational notebooks or scripts. Unlike execution-based methods, APEX-DAG leverages static code analysis to identify the dataflow, transformations, and dependencies within ML workflows without executing the code or the need to alter the code. Further, after an initial training phase, our system can identify pipelines Ziawasch Abedjan BIFOLD & TUBerlin Berlin, Germany abedjan@tu-berlin.de that built with previously unseen libraries.
