Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent
Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines for applications such as environmental monitoring, disaster assessment, and land-use mapping. These models include unimodal vision encoders trained in a single data modality and multimodal architectures trained in multiple sensor modalities, such as synthetic aperture radar (SAR), multispectral, and hyperspectral imagery, or jointly in image-text pairs in vision-language settings. FMs are adapted to diverse perception tasks, such as semantic seg mentation, image classification, change detection, and visual question answering, depending on their pretraining objectives and architectural design. However, selecting the most suit able remote sensing foundation model (RSFM) for a specific task remains challenging due to scattered documentation, heterogeneous formats, and complex deployment constraints. To address this, we first introduce the RSFM Database (RS-FMD), the first structured and schema-guided resource covering over 160 RSFMs trained on various data modalities, spanning different spatial, spectral, and temporal resolutions, considering different learning paradigms. Built upon RS-FMD, we further present Remsa (Remote-sensing Model Selection Agent), a constraint-aware agent that enables automated RSFM selection from natural language queries. Remsa combines structured FM metadata grounding with task-aware orchestration for retrieval, clarification, ranking, and explanation. In detail, it interprets user input, clarifies missing constraints, ranks models via in-context learning, and provides trans parent justifications. Our system supports various RS tasks and data modalities, enabling personalized, reproducible, and efficient FM selection. To evaluate Remsa, we construct a benchmark of 100 expert-verified RS query scenarios. Each query is evaluated across 4 systems and 3 LLM backbones, with the top-3 selected models assessed by domain experts using a fixed suitability rubric. This results in 3,000 expert-scored task–system–model configurations under our rubric-based expert evaluation protocol. Remsa outperforms multiple baselines, including naive agent-based method, dense retrieval, and unstructured retrieval augmented generation approaches, showing its practical utility in real decision-making applications. Remsa operates entirely on publicly available metadata of open-source RSFMs, without accessing private or sensitive data. Our code and data are publicly available at: github.com/be-chen/REMSA.
