Examining vision foundation models for classification and detection in optical and radio astronomy

¹ Department of Computer Science, University of Geneva, 7 route de Drize, 1227 Carouge, Switzerland
² Department of Astronomy, University of Geneva, 51 Chemin Pegasi, 1290 Versoix, Switzerland
³ SKA Observatory, Jodrell Bank, Lower Withington, Macclesfield SK11 9FT, UK

^★ Corresponding author: erica.lastufka@unige.ch

Received: 7 January 2025
Accepted: 8 September 2025

Abstract

Context. Vision foundation models, which have demonstrated significant potential in many multimedia applications, are often underutilized in the natural sciences. This is primarily due to mismatches between the nature of domain-specific scientific data and the typical training data used for foundation models, leading to distribution shifts. Scientific data often differ substantially in structure and characteristics, and researchers frequently face the challenge of optimizing model performance with limited labeled data of only a few hundred or thousand images.

Aims. This work evaluates the performance of vision foundation models in astrophysics, with a focus on identifying the best practices for adapting these models to domain-specific datasets. We aim to establish a framework for selecting, fine-tuning, and optimizing these models for common tasks in optical and radio astronomy.

Methods. We compared multiple foundation models, including self-supervised, weakly supervised, and distillation-based architectures, across two representative optical and radio datasets. Experiments involved different fine-tuning strategies, projector heads, and data preprocessing techniques, with performance evaluated on classification and detection metrics.

Results. Features extracted by specific foundation models improved classification accuracy for optical galaxy images compared to conventional supervised training. Similarly, these models achieved equivalent or superior performance in object detection tasks with radio images. However, classification performance for radio galaxy images was generally poor, often falling short of traditional supervised approaches.

Conclusions. These findings suggest that selecting suitable vision foundation models for astrophysics applications requires careful consideration of the model characteristics and alignment with the specific requirements of the downstream tasks. This study demonstrates that vision foundation models can be effectively adapted to astrophysical applications, provided practitioners iterate on model selection, training strategies, and data handling. The proposed framework bridges the gap between these advanced models and the unique demands of astronomy, enabling broader adoption of deep learning in the field.