Bridging the Gap: Metadata, Multimodal Data, and Hybrid Machine Learning Approaches in Citizen Science

Abstract

The majority of machine learning applications in citizen science have focused on image classification, with algorithms trained to identify species or classify objects from photographs. However, many citizen science projects collect diverse types of data, including images, audio, text, and structured metadata such as location, date, and observer characteristics. These diverse data types, when integrated through multimodal machine learning approaches, can significantly enhance the accuracy and robustness of predictions, particularly for challenging classification tasks where images alone are insufficient. This review examines the integration of metadata and multimodal data in citizen science machine learning applications. We analyze the types of metadata that are commonly collected, including spatial, temporal, environmental, and observer-related information, and examine how these data can be combined with images and other unstructured data to improve model performance. We review case studies where multimodal approaches have been applied, including species identification projects that combine images with location data, acoustic monitoring that integrates environmental information, and quality assessment systems that incorporate observer expertise. We analyze the technical approaches, including early and late fusion strategies, and identify the conditions under which multimodal approaches provide significant benefits. We also address challenges, including data sparsity, heterogeneity, and the need for integrated data collection protocols. We propose best practices for designing citizen science projects that support multimodal machine learning and identify future research directions.