Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification

Imagine you're building a system that needs to analyze data from various sources, like images, text, and audio, to make predictions. In real-world applications, you often face situations where some of these data sources might not be available due to issues like sensor failures or privacy concerns. Current methods typically focus on combining available data from two sources, but they struggle when the data is missing in unpredictable ways, which is known as missing arbitrary modalities. This can lead to poor performance when the system encounters unexpected gaps in data during operation.

To tackle this, the authors propose a multi-modal co-learning framework that emphasizes collaboration between different data types rather than just fusing them together. They introduce two methods that utilize information at both the feature and decision levels, allowing the system to adapt more effectively when certain modalities are missing. Their experiments on multi-modal classification tasks reveal that one method excels when only one modality is absent, while the other is better suited for scenarios where all but one modality is missing. This approach represents a significant step forward from previous work, as it provides a more flexible and robust solution for real-world applications where data availability can be inconsistent.

The approach shifts focus from fusion to co-learning in multi-modal settings.

Experiments on benchmarks show solid performance under various missing modality conditions.