UEmbed: Unified Sparse and Dense Multimodal Embeddings

Imagine you're building a search system that needs to handle both text and images seamlessly. Current methods often rely on separate models for different types of data, which can lead to inefficiencies and missed connections between modalities. For instance, when searching for a product, a system might struggle to relate a text description to an image, resulting in less relevant results. This is what's called a lack of integration in multimodal retrieval systems, where the separation of dense and sparse representations can hinder performance.

To tackle this, UEmbed introduces a novel approach that combines both dense and sparse embeddings in a single model. By using a decoder-only architecture, it generates representations that can effectively capture the nuances of both text and images in one go. The model employs learnable special tokens that help predict sparse weights over different subsets of vocabulary, allowing it to create a comprehensive sparse vector from the input. This unified method not only enhances retrieval performance but also simplifies the architecture needed for multimodal tasks. Compared to previous models, UEmbed demonstrates improved effectiveness on benchmarks while maintaining efficiency, making it a compelling choice for developers looking to build more integrated search systems.

UEmbed introduces a unified approach to dense and sparse retrieval in multimodal contexts.

The model shows competitive performance on established benchmarks, indicating solid reliability.