Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model

Imagine you're developing a language model that needs to understand words not just from text but also from images. Typically, these models start with random word meanings, which means they have to learn everything from scratch based on the text they see. This can lead to gaps in understanding, especially for words that are visually rich or context-dependent. For instance, a model might struggle with the nuances of words like 'banana' without any visual context, which is where the approach in this paper comes in. By initializing word embeddings with visual information from images that correspond to those words, the model can retain a clearer understanding of their meanings throughout training. This method, inspired by St. Augustine's idea of learning through observation, shows that visual grounding can leave a lasting imprint on the model's performance. However, while this visual initialization helps in specific tasks, it doesn't seem to improve performance on standard benchmarks that focus on abstract grammatical knowledge. The research highlights that the benefits of visual grounding are particularly evident in tasks related to object properties, suggesting a nuanced approach to training language models that could be more effective in real-world applications. For builders, this means that incorporating visual context into language models could lead to better understanding and performance in specific scenarios, even if traditional benchmarks don't fully capture these improvements.

The approach of visually grounding word embeddings is a meaningful extension of existing methods.

The findings are supported by causal tests, though benchmarks do not capture all effects.