Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation
Haozhe Wang, Weijia Feng, Jinpeng Yu, Che Liu, Ping Nie, Fangzhen Lin, Jiaming Liu, Ruihua Huang, Jimmy Lin, Wenhu Chen, Cong Wei
Read on arXiv →
Key claim
Co-training improves visual generation by addressing knowledge gaps.
In plain English
Imagine you're building a visual generator that creates images based on user requests. The challenge is that users often ask for things that the generator hasn't seen before, like new characters or recent events. This creates a problem where the generator confidently produces incorrect images because it lacks the necessary knowledge. This is what's called a knowledge boundary — the gap between what the generator knows from its training and what it needs to know to fulfill user requests accurately.
Currently, many visual generators are trained on fixed datasets, which means they can't adapt to new information or trends. When users ask for something outside of this training, the generator might try to guess, leading to poor results. This is particularly problematic in a world where requests are constantly changing and evolving. The authors point out that simply using search tools to fill in these gaps often backfires, as it can introduce irrelevant information that confuses the generator further. This failure mode is known as noise injection.
To tackle these issues, the authors propose a new approach that combines teaching the generator with searching for relevant information. This co-training framework allows the generator to learn from its mistakes and improve over time. Even a basic version of this method shows consistent improvements in performance, which means that visual generators can become more reliable in meeting user demands. This is a significant step forward compared to previous methods, as it lays the groundwork for ongoing enhancements in visual generation that are grounded in real-world knowledge.
The paper introduces a new framework for improving visual generation by addressing the knowledge boundary issue.
The claims are supported by a comprehensive dataset and experimental results, though some aspects could benefit from further validation.
Deep reliability assessment
The methodology supports the claim that integrating search tools can improve visual generation by addressing the knowledge bottleneck, but the effectiveness of the proposed co-training framework may be overclaimed without extensive real-world testing.
Reproducibility
Yes, the paper mentions the release of the full dataset, co-training corpus, and search corpus for reproducible research.
Key figure
Figure 1 illustrates representative search-augmented generations from SEARCHGEN-20K, highlighting the diverse user requests and the knowledge gaps in visual generation.