How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation
Maria Levchenko
Read on arXiv →
Key claim
Historical texts can be processed effectively with proper adaptation.
In plain English
Imagine you're trying to use a large language model to help with a digital library that contains historical texts. The challenge is that these texts often have different spellings, grammar, and vocabulary compared to modern language, making it hard for the model to understand them. Right now, people often treat this historical difficulty as a single problem, but that approach can lead to misunderstandings about how well the model can actually process these texts. For instance, you might think that if a model struggles with one old text, it will struggle with all of them, which isn't necessarily true. This is what's called a monolithic view of historical language processing.
What this paper does is break down the problem into four specific areas: how hard it is to tokenize the text, how uncertain the model is about its predictions, how robust the meaning is across different texts, and how sensitive the model is to the context it receives. By analyzing these factors across three different historical datasets, the authors found that while older Italian texts are more surprising to the model, it can still represent their meanings well. They also discovered that a simple prompt can reduce the model's uncertainty significantly, making it easier to work with these texts.
In practical terms, this means that while historical texts do present challenges, we can still use large language models effectively for tasks like semantic retrieval in digital libraries, as long as we adapt our approach to account for the unique characteristics of these texts.
The paper introduces a diagnostic framework that breaks down historical language processing challenges into distinct dimensions.
The evaluation is based on multiple datasets from different centuries, providing a solid basis for the claims made.
Deep reliability assessment
The methodology supports the narrower claim that tokenization cost, surprisal, embedding similarity, and prompt sensitivity can diverge on the sampled historical Italian and Russian corpora. The broader claim that digital libraries can safely deploy LLMs for semantic retrieval is more overclaimed because the excerpt reports proxy embedding similarity rather than downstream retrieval quality, and pretraining exposure/model choice cannot be fully audited.
Reproducibility
No open-source code repository or downloadable dataset URL is mentioned in the provided paper text. The paper describes a newly curated 17th-century Italian corpus digitized from page images, but the excerpt does not indicate whether the corpus, preprocessing scripts, prompts, or evaluation code are released.
Key figure
No Figure 1 is visible in the supplied excerpt; the key conceptual structure is a diagnostic framework separating historical-language difficulty into tokenization cost, predictive uncertainty, semantic robustness, and context sensitivity.