Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou
Read on arXiv →
Key claim
Hindcast closes data leakage in forecasting evaluations.
In plain English
Forecasting models are often evaluated using backtesting, but this can be flawed due to data leakage from future information. Current methods can unfairly benefit models that retrieve data from after an event has occurred. Hindcast addresses this by evaluating models based on a fixed past snapshot, ensuring that only prior information is considered. Builders might find this approach valuable for developing more reliable forecasting systems that are less biased by future data.
Introduces a novel evaluation method that mitigates data leakage in forecasting.
Employs a solid methodology with clear baselines for evaluation.
Deep reliability assessment
The methodology supports evaluating LLMs' forecasting ability by simulating past conditions, but it may overclaim foresight by not fully accounting for the limitations of the fixed archive and the short pre-resolution lookback period.
Reproducibility
No open source code or dataset is mentioned in the paper.
Key figure
Figure 1 illustrates the HINDCAST method by showing how a model's forecast is compared to market probabilities using only pre-event Reddit data.