Multimodal forecasting models that fuse time‑series data with textual annotations promise richer predictions by leveraging textual context. The key question is whether a given text annotation meaningfully contributes to the forecast, which is fundamentally an information‑theoretic problem. To assess whether information‑theoretic metrics can reliably quantify the predictive value of an annotation, a ground‑truth benchmark is required, yet none existed.
We generate a synthetic time‑series signal and attach annotations in three categories: semantically correct, semantically incorrect, and irrelevant. Because the data generation process is fully controlled, the true information content of each annotation is known exactly, enabling principled evaluation of six complementary mutual information estimators: KSG, MINE, InfoNCE, CCA, PID, and V‑information.
Results show that all six estimators identify the correct annotations as the most informative and can audit mixed text corpora, selecting annotations that yield the best downstream forecasting performance without any model training. The benchmark also exposes limitations of each estimator, which we validate on seven real‑world datasets, illustrating how estimator performance varies on weak signals.
Based on these findings we propose practical guidelines for implementing these metrics in annotation auditing and fusion selection.
Review