MGAL is the first multilingual, granularity‑ and position‑aware benchmark for long‑context evaluation. It is built from United Nations reports in the six official UN languages, with document lengths ranging from 8K to 128K tokens. The dataset is annotated at four linguistic granularity levels—word, sentence, paragraph and document—and each entry is further indexed by its position within the document (beginning, middle, end) at both document and paragraph levels, enabling systematic diagnosis of multilingual long‑context comprehension across different granularities.
Experiments reveal that current LLMs perform well on word‑level tasks but struggle markedly on coarser levels such as sentence, paragraph and document; closed‑source models retain a clear advantage in lower‑resource languages.
Two new challenges are identified. First, under local semantic crowding where neighboring sentences share topics and entities, models tend to rely on surface cues (e.g., the connective "however" or repeated entities) rather than the discourse role of the sentence (background, outcome, etc.). Second, a fluency‑consistency gap appears: generated text reads smoothly but drifts from the source facts.
These observations align with prior studies, showing that models depend heavily on nearby evidence and reuse options when uncertain.
Blogger's Review: MGAL offers a fine‑grained perspective for multilingual long‑context evaluation, exposing the shortcomings of existing LLMs in cross‑language and cross‑granularity understanding and pointing to concrete directions for future model improvements.