The study indicates that a national language model provides its linguistic community with a tool to measure what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model, excels in agreement alone: when tasked with coding the moral foundation of authority, it agrees with trained human coders within a mere six F1 points, despite being only one-eighth to one-thirteenth the size of open models.
However, agreement reflects reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the key question is whether the model follows the construct's theory or reaches the correct code through correlated shortcuts.
We test this with the recovery gap: the performance loss when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. If calibration can close this gap, some portability should survive across models and languages; if not, the construct-model instrument is likely the site of failure.
We investigate whether a calibrated English instrument transfers to AMALIA-9B and to European Portuguese. The results indicate it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates, particularly moral outrage near authority figures.
In contrast, an open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, indicating the corpus is not the main explanation. While AMALIA can still screen and pre-code at scale, it cannot yet measure this construct effectively enough to stand alone.
This study serves as a single counterexample, not a verdict on national models; it argues that sovereign-LLM benchmark tests should assess not only agreement with human coders but also the evidential route through which that agreement is warranted.
Blogger's Review: This study reveals the potential limitations of national language models in terms of validity, especially in measuring moral constructs. While AMALIA performs well in agreement, its effectiveness remains to be validated, highlighting the need to focus on the reasoning paths of LLMs rather than just the outcomes. The findings provide significant insights for future model development and testing.