Recent advances have markedly improved the performance of Abstract Meaning Representation (AMR) parsers, yet systematic guidance on when their outputs can be trusted is still lacking. This work presents the first uncertainty estimation study for AMR parsers in a black‑box setting using top‑k beam‑search candidates. Uncertainty is quantified by measuring pairwise similarities among candidates and evaluated with the Prediction Rejection Ratio (PRR).
The experiments first map the relationship between parsing performance and uncertainty, showing that sentences with higher uncertainty tend to receive lower Smatch scores. Next, the reliability of the beam candidate distribution is examined, revealing that diversity within the candidate set reflects the parser's confidence. We then analyze how different evaluation metrics (e.g., Smatch, S2S, BLEU) affect uncertainty assessment, finding that metric choice can alter the correlation between uncertainty and performance. Finally, several mainstream AMR parsers are compared, identifying models that achieve the best trade‑off between performance and reliability. Overall, the results demonstrate that uncertainty estimation serves as a complementary dimension for evaluating AMR parsers and offers practical guidance for downstream applications.
Review: Uncertainty estimation provides a practical tool for assessing the trustworthiness of AMR parsing, especially valuable in black‑box scenarios.