Abstract
Zero-shot summarization using Large Language Models (LLMs) has significantly advanced the abstractive summarization task by producing coherent and fluent summaries. However, the underlying stochasticity of LLMs raises concerns about the stability and trustworthiness of the generated summaries. This issue has become increasingly important due to the proliferation of LLM-generated summaries in educational settings, where students and researchers summarize complex academic materials in a zero-shot manner.
We propose a novel two-level diagnostic protocol for benchmarking LLM-summarizers based on the stability of the generated summaries. At the lower level, document-level stability analysis is performed over multiple LLM-summaries generated under a controlled environment, and the stability coefficient is computed. Each generated summary is scored for semantic and factual alignment with the original document, enabling estimation of stability across multiple dimensions.
At the next level, observations from a stratified sample of documents drawn from the corpus are consolidated to estimate the stability index of the LLM-summarizer, which serves as a proxy for its trustworthiness. Our empirical investigation of three LLM-summarizers across three genres of documents reveals statistically significant differences in generation-level variability among LLMs across summary evaluation metrics. This study advances the LLM-summarization research by evidential recognition of the stability problem in LLM-summaries and motivates further research towards the development of robust, reliable, and trustworthy LLM-summarizers.
Blogger's Review: This paper highlights the stability issue in LLM summarization and proposes an innovative evaluation method, laying the groundwork for future research. With the widespread application of LLM technologies, ensuring the trustworthiness of generated content becomes increasingly important, warranting further attention and exploration.