NeFut Logo NeFut
Admin Login

[CS.AI] A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

Published at: 2026-09-18 22:00 Last updated: 2026-09-20 12:54
#AI #Machine Learning

Benchmark scores alone give a partial view of the trustworthiness of modern AI systems. Large language models (LLMs), agentic systems and multimodal models (MLLMs) each demand distinct assessment methods, yet the evidence must stay interpretable for development and oversight. We introduce a unified framework that links output‑level, trajectory‑level and cross‑modal evaluation through eight trustworthiness dimensions: capability, robustness, safety, fairness, transparency, governance, oversight and efficiency. The framework retains system‑specific metrics while mapping native measurements to common performance bands, adding uncertainty estimates and traceable evidence. A meta‑evaluation layer checks the validity, reliability and reproducibility of the evaluation itself. Multidimensional profiles expose strengths and weaknesses, and safety‑critical overrides stop aggregate scores from hiding critical failures. Mappings to governance frameworks, international standards and EU regulatory requirements bridge technical assessment with oversight needs. The framework offers a structured basis for judging both system performance and the credibility of supporting evidence, while empirical validation in deployment contexts remains a necessary next step.

Review

Original Source: https://arxiv.org/abs/2609.19524

[h] Back to Home