NeFut Logo NeFut
中 Admin Login

[CS.AI] Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit

Published at: 2026-09-24 22:00 Last updated: 2026-09-28 00:49
#AI #Machine Learning #LLM

Organizations are increasingly deploying large language models (LLMs) for policy, compliance, risk, and operational decision support. Such use cases demand evidence validation, review routing, version control, and auditability of model outputs. To address these needs we introduce Policy-as-Skill (PaS), a modular runtime that packages these governance functions as executable, versioned policy capabilities. PaS unifies evidence checking, review routing, policy referencing, and audit logging into a single decision pipeline via a consistent interface.

In our evaluation we fixed the backend to Gemma‑4 and tested 13 variants on 600 development tasks. PaS+Audit outperformed the baseline LLM+RAG on most governance metrics: exact accuracy 53.8%, macro‑F1 0.346, review F1 0.854, citation precision 1.000, policy‑reference recall 0.984, and audit completeness 1.000.

Deterministic control raised the aggregate accuracy to 61.2% but its benefit varied strongly across tasks, suggesting that rule‑based intervention should be applied selectively rather than universally.

Review: PaS provides a concrete pathway for embedding governance requirements into LLM‑driven decision making, especially excelling in evidence traceability and audit completeness. The task‑dependent nature of deterministic control highlights the importance of task‑level analysis when deciding whether to enable rule‑based overrides.

Original Source: https://arxiv.org/abs/2609.27087

[h] Back to Home