NeFut Logo NeFut
Admin Login

[CS.AI] SKIP: A Self-knowledge-guided Step-wise Preference Learning Framework for Concise Reasoning

Published at: 2026-09-16 22:00 Last updated: 2026-09-18 00:46
#Machine Learning #optimization #LLM

Chain‑of‑Thought (CoT) reasoning improves the problem‑solving ability of large language models (LLMs) but often leads to overthinking, incurring extra computation, latency, and even performance drops in very large models. Existing concise‑reasoning methods reduce output length at the cost of noticeable accuracy loss.\ \ We introduce SKIP (Self‑knowledge‑guided step‑wise Preference learning). First, a lightweight fine‑tuning stage adjusts the model’s style to emit an answer at each reasoning step. Then a knowledge probing mechanism forces the model to produce a concrete answer after every intermediate step, yielding verifiable sub‑answers. Based on the correctness of these sub‑answers, we build preference data and apply Direct Preference Optimization (DPO) to steer the model toward more efficient and correct reasoning while preserving overall accuracy.\ \ Extensive experiments on several benchmarks show that SKIP dramatically compresses the number of reasoning steps with minimal accuracy degradation and mitigates the typical fine‑tuning performance drop. Moreover, SKIP generalizes well to out‑of‑distribution datasets. Ablation studies on key hyper‑parameters (fine‑tuning learning rate, probing threshold, DPO weight) confirm the contribution of each component.\ \ Review: SKIP cleverly combines self‑knowledge probing with preference learning, offering a practical trade‑off between efficiency and correctness that is highly relevant for deploying large models in real‑world scenarios.

Original Source: https://arxiv.org/abs/2609.17019

[h] Back to Home