NeFut Logo NeFut
中 Admin Login

[CS.AI] Rules, Traces, and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2

Published at: 2026-11-06 22:00 Last updated: 2026-11-07 12:32
#AI #LLM #Open Source

We evaluated three open‑source models on ARC‑AGI‑2 tasks under offline training and calibration constraints: four L4 GPUs for at most 12 hours. The study comprised 189 runs covering 120 hidden semi‑private tasks and offline comparisons on public benchmarks.\ \ For the NVARC Qwen3‑4B grid model we built a corpus of 12,000 executable rules in a typed language and 95,993 tasks. A rule is admitted only if its behavior on a fixed probe set differs from every already admitted rule, and a task is admitted only if its pairs refute a shortlist of nearest competing rules and defeat 12 shortcut solvers. Continued fine‑tuning on 1,920 real‑task records with a revised four‑GPU scheduler raised the semi‑private score from 28.96 (mean of the released model) to 33.06 (mean of five runs), best 34.86. No checkpoint trained solely on synthetic tasks, even all 95,993 of them, improved over the real‑task model.\ \ For the Qwen3.8‑27B reasoning model we fine‑tuned on 1,036 of its own reasoning traces containing correct final answers, and compared sets that added traces from other models and synthetic tasks. An inference pipeline built around the model (a "thinking cap" enforced in the serving engine, a Python program checked on the demonstrations after each answer, a long first attempt per input) lifted the score from 26.81 to 44.03, surpassing the ARC Prize Foundation’s time‑unlimited baseline of 42.36; most of the gain stemmed from the inference procedure.\ \ For the Qwen3.8‑Flash‑Next mixture‑of‑experts model (125B parameters, 6B active) GPU memory limited concurrency. Pruning experts per layer from 512 to 384 using router‑weighted activation saliency grew the key‑value cache from 0.55 M to 1.39 M tokens. A 296‑expert list adapted with ARC routing statistics, retaining experts for program tokens, plus an FP8 indexer cache, reached 1.96 M. Coupled with a tool that checks programs, the model scored 56.67, the best result in this study.\ \ Review: The results demonstrate that fine‑tuning on genuine task data remains the most effective way to boost performance on ARC‑AGI‑2, while synthetic rules or traces alone provide limited benefit. Engineering enhancements to the inference pipeline—such as the thinking‑cap mechanism and post‑answer verification—contribute substantially to score gains, especially for larger models. Expert pruning combined with routing statistics enables better cache utilization on constrained hardware, leading to higher task success rates.

Original Source: https://arxiv.org/abs/2611.06827

[h] Back to Home