This study explores the effective application of large language models (LLMs) in the field of academic supervision. While LLMs produce fluent responses to single-shot prompts, deploying them as reliable components of a domain decision system poses significant challenges. To bridge this gap, we introduce 'harness engineering,' which involves constructing deterministic scaffolding around the LLM core, including symbolic filters, retrieval, schema-typed I/O, LLM-as-judge loops, human-in-the-loop (HITL) gates, persistent state, and audit trails.
We conducted a case study comparing a baseline model (ASA), a GPT-5 chatbot without scaffolding, against a multi-module system (ASuS) that encapsulates the smaller GPT-4o-mini within a LangGraph harness, featuring symbolic-semantic retrieval, schema-validated outputs, LLM-as-judge with bounded retry, HITL gates, deterministic weighted risk scoring, and per-node SQLite audit trails.
The evaluation rubric was retargeted at six harness-mechanism dimensions (grounding, explainability, consistency, process integrity, cognitive load, constraint adherence). A blind ten-rater hybrid evaluation, supplemented by a 2 x 2 model-harness ablation, found that ASuS, despite using a much smaller base model, outperformed ASA across every dimension. The pooled mean for ASuS was 4.08 compared to 1.23 for ASA, with 8 of 10 raters rejecting the null hypothesis at alpha = 0.05; full numbers can be found in Sections 6.4 and 6.7. The ablation confirms that the structural contributions of the harness are largely model-invariant.
We extract seven recurring harness-engineering patterns and argue that when reliability, traceability, and institutional consistency matter more than open-ended fluency, harness engineering challenges the prevailing 'bigger model is better' intuition.