This paper introduces the Smart Coder Framework to structure human‑LLM collaborative qualitative coding. The framework consists of three phases: Instruction (explicit coding directives for the model), Prompt Strategy (designing few‑shot or few‑shot‑plus‑chain‑of‑thought prompts), and Reliability Check (evaluating coding stability via single‑pass or self‑consistency methods).
We evaluated the framework on 72 middle‑school mathematics teacher interview transcripts, using 12 transcripts for prompt development and calibration and the remaining 60 for formal testing. A factorial design compared instructed vs. uninstructed prompts, few‑shot vs. few‑shot‑plus‑CoT, and single‑pass vs. self‑consistency coding.
Instruction consistently improved inter‑coder agreement. Chain‑of‑thought prompting yielded additional gains when instruction was adequate, while self‑consistency offered modest, theme‑dependent improvements but proved useful for flagging unstable decisions for human review. Performance also varied with transcript characteristics.
In sum, the Smart Coder Framework systematically structures and monitors LLM‑assisted qualitative coding while preserving human authority over coding decisions.
Review