Revisions of aerospace electrical designs often contain multiple genuine changes, yet an engineering request may authorize only a subset. Propagating every detected difference therefore yields overly broad impact reports. To address this, we introduce LOGIC—a controlled benchmark and evaluation framework. LOGIC deploys locally runnable language models to ground a request in a deterministic candidate‑change inventory before propagating the selected changes through a typed electrical traceability graph. This separation allows candidate‑selection errors to be distinguished from downstream propagation errors. LOGIC comprises 168 scenarios, including 144 selection and 24 abstention cases. We evaluate three 7–8B models against intent‑agnostic, lexical, and structured‑evidence methods, using an oracle‑root upper bound.
On 96 explicitly anchored selection cases, gate‑only structured evidence achieves a candidate F1 of 1.0000, compared with 0.9677 for token‑lexical matching. On 12 relational‑paraphrase cases, token‑lexical F1 drops to 0.1772 and gate‑only to 0.0000, while the large language models attain F1 scores between 0.5000 and 0.6400. Model grounding degrades as the candidate inventory expands from 4 to 64 changes, yet affected‑element and typed‑path accuracy remain relatively stable over graphs ranging from ~1K to 100K nodes. Strict evidence gating suppresses false positives but can discard correct semantic selections. An evidence‑empty abstention policy raises strict abstention accuracy to 0.6667 and reduces unsafe‑report rates to 0.1667, at the cost of a 16.0–27.1 percentage‑point drop in answerable‑case coverage. Four of six conflicting requests remain unsafe for each model.
These findings support a hybrid approach that combines literal evidence, language‑model reasoning, and engineering review when intent cannot be reliably established, improving both safety and accuracy of change‑impact assessment.
Review