This paper reports a production migration of a customer‑support assistant at a large accommodation marketplace from a monolithic Qwen3-235B-A22B blended responder to a Dynamic Response (DR) architecture. The original system performed retrieval, action selection, escalation, and wording in a single model path, whereas the new system separates these roles:
- A bounded ReAct orchestrator dispatches typed tools (e.g., a reservation selector, escalation checker);
- A smaller generator produces replies only from a backend‑validated context contract.
The migration also altered prompts, alignment, and serving. To isolate architectural effects, the authors replayed identical turns and measured changes. Key findings:
- Typed entity selection moves the reservation selector to a precision‑first operating point (precision 8.3% → 89.1%, recall 75.2% → 67.3%);
- Typed action IDs with a membership check eliminate structured‑action hallucination (2.14% → 0%).
A low‑ramp A/B test reproduces escalation reductions: hard‑escalation responses drop from 5.60% to 3.08% and soft‑escalation from 9.56% to 2.49%, while handoff volume stays roughly steady; self‑solve shows a directional increase of about 5.1 points (95% CI [-2, +12]).
Serving optimizations cut orchestrator P90 latency from 3.87 s to 2.24 s with GPU footprint reduced by roughly one‑third, and self‑hosting lowers estimated annual model‑serving cost by more than an order of magnitude.
Review