Training language‑model agents for long‑horizon strategy games suffers from sparse and delayed deterministic feedback; the final win/loss is only known at the end, while in‑game scores often fail to predict the outcome. In eight benchmark games, 8%‑68% of position pairs form a scoreboard trap where the higher‑scoring position leads to a worse result.\n\nFreeciv is a turn‑based empire‑building game where players manage units, cities and diplomacy over hundreds of turns. To obtain richer signals we built CivHarness, an open‑source system that snapshots the current GameState, branches on alternative actions and replays the game forward to future turns. Using this tool we trained CivTelescope, an LLM‑based judge that receives two position descriptions and predicts which one is more likely to yield a better final outcome.\n\nOn scoreboard‑trap pairs CivTelescope achieves 1.3‑2.0× higher accuracy than off‑the‑shelf language models and generalises to unseen openings, later turns and new rule sets. We then use CivTelescope as a dense reward to train a Freeciv agent named CivMarshmallow. In the mid‑game, where traps are common, CivMarshmallow gains an average of 8.94 points per training phase, a 77% improvement over the best baseline that relies only on the end‑of‑game reward. Over the full 120‑turn match the agent employs a hybrid reward that starts with the scoreboard or end reward and gradually shifts to CivTelescope; this hybrid scheme attains the highest score among all tested rewards.\n\nImplementation highlights\n- CivHarness: saves GameState, clones at branch points and invokes the Freeciv engine to replay to a target turn.\n- CivTelescope: prompt template Given two positions A and B, which one is more likely to lead to a better final outcome?, fine‑tuned on LLaMA‑2‑7B‑Chat.\n- CivMarshmallow: reward = α·Scoreboard + β·EndReward + γ·CivTelescope, with α, β, γ decaying linearly as the game progresses.\n\nReview: The replay‑grounded feedback effectively mitigates scoreboard traps and demonstrates a scalable approach for dense reward design in long‑horizon strategy reinforcement learning.