In many scenarios where post‑training of large language models (LLMs) is desirable, high‑quality answers are easy to recognize but hard to score with explicit rewards. Reinforcement learning with verifiable rewards (RLVR) is limited to tasks that provide a clear correctness signal. Existing adversarial training approaches either rely on co‑trained critics alongside verifiable rewards or require reference reasoning traces. This work asks whether a co‑trained LLM judge can turn reference answers alone into the sole reward for on‑policy post‑training of reasoning LLMs. We name this approach Reinforced Adversarial Learning (RAL).
We first validate RAL on three verifiable tasks: integer multiplication, code‑based arithmetic‑expression synthesis, and SMILES‑to‑formula translation, using verifiers only for evaluation. Results show that with sufficiently capable judges and enough training steps, RAL matches the performance of direct RLVR.
We then apply RAL to scientific abstract generation in two settings. In the first, the generator produces abstracts that belong to the scientific domain of the reference abstracts even though the prompt does not specify the domain. In the second, abstracts are generated for U.S. NIH grant proposals conditioned on the grant title. Human scientists and an ensemble of LLM judges find our generations harder to distinguish from real abstracts than those of frontier‑LLM baselines.
These findings suggest a pathway toward post‑training reasoning models in scientific and other high‑value domains, where increasingly capable judges infer hard‑to‑specify reward functions from example answers. Review