Reward Machines (RMs) represent reward functions as automata, enabling the handling of non‑Markovian tasks where credit assignment depends on history within a Markovian RL framework. Traditional RMs require a manually engineered labeling function to map observations to propositions, which limits scalability.\ \ RM-VLM removes this bottleneck with two models. A large language model (LLM) generates an RM directly from a natural‑language task description. A vision‑language model (VLM) serves as the labeling function, grounding the RM’s propositions on image‑based observations and estimating the agent’s progress toward each transition. The VLM‑based value estimator provides dense progress signals that are turned into reward.\ \ Experiments cover three domains: non‑Markovian grid‑world navigation, robotic manipulation, and 3D object collection. Across all tasks, RM‑VLM consistently outperforms existing VLM‑based reward methods in success rate, confirming the benefit of coupling reward machines with visual language grounding.\ \ This approach demonstrates a full pipeline where language models define task structure and vision models handle perception, drastically reducing manual labeling effort and improving learning efficiency for temporally complex tasks.\ \ Review: RM‑VLM bridges task abstraction and perception, offering a scalable solution for non‑Markovian reinforcement learning that merits further testing in real‑world robotics and interactive settings.