Robots walking down the street, surrounded by astounded onlookers, is an increasingly common sight. However, these machines aren’t yet the do-it-all assistants you’d want working in a kitchen or factory, and a major bottleneck is data. Like humans, robots learn best by experience. The challenge is that it’s labor-intensive and time-consuming to physically teach these machines so many actions across different settings. "One natural idea is to use simulation as a training ground," says Russ Tedrake, the Toyota Professor of Electrical Engineering and Computer Science at MIT. AI agents, or semi-autonomous programs that "think" and complete well-defined tasks, could help produce the lifelike virtual settings that robots need.
The new “SceneSmith” system developed by researchers at MIT CSAIL and Toyota Research Institute uses three agents to piece together the objects, walls, and overall look of a 3D scene. Its recreations of indoor spaces such as restaurants, bedrooms, and hotels are more realistic and detailed than prior systems, helping robots practice skills and try out different ways of doing tasks before they’re powered on, saving engineers time on real-world testing.
The agents call on a multi-modal system called a vision-language model (VLM), specifically the state-of-the-art VLM GPT-5.2, which is trained on lots of text and images from the internet. This advanced model gives each agent a sort of spatial knowledge: First, a “designer” agent generates the elements of a scene, then a “critic” advises on realism, and finally, an “orchestrator” manages their back-and-forth, deciding when the design is complete. Once the three VLMs finish their collaboration, the scene is ready to load into physics simulation software.
“We’ve found that the system can construct 3D scenes the way a human designer would,” says MIT EECS PhD student Nicholas Pfaff. The researchers created over 1,300 scenes using a leading VLM that has internet-scale priors, generating incredibly creative and diverse arrangements. Users can ask SceneSmith to generate things like “a garage with a car, a workbench, tires stacked in the corner, and a ladder against the wall,” resulting in a rich virtual playground for robots. These rooms contain up to six times more items per scene than prior methods, making them ideal for helping robots learn skills such as putting a cup in the sink and moving a soda can from a shelf to a table.
The researchers tested different action plans in SceneSmith’s digital worlds, generating 100 unique spaces in the process. A VLM agent evaluated each attempt, finding that the robot’s plans were often faulty. Humans agreed with the model’s evaluations over 99 percent of the time, helping roboticists weed out flawed approaches in simulation before a robot moves in the real world.
But how realistic are these virtual worlds? To answer this, the researchers conducted several tests. The most telling test involved dropping a pretrained robot policy—an AI controller trained largely on real-world data—into the generated environments. In one test, users instructed the system to “take the apple from the bowl and place it onto the cutting board,” and the simulated robot executed the task perfectly. If the scenes didn’t closely resemble the real settings the policy had learned from, it wouldn’t have worked. The team also teleoperated robots through the virtual spaces, guiding them to open cabinets and navigate between rooms. Their experiments showed that the environments held up under sustained physical interaction.
SceneSmith’s agents each have a well-defined role in the generative process, fleshing out scenes in stages. For example, the “designer” VLM starts with a general layout, which is reviewed by the “critic,” and then signed off by the “orchestrator.” They repeat this approach for each step: adding furniture, placing objects on walls and ceilings, and finally, dropping in manipulable objects. For instance, VLMs can add cabinets that robots can open and close—articulated items that prior baselines didn’t often include. At each stage, the second VLM ensures the scene is practical, while the third VLM ensures high quality, even reverting the design process if the visuals aren’t satisfactory.
Once the three VLMs finalize their collaboration, the mechanics of the physical world are added via simulation software. With a sound understanding of how rooms should look and where objects should be placed, SceneSmith has a noticeable edge over prior methods. Compared to scene-generation baselines like “HSM” and “Holodeck,” SceneSmith produced environments with more objects, including a private office and a Minecraft-themed gaming room. Over 200 users found SceneSmith’s visuals to be more realistic over 90% of the time, and it generally adhered more closely to prompts than other approaches.
Realism, diversity, and richness are all strong suits for SceneSmith, even in generating individual 3D objects. You can prompt it to create a rolling serving cart, and it will produce a 2D image that it then turns into a detailed model with physical properties. This detailed process does come with a speed trade-off, as generating a single scene can take multiple hours. With more computing power, the system could see dramatic increases in efficiency. CSAIL engineers are also looking to expand to deformable objects should extensive 3D libraries become available.
“SceneSmith represents a significant advance in this regard by providing an agentic framework for generating simulation-ready indoor environments from a simple text prompt,” says Jeremy Binagia, an applied scientist at Amazon Robotics. It advances the state of the art in several ways, including enhancing the density of objects in the simulated environment and ensuring all objects are physically accurate. Pfaff and Tedrake co-authored the paper with Thomas Cohn and Toyota Research Institute roboticists Sergey Zakharov and Rick Cory. Their work was supported by Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation. The team presented their findings at the recent International Conference on Machine Learning.