The “what you see is what you get” principle works well for traditional software, but generative AI often produces 3D models that look right yet fail to function—e.g., a mug that can’t hold coffee. To bridge this gap, researchers from MIT CSAIL, Google, and Northeastern University created InstructMesh, which couples the text‑and‑image‑to‑3D generator TRELLIS with the large language model GPT‑4. Users can generate a design, highlight parts they want refined, and let the system apply geometric changes in the model’s latent space based on natural‑language instructions.
In practice, a user might ask for a pair of glasses, then select the bridge area to widen it or the temples to add a decorative motif. The interface requires no prior 3D modeling expertise, making it accessible to novices.
Creative outputs demonstrated include a mug whose handle is a dragon’s tail, a shiny blue whistle shaped like a shell, glasses with butterfly‑wing extensions, an octopus‑style dispenser that pours liquid from each tentacle, a denim‑look knee brace matching a patient’s jeans, and a “bristle bot” resembling a colorful shrimp with a hidden motor that slides across surfaces.
In a validation study, TRELLIS recreated popular Thingiverse models; about 80% of the generated meshes contained structural flaws. Novice participants used InstructMesh to locate and fix these issues, achieving roughly a 90% success rate as judged by experts.
Key strengths of InstructMesh are:
- Natural‑language driven geometry edits via latent‑space manipulation;
- Slider controls for fine‑grained adjustments such as scaling or extruding specific components;
- Immediate visual feedback ensuring “what you see is what you get.”
The lead author envisions embedding InstructMesh in an augmented‑reality workflow: users describe a needed object in context (e.g., a phone case that matches their wallet), and the system rapidly generates and 3D‑prints it. Future work may add physics simulation to predict breakage or material suitability and integrate the newer TRELLIS.2 for even finer detail refinement.
Review