Engineers often use vision-language models to produce new designs, such as for airplane or automobile components. To simulate how those components will perform in realistic situations, they’ll use tried-and-true computer-aided design (CAD) software to generate 3D models of those designs, which they can put through virtual crash or durability tests. Researchers from MIT and elsewhere have now developed a system that can teach a vision-language model to automatically convert 2D designs into CAD programs that are much more accurate and functional compared to other approaches, while using only a fraction of the computation.
By improving the performance and efficiency of AI-driven CAD generation, this technique could streamline the rapid prototyping process and reduce costs. It could also help engineers identify beneficial design choices they might otherwise overlook.
The system generates new data based on the model’s abilities as it attempts to convert a 2D image into a CAD program. The framework corrects the model’s failures and incorporates them into a dataset with its successful solutions. It uses these data to teach the model how to fix specific mistakes and tackle tricky problems it would struggle with on its own. “We want engineers to be able to point our framework at an underperforming CAD model, set a compute budget, and let the system take over — turning the model’s own mistakes into better training data,” says lead author Giorgio Giannone.
The researchers are working toward building vision-language models (VLMs) for CAD generation. These VLMs take a 2D image and some descriptive text and output Python code that can be executed in a CAD software program to generate a 3D model of a physical object. They studied the challenges of deploying existing VLMs for this task and determined the main bottleneck that limits their capabilities is the lack of diverse, high-quality CAD datasets to train them. To remedy this, they sought to create new data to teach a model how to perform CAD generation, using a process known as data augmentation.
MIT researchers built a data augmentation system called GIFT (Geometric Inference Feedback Tuning) that generates data designed to improve the performance of a VLM for a specific task. GIFT develops an understanding of the model’s strengths and weaknesses by testing it. Then it uses this knowledge to generate data that could improve the model’s performance on the CAD generation problems it struggles to solve.
GIFT asks the model to generate code that solves a CAD generation problem multiple times in parallel. It checks the correctness of these guesses to understand how well the model can solve this problem. “For a model, generating CAD query code that is almost correct is not that hard, but generating code that is perfectly correct and can be executed is much more challenging for a standard VLM,” Giannone says. For guesses that are nearly correct, GIFT adjusts them to become successful solutions. It saves these “near-misses” and successful solutions in a new dataset that can teach the model how to overcome problems that would usually trip it up.
This automatic system does not require human intervention to correct the model’s mistakes. GIFT creates data augmentations from a pre-trained VLM using a process known as inference-time scaling. This process allows a static model, which has already been trained, to generate better outputs without the high computational costs of retraining the entire model. Using inference-time scaling, the user can determine how much computation they want to use for GIFT, tailoring it to their time and budget constraints.
GIFT outperformed several competing techniques, generating CAD programs that were more accurate while using only about 20 percent as much computation. The CAD models generated by VLMs using GIFT were better aligned with the shapes of ground-truth models. “With GIFT, we started with geometry because with engineering problems, if the geometry of a 3D shape is not correct, nothing else will be correct, but there are many other aspects to consider,” Giannone says.
In the future, the researchers want to expand GIFT so the framework can teach models to generate CAD programs that improve the performance and manufacturability of 3D models. They also want to apply the system to larger models and more diverse CAD generation tasks. This research was funded, in part, by the MIT-IBM Computing Research Lab.
Blogger's Review: MIT's breakthrough in CAD generation through self-learning and data augmentation techniques significantly enhances model accuracy and efficiency. This advancement not only expedites product design processes but also holds the potential to transform the future of engineering design, making it a development to watch closely.