The AI tool that could make manufacturing faster and more efficient—by using Lego bricks

A new AI-powered tool created by researchers at Carnegie Mellon University’s School of Computer Science could change the way we manufacture and build things.

BrickGPT uses text prompts to help people—and even robots—bring ideas to life with Lego bricks. It takes a simple word, such as “guitar,” and creates a brick-by-brick guide for a person or a robot to build a physically stable model of that object. The tool is currently focused on building with Lego bricks, but turning a text prompt into something physically stable goes beyond play.

“This research paves the way toward generative manufacturing, which is when people can use a generative model to design everyday objects they can build themselves,” said Jun-Yan Zhu, the Michael B. Donohue Assistant Professor of Computer Science and Robotics in CMU’s Robotics Institute (RI).

“They can build a chair, a sofa or a kids’ toy. This is a new frontier, a new usage of these models beyond creating social media videos or photos. These brick toy pieces are a simple medium, and it’s a starting point.”

Researchers said this fusion of AI and robotics could speed up the process of designing and building new things.

“This could be a huge benefit to the manufacturing world,” said Changliu Liu, an associate professor at the RI. “It takes a long time to turn ideas into a physical design and prototype. But if you can integrate generative AI into the process, it can significantly improve efficiency and reduce the roadblocks to kicking off projects.”

Currently, the BrickGPT demo can produce step-by-step guides for humans or robots to build 21 types of models out of Lego bricks, including a birdhouse, sofa and piano. If someone wanted to generate a sofa, they would type “sofa” into BrickGPT, which generates a 3D model. Then, an algorithm transforms the 3D model into brick structures and BrickGPT checks to ensure the structure is stable. A person or robotic arm can follow the steps and build the sofa.

To train BrickGPT, researchers generated StableText2Brick, a dataset containing more than 47,000 brick structures made from more than 28,000 unique 3D objects accompanied by detailed captions. Researchers took an existing dataset of 3D shapes, ShapeNetCore, and converted these shapes into a grid of small cubes, a process the study describes as voxelizing. They then trained an autoregressive large language model (LLM), which predicts future values based on previous ones.

For example, in BrickGPT, the LLM predicts the next brick based on the previous one, ensuring that the structure is stable and won’t fall over. If there’s an error along the way, BrickGPT goes back and eliminates unstable points to guarantee the structure’s stability.

Along with Liu and Zhu, the SCS research team includes Ava Pun, a doctoral student in the Computer Science Department; Kangle Deng and Ruixuan Liu, doctoral students in the RI; and Deva Ramanan, a professor in the RI.

“If a structure is unstable, there’s a rollback process,” said Pun. “During that step, the model determines which bricks were wrong or unstable and we roll back to the point before that. We detect instability with our physics reasoning algorithm, which generates a stability score for each brick in the structure. If the score is high enough, that means that brick was stable.”

Researchers hope to scale up this model, enabling it to generate more than the current 21 objects. They also hope to expand the diversity of their library pieces to increase the accuracy and complexity of generated designs.