Posted in | News | Machining Robotics

New Machine Learning Tool Helps Create a Realistic 2D Sketch

At Carnegie Mellon University’s Robotics Institute (RI), a machine learning tool engineered by scientists could enable beginner and professional designers to make 3D virtual models of everything ranging from tailored household furniture to video game content.

Pix2pix3d allows anyone to create a realistic, 3D representation of a 2D sketch using generative artificial intelligence tools similar to those powering popular AI photo generation and editing applications. Image Credit: Carnegie Mellon University.

Pix2pix3d enables everyone to make a realistic and 3D representation of a 2D sketch with the help of generative artificial intelligence tools that are quite similar to those powering famous AI photo generation and editing applications.

Our research goal is to make content creation accessible to more people through the power of machine learning and data-driven approaches.

Jun-Yan Zhu, Assistant Professor, School of Computer Science, Carnegie Mellon University

Zhu is a member of the pix2pix3d team.

Contrary to other tools that have the potential to create two-dimensional images, pix2pix3d is a 3D-aware conditional generative model enabling a user to input a two-dimensional sketch or more elaborate information from label maps, like an edge map or a segmentation.

Further, pix2pix3d synthesizes a 3D-volumetric representation of geometry, labels, and appearance that could be imparted from several viewpoints to make a realistic and three-dimensional image similar to a photograph.

As long as you can draw a sketch, you can make your own customized 3D model.

Kangle Deng, Robotics Institute Doctoral Candidate, Carnegie Mellon University

Deng was part of the research group with Zhu, Professor Deva Ramanan, and Ph.D. student Gengshan Yang.

Pix2pix3d has been qualified on data sets such as cats, cars, and human faces, and the team is working to extend those capabilities. Ahead, it can be utilized to design consumer products, like providing people the power to tailor furniture for their homes. Both beginner and professional designers could use it to tailor items in virtual reality surroundings or video games or to add effects to films.

As soon as pix2pix3d generates a 3D image, the user can alter it in real-time by deleting and redrawing the innovative two-dimensional sketch. This feature provides the user more freedom to tailor and improve the image without having to provide the entire project. In the 3D model, changes are reflected and are precise from multiple viewpoints.

Its interactive editing feature sets pix2pix3d apart from other modeling tools since users can make adjustments in a rapid and efficient manner. Particularly, this capability could be beneficial in fields like manufacturing since users could easily design, test, and adjust a product.

For instance, if a designer intakes a sketch of a car with a square hood, the model will offer a 3D rendering of such a car. If the designer wipes out that part of the sketch and substitutes the square hood with a round one, the 3D model updates instantly. Also, the team plans to continue refining and enhancing this feature in the coming days.

Deng noted that even the minimal artistic user will be able to achieve a satisfactory outcome. The model could generate a precise output of a simple or tough sketch. When offered highly accurate and elaborate data from edge maps or segmentation, the model can create a highly refined 3D image.

Our model is robust to user errors.

Kangle Deng, RI Doctoral Candidate, Carnegie Mellon University

Deng stated that the addition of even a 2D sketch that resembles a cat loosely will generate a cat’s 3D image.

3D-aware Conditional Image Synthesis (pix2pix3D)

3D-aware Conditional Image Synthesis. Video Credit: Carnegie Mellon University.


Tell Us What You Think

Do you have a review, update or anything you would like to add to this news story?

Leave your feedback
Your comment type
Azthena logo powered by Azthena AI

Your AI Assistant finding answers from trusted AZoM content

Your AI Powered Scientific Assistant

Hi, I'm Azthena, you can trust me to find commercial scientific answers from

A few things you need to know before we start. Please read and accept to continue.

  • Use of “Azthena” is subject to the terms and conditions of use as set out by OpenAI.
  • Content provided on any AZoNetwork sites are subject to the site Terms & Conditions and Privacy Policy.
  • Large Language Models can make mistakes. Consider checking important information.

Great. Ask your question.

While we only use edited and approved content for Azthena answers, it may on occasions provide incorrect responses. Please confirm any data provided with the related suppliers or authors. We do not provide medical advice, if you search for medical information you must always consult a medical professional before acting on any information provided.

Your questions, but not your email details will be shared with OpenAI and retained for 30 days in accordance with their privacy principles.

Please do not ask questions that use sensitive or confidential information.

Read the full Terms & Conditions.