Precise-Physics Driven Text-to-3D Generation

Text-to-3D generation has shown great promise in generating novel 3D content based on given text prompts. However, existing generative methods mostly focus on geometric or visual plausibility while ignoring precise physics perception for the generated 3D shapes. This greatly hinders the practicality of generated 3D shapes in real-world applications.

In this article, we propose Phy3DGen, a precise-physics-driven text-to-3D generation method. By analyzing the solid mechanics of generated 3D shapes, we reveal that the 3D shapes generated by existing text-to-3D generation methods are impractical for real-world applications as the generated 3D shapes do not conform to the laws of physics.

1 Introduction

It is critical for a variety of applications, including personalized customization, film-making, robotics simulation, gaming, and so on. Recently, text-to-3D generation have achieved promising results with the development of generative models (e.g., generative adversarial networks and diffusion models) and 3D representations ( e.g., DeepSDF and neural radiance fields). Existing text-to-3D generation methods mostly focus on the geometric or visual realism of generated 3D shapes.

However, these methods do not incorporate precise physics information, most 3D shapes generated are ill-defined and impractical for real-world applications. Therefore minimizing the maximum stress and uniformly distributing the stress throughout the geometry can reduce susceptibility to failure. This is highly valuable in engineering design. This paper aims to achieve geometry generation and optimization, considering not only computer vision but also physics. This approach ensures that the obtained geometry not only satisfies visual preferences but also meets engineering requirements to a certain extent.

2 Challenges

However, it is challenging to incorporate precise physics information into text-to-3D generation methods. On the one hand, in order to optimize geometry with physics, a differentiable physics solver is expected to compute the solid mechanics for the intermediate 3D shapes on the fly.

On the other hand, recent Physics Informed Neural Networks (PINN) provide a new direction to solve Partial Differential Equations (PDE) describing the laws of physics. However, their precision is poor for complex geometries. This further prevents precise physics perception for generative 3D methods.

3 Proposed Method

To address the aforementioned challenges, we propose a precise-physics-driven text-to-3D generation method to inject the physical laws into the generative 3D modeling. Specifically, our method contains two stages. In the first stage, we initialize a 3D shape based on 3D diffusion models and convert it into implicit geometry representations, Signed Distance Function (SDF).

This allows our approach to be flexibly combined with existing generative methods. In the second stage, we design a data-driven differentiable physics layer to enable geometry and physics optimization at the same time. Our differentiable physics layer is parameterized by neural networks and used to learn the solid mechanics of generated 3D shapes on demand when optimizing geometries.

To guarantee the precision of the feedback physics information, we leverage the FEM results computed for the initial generated 3D shape to initialize our differentiable physics layer. We introduce a relaxed geometry loss and a series of physics losses during training to guide the optimization.

4 Conclusion

Overall, our contributions are summarized as follows: We propose a precise-physics-driven text-to-3D generation method, called Phy3DGen, to incorporate the physical laws into generated 3D shapes.

Experiments demonstrate that our method can improve geometric or visual preferences without requiring extra training data and manual interaction and endow generated 3D shapes with precise physical perception capabilities.

Given a text describing our target object, we aim to generate a 3D shape that simultaneously satisfies geometric plausibility and precise physics perception. For this purpose, we propose a precise-physics-driven text-to-3D generation method.