3D Machine Learning

In recent years, tremendous amount of progress is being made in the field of 3D Machine Learning, which is an interdisciplinary field that fuses computer vision, computer graphics and machine learning. This repo is derived from my study notes and will be used as a place for triaging new research papers.

I'll use the following icons to differentiate 3D representations:

• 📷 Multi-view Images
• 👾 Volumetric
• 🎲 Point Cloud
• 💎 Polygonal Mesh
• 💊 Primitive-based

Get Involved

To make it a collaborative project, you may add content throught pull requests or open an issue to let me know.

Available Courses

Stanford CS468: Machine Learning for 3D Data (Spring 2017)

MIT 6.838: Shape Analysis (Spring 2017)

Princeton COS 526: Advanced Computer Graphics (Fall 2010)

Princeton CS597: Geometric Modeling and Analysis (Fall 2003)

Datasets

To see a survey of RGBD datasets, I recommend to check out Michael Firman's collection as well as the associated paper, RGBD Datasets: Past, Present and Future. Point Cloud Library also has a good dataset catalogue.

Single Object Classification

to be added

Multiple Objects Detection

to be added

Part Segmentation

to be added

3D Synthesis/Reconstruction

Parametric Morphable Model-based methods

A Morphable Model For The Synthesis Of 3D Faces (1999) [Paper][Github]

The Space of Human Body Shapes: Reconstruction and Parameterization from Range Scans (2003) [Paper]

Part-based Template Learning methods

Modeling by Example (2004) [Paper]

Model Composition from Interchangeable Components (2007) [Paper]

Data-Driven Suggestions for Creativity Support in 3D Modeling (2010) [Paper]

Photo-Inspired Model-Driven 3D Object Modeling (2011) [Paper]

Probabilistic Reasoning for Assembly-Based 3D Modeling (2011) [Paper]

A Probabilistic Model for Component-Based Shape Synthesis (2012) [Paper]

Structure Recovery by Part Assembly (2012) [Paper]

Fit and Diverse: Set Evolution for Inspiring 3D Shape Galleries (2012) [Paper]

AttribIt: Content Creation with Semantic Attributes (2013) [Paper]

Learning Part-based Templates from Large Collections of 3D Shapes (2013) [Paper]

Topology-Varying 3D Shape Creation via Structural Blending (2014) [Paper]

Estimating Image Depth using Shape Collections (2014) [Paper]

Single-View Reconstruction via Joint Analysis of Image and Shape Collections (2015) [Paper]

Interchangeable Components for Hands-On Assembly Based Modeling (2016) [Paper]

Shape Completion from a Single RGBD Image (2016) [Paper]

Deep Learning Methods

📷 Learning to Generate Chairs, Tables and Cars with Convolutional Networks (2014) [Paper]

🎲 Analysis and synthesis of 3D shape families via deep-learned generative models of surfaces (2015) [Paper]

📷 Multi-view 3D Models from Single Images with a Convolutional Network (2016) [Paper] [Code]

📷 View Synthesis by Appearance Flow (2016) [Paper] [Code]

👾 Voxlets: Structured Prediction of Unobserved Voxels From a Single Depth Image (2016) [Paper]

👾 3D-R2N2: 3D Recurrent Reconstruction Neural Network (2016) [Paper]

👾 TL-Embedding Network: Learning a Predictable and Generative Vector Representation for Objects (2016) [Paper]

👾 3D GAN: Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016) [Paper]

📷 Unsupervised Learning of 3D Structure from Images (2016) [Paper]

📷 Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency (2017) [Paper]

📷 Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks (2017) [Paper]

👾 Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017) [Paper]

🎲 A Point Set Generation Network for 3D Object Reconstruction from a Single Image (2017) [Paper]

📷 Transformation-Grounded Image Generation Network for Novel 3D View Synthesis (2017) [Paper]

👾 Interactive 3D Modeling with a Generative Adversarial Network (2017) [Paper]

Style Transfer

Style-Content Separation by Anisotropic Part Scales (2010) [Paper]

Design Preserving Garment Transfer (2012) [Paper]

Analogy-Driven 3D Style Transfer (2014) [Paper]

Elements of Style: Learning Perceptual Shape Style Similarity (2015) [Paper]

Functionality Preserving Shape Style Transfer (2016) [Paper]

Unsupervised Texture Transfer from Images to Model Collections (2016) [Paper]

Learning Detail Transfer based on Geometric Features (2017) [Paper]