CIMS Core Faculty: Oliver van Kaick
In the Semantic understanding of indoor environments project, we are employing a variety of computational methods to obtain a semantic understanding of digital representations of building interiors, that is, to automatically discover the objects and architectural features that are present in the 3D scan of a room.
Indoor environments such as a room in a building can be digitized through a 3D scanning process. The result of the scanning process is a representation of the room as a large set of 3D points. Thus, our goal is to automatically discover the objects present in this scan. However, indoor environments can be challenging to analyze given that they may contain a variety of geometric shapes in arbitrary configurations, including architectural elements such as doors and windows, but also objects such as furniture, works of art, and miscellaneous artifacts. In addition, acquiring rooms through a 3D scanning process can further introduce noise and other sampling artifacts that make the processing difficult.
The problem of analyzing a scan can be formulated as a shape segmentation problem: our goal is to partition the points in the scan into groups, and assign semantic labels to the different groups, indicating which points are part of what objects, for example, which points constitute a chair, door, etc. In this project, we are using supervised and unsupervised machine learning methods to automatically segment rooms composed of multiple objects, given that these methods are robust to noise and other defects in the scans.
The first step of our pipeline is to recognize the different categories of objects present in the room. For this task, we are applying a state-of-the-art supervised machine learning method known as a deep neural network to segment the room's scan and assign semantic labels to the points. This approach requires the careful preparation of training data in the form of labeled scenes. Once the network is trained with this data, it can be applied to predict the segmentation of any scan provided as input. Although manually creating the training data can be a laborious task, the benefit of this approach is the high accuracy of the results obtained, as shown below.
(a) Input room
(b) Prediction provided by the deep network, where each color denotes a different class label
(c) Ground-truth prepared by a human for comparison.
The second step of the pipeline is to apply unsupervised learning methods such as clustering to further partition each class into individual objects, for example, to partition all points labeled as “chair” into individual chairs.
Finally, in subsequent steps of the pipeline, we refine the segmentation by applying specialized techniques to recognize specific types of objects. Here, we are experimenting with combining modern learning approaches with classic shape analysis methods. For example, we are using plane fitting methods to partition walls into logical subgroups (front wall, side wall, etc.), and geometry detection methods to recognize individual objects such as chairs, as illustrated below.
Walls segmented according to plane fitting methods.
(a) Input room seen from two different views
(b) Detected chairs
I am an Associate Professor at the School of Computer Science at Carleton University in Ottawa, Canada. My research is concentrated in computer graphics, especially in the areas of geometric modeling and shape analysis. These areas involve the development of mathematical models and computer algorithms for analyzing and processing geometric data, especially digital representations of 3D shapes.
Examples of research problems that I addressed in my past work include shape segmentation (decomposing a shape into meaningful parts), shape correspondence (computing a mapping between two or more shapes), shape similarity estimation (quantifying how similar shapes are), and inferring shape functionality (automatically understanding how shapes can be used). Obtaining solutions to these problems can be useful for a variety of applications that require organizing or modifying geometric datasets, for example, shape retrieval (where the goal is to find a specific object in a large collection of objects) or shape editing (where the goal is to let a user more easily manipulate and modify a geometric dataset). The methods developed in my research often involve a combination of geometric analysis, optimization, and machine learning.
In my collaboration with Carleton's Immersive Media Studio (CIMS), our team is developing methods for analyzing geometric data in the context of architecture. Specifically, we are employing supervised and unsupervised machine learning methods to obtain a semantic understanding of building interiors.