-
Panoptic 3D Scene Reconstruction From a Single RGB Image
Authors:
Manuel Dahnert,
Ji Hou,
Matthias Nießner,
Angela Dai
Abstract:
Understanding 3D scenes from a single image is fundamental to a wide variety of tasks, such as for robotics, motion planning, or augmented reality. Existing works in 3D perception from a single RGB image tend to focus on geometric reconstruction only, or geometric reconstruction with semantic segmentation or instance segmentation. Inspired by 2D panoptic segmentation, we propose to unify the tasks…
▽ More
Understanding 3D scenes from a single image is fundamental to a wide variety of tasks, such as for robotics, motion planning, or augmented reality. Existing works in 3D perception from a single RGB image tend to focus on geometric reconstruction only, or geometric reconstruction with semantic segmentation or instance segmentation. Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations. We thus propose a new approach for holistic 3D scene understanding from a single RGB image which learns to lift and propagate 2D features from an input image to a 3D volumetric scene representation. We demonstrate that this holistic view of joint scene reconstruction, semantic, and instance segmentation is beneficial over treating the tasks independently, thus outperforming alternative approaches.
△ Less
Submitted 16 May, 2022; v1 submitted 3 November, 2021;
originally announced November 2021.
-
Joint Embedding of 3D Scan and CAD Objects
Authors:
Manuel Dahnert,
Angela Dai,
Leonidas Guibas,
Matthias Nießner
Abstract:
3D scan geometry and CAD models often contain complementary information towards understanding environments, which could be leveraged through establishing a mapping between the two domains. However, this is a challenging task due to strong, lower-level differences between scan and CAD geometry. We propose a novel approach to learn a joint embedding space between scan and CAD geometry, where semanti…
▽ More
3D scan geometry and CAD models often contain complementary information towards understanding environments, which could be leveraged through establishing a mapping between the two domains. However, this is a challenging task due to strong, lower-level differences between scan and CAD geometry. We propose a novel approach to learn a joint embedding space between scan and CAD geometry, where semantically similar objects from both domains lie close together. To achieve this, we introduce a new 3D CNN-based approach to learn a joint embedding space representing object similarities across these domains. To learn a shared space where scan objects and CAD models can interlace, we propose a stacked hourglass approach to separate foreground and background from a scan object, and transform it to a complete, CAD-like representation to produce a shared embedding space. This embedding space can then be used for CAD model retrieval; to further enable this task, we introduce a new dataset of ranked scan-CAD similarity annotations, enabling new, fine-grained evaluation of CAD model retrieval to cluttered, noisy, partial scans. Our learned joint embedding outperforms current state of the art for CAD model retrieval by 12% in instance retrieval accuracy.
△ Less
Submitted 19 August, 2019;
originally announced August 2019.
-
Looking beyond the horizon: Evaluation of four compact visualization techniques for time series in a spatial context
Authors:
Manuel Dahnert,
Alexander Rind,
Wolfgang Aigner,
Johannes Kehrer
Abstract:
Visualizing time series in a dense spatial context such as a geographical map is a challenging task, which requires careful balance between the amount of depicted data and perceptual precision. Horizon graphs are a well-known technique for compactly representing time series data. They provide fine details while simultaneously giving an overview of the data where extrema are emphasized. Horizon gra…
▽ More
Visualizing time series in a dense spatial context such as a geographical map is a challenging task, which requires careful balance between the amount of depicted data and perceptual precision. Horizon graphs are a well-known technique for compactly representing time series data. They provide fine details while simultaneously giving an overview of the data where extrema are emphasized. Horizon graphs compress the vertical resolution of the individual line graphs, but they do not affect the horizontal resolution. We present two variations of a new visualization technique called collapsed horizon graphs which extend the idea of horizon graphs to two dimensions. Our main contribution is a quantitative evaluation that experimentally compares four visualization techniques with high visual information resolution (compact boxplots, horizon graphs, collapsed horizon graphs, and braided collapsed horizon graphs). The experiment investigates the performance of these techniques across tasks addressing both individual graphs as well as groups of adjacent graphs. Compact boxplots consistently provide good results for all tasks, horizon graphs excel, for instance, in maximum tasks but underperform in trend detection. Collapsed horizon graphs shine in certain tasks in which an increased horizontal resolution is beneficial. Moreover, our results indicate that the visual complexity of the techniques highly affects users' confidence and perceived task difficulty.
△ Less
Submitted 18 June, 2019;
originally announced June 2019.
-
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
Authors:
Armen Avetisyan,
Manuel Dahnert,
Angela Dai,
Manolis Savva,
Angel X. Chang,
Matthias Nießner
Abstract:
We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry. To tackle this problem, we create a new scan-to-CAD a…
▽ More
We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry. To tackle this problem, we create a new scan-to-CAD alignment dataset based on 1506 ScanNet scans with 97607 annotated keypoint pairs between 14225 CAD models from ShapeNet and their counterpart objects in the scans. Our method selects a set of representative keypoints in a 3D scan for which we find correspondences to the CAD geometry. To this end, we design a novel 3D CNN architecture that learns a joint embedding between real and synthetic objects, and from this predicts a correspondence heatmap. Based on these correspondence heatmaps, we formulate a variational energy minimization that aligns a given set of CAD models to the reconstruction. We evaluate our approach on our newly introduced Scan2CAD benchmark where we outperform both handcrafted feature descriptor as well as state-of-the-art CNN based methods by 21.39%.
△ Less
Submitted 27 November, 2018;
originally announced November 2018.
