-
Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8
Authors:
Mikołaj Łysakowski,
Kamil Żywanowski,
Adam Banaszczyk,
Michał R. Nowicki,
Piotr Skrzypczyński,
Sławomir K. Tadeja
Abstract:
This paper introduces a software architecture for real-time object detection using machine learning (ML) in an augmented reality (AR) environment. Our approach uses the recent state-of-the-art YOLOv8 network that runs onboard on the Microsoft HoloLens 2 head-mounted display (HMD). The primary motivation behind this research is to enable the application of advanced ML models for enhanced perception…
▽ More
This paper introduces a software architecture for real-time object detection using machine learning (ML) in an augmented reality (AR) environment. Our approach uses the recent state-of-the-art YOLOv8 network that runs onboard on the Microsoft HoloLens 2 head-mounted display (HMD). The primary motivation behind this research is to enable the application of advanced ML models for enhanced perception and situational awareness with a wearable, hands-free AR platform. We show the image processing pipeline for the YOLOv8 model and the techniques used to make it real-time on the resource-limited edge computing platform of the headset. The experimental results demonstrate that our solution achieves real-time processing without needing offloading tasks to the cloud or any other external servers while retaining satisfactory accuracy regarding the usual mAP metric and measured qualitative performance
△ Less
Submitted 6 June, 2023;
originally announced June 2023.
-
Learning an Efficient Terrain Representation for Haptic Localization of a Legged Robot
Authors:
Damian Sójka,
Michał R. Nowicki,
Piotr Skrzypczyński
Abstract:
Although haptic sensing has recently been used for legged robot localization in extreme environments where a camera or LiDAR might fail, the problem of efficiently representing the haptic signatures in a learned prior map is still open. This paper introduces an approach to terrain representation for haptic localization inspired by recent trends in machine learning. It combines this approach with t…
▽ More
Although haptic sensing has recently been used for legged robot localization in extreme environments where a camera or LiDAR might fail, the problem of efficiently representing the haptic signatures in a learned prior map is still open. This paper introduces an approach to terrain representation for haptic localization inspired by recent trends in machine learning. It combines this approach with the proven Monte Carlo algorithm to obtain an accurate, computation-efficient, and practical method for localizing legged robots under adversarial environmental conditions. We apply the triplet loss concept to learn highly descriptive embeddings in a transformer-based neural network. As the training haptic data are not labeled, the positive and negative examples are discriminated by their geometric locations discovered while training. We demonstrate experimentally that the proposed approach outperforms by a large margin the previous solutions to haptic localization of legged robots concerning the accuracy, inference time, and the amount of data stored in the map. As far as we know, this is the first approach that completely removes the need to use a dense terrain map for accurate haptic localization, thus paving the way to practical applications.
△ Less
Submitted 26 May, 2023; v1 submitted 29 September, 2022;
originally announced September 2022.
-
MinkLoc3D-SI: 3D LiDAR place recognition with sparse convolutions, spherical coordinates, and intensity
Authors:
Kamil Żywanowski,
Adam Banaszczyk,
Michał R. Nowicki,
Jacek Komorowski
Abstract:
The 3D LiDAR place recognition aims to estimate a coarse localization in a previously seen environment based on a single scan from a rotating 3D LiDAR sensor. The existing solutions to this problem include hand-crafted point cloud descriptors (e.g., ScanContext, M2DP, LiDAR IRIS) and deep learning-based solutions (e.g., PointNetVLAD, PCAN, LPDNet, DAGC, MinkLoc3D), which are often only evaluated o…
▽ More
The 3D LiDAR place recognition aims to estimate a coarse localization in a previously seen environment based on a single scan from a rotating 3D LiDAR sensor. The existing solutions to this problem include hand-crafted point cloud descriptors (e.g., ScanContext, M2DP, LiDAR IRIS) and deep learning-based solutions (e.g., PointNetVLAD, PCAN, LPDNet, DAGC, MinkLoc3D), which are often only evaluated on accumulated 2D scans from the Oxford RobotCar dataset. We introduce MinkLoc3D-SI, a sparse convolution-based solution that utilizes spherical coordinates of 3D points and processes the intensity of 3D LiDAR measurements, improving the performance when a single 3D LiDAR scan is used. Our method integrates the improvements typical for hand-crafted descriptors (like ScanContext) with the most efficient 3D sparse convolutions (MinkLoc3D). Our experiments show improved results on single scans from 3D LiDARs (USyd Campus dataset) and great generalization ability (KITTI dataset). Using intensity information on accumulated 2D scans (RobotCar Intensity dataset) improves the performance, even though spherical representation doesn't produce a noticeable improvement. As a result, MinkLoc3D-SI is suited for single scans obtained from a 3D LiDAR, making it applicable in autonomous vehicles.
△ Less
Submitted 27 December, 2021; v1 submitted 13 December, 2021;
originally announced December 2021.
-
Navigating by Touch: Haptic Monte Carlo Localization via Geometric Sensing and Terrain Classification
Authors:
Russell Buchanan,
Jakub Bednarek,
Marco Camurri,
Michał R. Nowicki,
Krzysztof Walas,
Maurice Fallon
Abstract:
Legged robot navigation in extreme environments can hinder the use of cameras and laser scanners due to darkness, air obfuscation or sensor damage. In these conditions, proprioceptive sensing will continue to work reliably. In this paper, we propose a purely proprioceptive localization algorithm which fuses information from both geometry and terrain class, to localize a legged robot within a prior…
▽ More
Legged robot navigation in extreme environments can hinder the use of cameras and laser scanners due to darkness, air obfuscation or sensor damage. In these conditions, proprioceptive sensing will continue to work reliably. In this paper, we propose a purely proprioceptive localization algorithm which fuses information from both geometry and terrain class, to localize a legged robot within a prior map. First, a terrain classifier computes the probability that a foot has stepped on a particular terrain class from sensed foot forces. Then, a Monte Carlo-based estimator fuses this terrain class probability with the geometric information of the foot contact points. Results are demonstrated showing this approach operating online and onboard a ANYmal B300 quadruped robot traversing a series of terrain courses with different geometries and terrain types over more than 1.2km. The method keeps the localization error below 20cm using only the information coming from the feet, IMU, and joints of the quadruped.
△ Less
Submitted 18 August, 2021;
originally announced August 2021.
-
Spatiotemporal Calibration of Camera and 3D Laser Scanner
Authors:
Michal R. Nowicki
Abstract:
The multi-sensory setups consisting of the laser scanners and cameras are popular as the measurements complement each other and provide necessary robustness for applications. Under dynamic conditions or when in motion, a direct transformation (spatial calibration) and time offset between sensors (temporal calibration) is needed to determine the correspondence between measurements. We propose an op…
▽ More
The multi-sensory setups consisting of the laser scanners and cameras are popular as the measurements complement each other and provide necessary robustness for applications. Under dynamic conditions or when in motion, a direct transformation (spatial calibration) and time offset between sensors (temporal calibration) is needed to determine the correspondence between measurements. We propose an open-source spatiotemporal calibration framework for a camera and a 3D laser scanner. Our solution is based on commonly available chessboard markers requiring one-minute calibration before the operation that offers accurate and repeatable results. The framework is based on batch optimization of point-to-plane constraints with a time offset calibration possible by a novel continuous representation of the plane equations based on a minimal representation in the Lie algebra and the use of B-splines. The framework's properties are evaluated in simulation while correctness is verified with two distinct sensory setups with Velodyne VLP-16 and SICK MRS6124 3D laser scanners.
△ Less
Submitted 29 August, 2020; v1 submitted 29 June, 2020;
originally announced June 2020.
