Lightweight Hybrid Feature-Based 3D Scene Reconstruction for Resource-Constrained AR Applications

Abstract

Augmented Reality (AR) applications cannot be used without effective and robust 3D reconstructions of the scenes to appropriately position virtual objects into the real world. However, the reconstruction of high-fidelity 3D using mobile and resource-constrained hardware remains a significant challenge due to memory, processing, sensor and battery life constraints. Conventional methodologies that are geometric, such as Structure from Motion (SfM), Multi-View Stereo (MVS), and feature-based Simultaneous Localization and Mapping (SLAM) have also proven useful in tracking a camera and sparse-to-dense mapping. However, these techniques tend to perform poorly in low-texture scenes, moving scenes, and complicated lighting situations. This paper will solve these shortcomings by presenting a lightweight hybrid 3D reconstruction system that combines traditional SLAM approaches with a small neural augmentation system. Within this framework, SLAM will be used to precisely estimate poses and map geometries and the neural component will be used to refine critical regions to improve the quality of the texture and fill smaller gaps in reconstructions without causing a lot of computational load. The system also takes advantage of the performance of the embedded systems and client-grade GPUs, as well as, uses the GPUs to attain nearly real-time performance through the use of GPU-based optimization, lightweight data structure, and adaptive processing scheme. The conducted experiments suggest that the suggested hybrid scheme notably enhances the accuracy of reconstructions and visual quality without compromising on the performance specifications that are vital in resource-constrained settings.