×

M-NeuS: volume rendering based surface reconstruction and material estimation. (English) Zbl 07873110

Summary: Although significant advances have been made in the field of multi-view 3D reconstruction using implicit neural field-based methods, existing reconstruction methods overlook the estimation of the material information (e.g. the base color, albedo, roughness, and metallic) during the learning process. In this paper, we propose a novel differentiable rendering framework, named as material NueS (M-NeuS), for simultaneously achieving precise surface reconstruction and competitive material estimation. For surface reconstruction, we perform multi-view geometry optimization by proposing an enhanced-low-to-high frequency encoding registration strategy (EFERS) and a second-order interpolated signed distance function (SI-SDF) for precise details and outline reconstruction. For material estimation, inspired by the NeuS, we first propose a volume-rendering-based material estimation strategy (VMES) to estimate the base color, albedo, roughness, and metallic accurately. And then, different from most material estimation methods that need ground-truth geometric priors, we use the geometry information reconstructed in the surface reconstruction stage and the directions of incidence from different viewpoints to model a neural light field, which can extract the lighting information from image observations. Next, the extracted lighting and the estimated base color, albedo, roughness, and metallic are optimized by the physics-based rendering equation. Extensive experiments demonstrate that our M-NeuS can not only reconstruct more precise geometry surface than existing state-of-the-art (SOTA) reconstruction methods but also can estimate competitive material information: the base color, albedo, roughness, and metallic.

MSC:

65Dxx Numerical approximation and computational geometry (primarily algorithms)
Full Text: DOI

References:

[1] Aanæs, H.; Jensen, R. R.; Vogiatzis, G.; Tola, E.; Dahl, A. B., Large-scale data for multiple-view stereopsis, Int. J. Comput. Vis., 120, 153-168, 2016
[2] Azinovic, D.; Li, T. M.; Kaplanyan, A.; Nießner, M., Inverse path tracing for joint material and lighting estimation, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019), 2447-2456
[3] Barnes, C.; Shechtman, E.; Finkelstein, A.; Goldman, D. B., Patchmatch: a randomized correspondence algorithm for structural image editing, ACM Trans. Graph., 28, 24, 2009
[4] Barron, J. T.; Mildenhall, B.; Tancik, M.; Hedman, P.; Martin-Brualla, R.; Srinivasan, P. P., Mip-nerf: a multiscale representation for anti-aliasing neural radiance fields, (Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021), 5855-5864
[5] Bernardini, F.; Mittleman, J.; Rushmeier, H.; Silva, C.; Taubin, G., The ball-pivoting algorithm for surface reconstruction, IEEE Trans. Vis. Comput. Graph., 5, 349-359, 1999
[6] Boss, M.; Braun, R.; Jampani, V.; Barron, J. T.; Liu, C.; Lensch, H., Nerd: neural reflectance decomposition from image collections, (Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021), 12684-12694
[7] Boss, M.; Groh, F.; Herholz, S.; Lensch, H. P., Deep dual loss brdf parameter estimation, (MAM@ EGSR, 2018), 41-44
[8] Boss, M.; Jampani, V.; Braun, R.; Liu, C.; Barron, J.; Lensch, H., Neural-pil: neural pre-integrated lighting for reflectance decomposition, Adv. Neural Inf. Process. Syst., 34, 10691-10704, 2021
[9] Broadhurst, A.; Drummond, T. W.; Cipolla, R., A probabilistic framework for space carving, (Proceedings Eighth IEEE International Conference on Computer Vision. Proceedings Eighth IEEE International Conference on Computer Vision, ICCV 2001, 2001, IEEE), 388-393
[10] Burley, B., Physically based shading at disney, 2012, SIGGRAPH
[11] Darmon, F.; Bascle, B.; Devaux, J. C.; Monasse, P.; Aubry, M., Improving neural implicit surfaces geometry with patch warping, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022), 6260-6269
[12] De Bonet, J. S.; Viola, P., Poxels: probabilistic voxelized volume reconstruction, (Proceedings of International Conference on Computer Vision (ICCV), 1999), 3
[13] Dong, Y.; Chen, G.; Peers, P.; Zhang, J.; Tong, X., Appearance-from-motion: recovering spatially varying surface reflectance under unknown lighting, ACM Trans. Graph., 33, 1-12, 2014 · Zbl 1396.68130
[14] Fan, Y.; Skorokhodov, I.; Voynov, O.; Ignatyev, S.; Burnaev, E.; Wonka, P.; Wang, Y., Factored-neus: reconstructing surfaces, illumination, and materials of possibly glossy objects, 2023, arXiv preprint
[15] Fu, Q.; Xu, Q.; Ong, Y. S.; Tao, W., Geo-neus: geometry-consistent neural implicit surfaces learning for multi-view reconstruction, Adv. Neural Inf. Process. Syst., 35, 3403-3416, 2022
[16] Galliani, S.; Lasinger, K.; Schindler, K., Gipuma: Massively Parallel Multi-View Stereo Reconstruction, Publikationen der Deutschen Gesellschaft für Photogrammetrie, Fernerkundung und Geoinformation e. V., vol. 25, 2, 2016
[17] Ge, W.; Hu, T.; Zhao, H.; Liu, S.; Chen, Y. C., Ref-neus: ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection, 2023, arXiv preprint
[18] Gropp, A.; Yariv, L.; Haim, N.; Atzmon, M.; Lipman, Y., Implicit geometric regularization for learning shapes, 2020, arXiv preprint
[19] Guo, K.; Lincoln, P.; Davidson, P.; Busch, J.; Yu, X.; Whalen, M.; Harvey, G.; Orts-Escolano, S.; Pandey, R.; Dourgarian, J., The relightables: volumetric performance capture of humans with realistic relighting, ACM Trans. Graph., 38, 1-19, 2019
[20] Han, J.; Moraga, C., The influence of the sigmoid function parameters on the speed of backpropagation learning, (From Natural to Artificial Neural Computation: International Workshop on Artificial Neural Networks Malaga-Torremolinos, Proceedings 3. From Natural to Artificial Neural Computation: International Workshop on Artificial Neural Networks Malaga-Torremolinos, Proceedings 3, Spain, June 7-9, 1995, 1995, Springer), 195-201
[21] Kajiya, J. T., The rendering equation, (Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, 1986), 143-150
[22] Kaza, S., Differentiable volume rendering using signed distance functions, 2019, Massachusetts Institute of Technology, Ph.D. thesis
[23] Kazhdan, M.; Hoppe, H., Screened Poisson surface reconstruction, ACM Trans. Graph., 32, 1-13, 2013 · Zbl 1322.68228
[24] Kellnhofer, P.; Jebe, L. C.; Jones, A.; Spicer, R.; Pulli, K.; Wetzstein, G., Neural lumigraph rendering, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021), 4287-4297
[25] Kingma, D.; Ba, J., Adam: a method for stochastic optimization, 2014, arXiv preprint
[26] Lawrence, J.; Rusinkiewicz, S.; Ramamoorthi, R., Efficient brdf importance sampling using a factored representation, ACM Trans. Graph., 23, 496-505, 2004
[27] Lensch, H. P.; Kautz, J.; Goesele, M.; Heidrich, W.; Seidel, H. P., Image-based reconstruction of spatially varying materials, (Rendering Techniques 2001: Proceedings of the Eurographics Workshop in London. Rendering Techniques 2001: Proceedings of the Eurographics Workshop in London, United Kingdom, June 25-27, 2001, vol. 12, 2001, Springer), 103-114
[28] Li, X.; Dong, Y.; Peers, P.; Tong, X., Modeling surface appearance from a single photograph using self-augmented convolutional neural networks, ACM Trans. Graph., 36, 1-11, 2017
[29] Li, Z.; Xu, Z.; Ramamoorthi, R.; Sunkavalli, K.; Chandraker, M., Learning to reconstruct shape and spatially-varying reflectance from a single image, ACM Trans. Graph., 37, 1-11, 2018
[30] Lin, C. H.; Ma, W. C.; Torralba, A.; Lucey, S., Barf: bundle-adjusting neural radiance fields, (Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021), 5741-5751
[31] Liu, S.; Zhang, Y.; Peng, S.; Shi, B.; Pollefeys, M.; Cui, Z., Dist: rendering deep implicit signed distance function with differentiable sphere tracing, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020), 2019-2028
[32] Lombardi, S.; Simon, T.; Saragih, J.; Schwartz, G.; Lehrmann, A.; Sheikh, Y., Neural volumes: learning dynamic renderable volumes from images, 2019, arXiv preprint
[33] Merrell, P.; Akbarzadeh, A.; Wang, L.; Mordohai, P.; Frahm, J. M.; Yang, R.; Nistér, D.; Pollefeys, M., Real-time visibility-based fusion of depth maps, (2007 IEEE 11th International Conference on Computer Vision, 2007, Ieee), 1-8
[34] Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; Ng, R., Nerf: representing scenes as neural radiance fields for view synthesis, Commun. ACM, 65, 99-106, 2021
[35] Munkberg, J.; Hasselgren, J.; Shen, T.; Gao, J.; Chen, W.; Evans, A.; Müller, T.; Fidler, S., Extracting triangular 3d models, materials, and lighting from images, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022), 8280-8290
[36] Oechsle, M.; Peng, S.; Geiger, A., Unisurf: unifying neural implicit surfaces and radiance fields for multi-view reconstruction, (Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021), 5589-5599
[37] Park, J. J.; Holynski, A.; Seitz, S. M., Seeing the world in a bag of chips, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020), 1417-1427
[38] Park, K.; Sinha, U.; Barron, J. T.; Bouaziz, S.; Goldman, D. B.; Seitz, S. M.; Martin-Brualla, R., Nerfies: deformable neural radiance fields, (Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021), 5865-5874
[39] Schonberger, J. L.; Frahm, J. M., Structure-from-motion revisited, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016), 4104-4113
[40] Seitz, S. M.; Dyer, C. R., Photorealistic scene reconstruction by voxel coloring, Int. J. Comput. Vis., 35, 151-173, 1999
[41] Sitzmann, V.; Zollhöfer, M.; Wetzstein, G., Scene representation networks: continuous 3d-structure-aware neural scene representations, Adv. Neural Inf. Process. Syst., 32, 2019
[42] Snavely, N.; Seitz, S. M.; Szeliski, R., Photo tourism: exploring photo collections in 3d, (ACM Siggraph 2006 Papers, 2006), 835-846
[43] Srinivasan, P. P.; Deng, B.; Zhang, X.; Tancik, M.; Mildenhall, B.; Barron, J. T., Nerv: neural reflectance and visibility fields for relighting and view synthesis, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021), 7495-7504
[44] Tancik, M.; Srinivasan, P.; Mildenhall, B.; Fridovich-Keil, S.; Raghavan, N.; Singhal, U.; Ramamoorthi, R.; Barron, J.; Ng, R., Fourier features let networks learn high frequency functions in low dimensional domains, Adv. Neural Inf. Process. Syst., 33, 7537-7547, 2020
[45] Verbin, D.; Hedman, P.; Mildenhall, B.; Zickler, T.; Barron, J. T.; Srinivasan, P. P., Ref-nerf: structured view-dependent appearance for neural radiance fields, (2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, IEEE), 5481-5490
[46] Walter, B.; Marschner, S. R.; Li, H.; Torrance, K. E., Microfacet models for refraction through rough surfaces, (Proceedings of the 18th Eurographics Conference on Rendering Techniques, 2007), 195-206
[47] Wang, P.; Liu, L.; Liu, Y.; Theobalt, C.; Komura, T.; Wang, W., Neus: learning neural implicit surfaces by volume rendering for multi-view reconstruction, 2021, arXiv preprint
[48] Wang, Y.; Skorokhodov, I.; Wonka, P., Hf-neus: improved surface reconstruction using high-frequency details, Adv. Neural Inf. Process. Syst., 35, 1966-1978, 2022
[49] Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E. P., Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process., 13, 600-612, 2004
[50] Wu, T.; Sun, J. M.; Lai, Y. K.; Gao, L., De-nerf: decoupled neural radiance fields for view-consistent appearance editing and high-frequency environmental relighting, (ACM SIGGRAPH 2023 Conference Proceedings, 2023), 1-11
[51] Xia, R.; Dong, Y.; Peers, P.; Tong, X., Recovering shape and spatially-varying surface reflectance under unknown illumination, ACM Trans. Graph., 35, 1-12, 2016
[52] Yao, Y.; Luo, Z.; Li, S.; Zhang, J.; Ren, Y.; Zhou, L.; Fang, T.; Quan, L., Blendedmvs: a large-scale dataset for generalized multi-view stereo networks, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020), 1790-1799
[53] Yao, Y.; Zhang, J.; Liu, J.; Qu, Y.; Fang, T.; McKinnon, D.; Tsin, Y.; Quan, L., Neilf: neural incident light field for physically-based material estimation, (Computer Vision-ECCV 2022: 17th European Conference, Proceedings, Part XXXI. Computer Vision-ECCV 2022: 17th European Conference, Proceedings, Part XXXI, Tel Aviv, Israel, October 23-27, 2022, 2022, Springer), 700-716
[54] Yariv, L.; Gu, J.; Kasten, Y.; Lipman, Y., Volume rendering of neural implicit surfaces, Adv. Neural Inf. Process. Syst., 34, 4805-4815, 2021
[55] Yariv, L.; Kasten, Y.; Moran, D.; Galun, M.; Atzmon, M.; Ronen, B.; Lipman, Y., Multiview neural surface reconstruction by disentangling geometry and appearance, Adv. Neural Inf. Process. Syst., 33, 2492-2502, 2020
[56] Yin, X.; Goudriaan, J.; Lantinga, E. A.; Vos, J.; Spiertz, H. J., A flexible sigmoid function of determinate growth, Ann. Bot., 91, 361-371, 2003
[57] Zach, C.; Pock, T.; Bischof, H., A globally optimal algorithm for robust tv-l 1 range image integration, (2007 IEEE 11th International Conference on Computer Vision, 2007, IEEE), 1-8
[58] Zhang, K.; Luan, F.; Wang, Q.; Bala, K.; Snavely, N., Physg: inverse rendering with spherical Gaussians for physics-based material editing and relighting, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021), 5453-5462
[59] Zhang, K.; Riegler, G.; Snavely, N.; Koltun, V., Nerf++: analyzing and improving neural radiance fields, 2020, arXiv preprint
[60] Zhang, X.; Srinivasan, P. P.; Deng, B.; Debevec, P.; Freeman, W. T.; Barron, J. T., Nerfactor: neural factorization of shape and reflectance under an unknown illumination, ACM Trans. Graph., 40, 1-18, 2021
[61] Zhang, Y.; Sun, J.; He, X.; Fu, H.; Jia, R.; Zhou, X., Modeling indirect illumination for inverse rendering, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022), 18643-18652
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.