Home
Publications
Photo Gallery
H2ONet: Hand-Occlusion-and-Orientation-aware Network for Real-time 3D Hand Mesh Reconstruction
Hao Xu, Tianyu Wang, Xiao Tang, Chi-Wing Fu
CVPR · 2023
Research summary
A network for real-time 3D hand mesh reconstruction that handles hand occlusion and orientation challenges.
Abstract
Real-time 3D hand mesh reconstruction is challenging, especially when the hand is holding some object. Beyond the previous methods, we design H2ONet to fully exploit non-occluded information from multiple frames to boost the reconstruction quality. First, we decouple hand mesh reconstruction into two branches, one to exploit finger-level non-occluded information and the other to exploit global hand orientation, with lightweight structures to promote real-time inference. Second, we propose finger-level occlusion-aware feature fusion, leveraging predicted finger-level occlusion information as guidance to fuse finger-level information across time frames. Further, we design hand-level occlusion-aware feature fusion to fetch non-occluded information from nearby time frames. We conduct experiments on the Dex-YCB and HO3D-v2 datasets with challenging hand-object occlusion cases, manifesting that H2ONet is able to run in real-time and achieves state-of-the-art performance on both the hand mesh and pose precision. The code will be released on GitHub.
Key insight
Addresses the overlooked challenges of hand occlusion and orientation in 3D hand mesh reconstruction, achieving real-time performance while significantly improving accuracy on occluded and rotated hand poses.
Paper & resources
BibTeX
@inproceedings{xu2023h2onet,
title={H2ONet: Hand-Occlusion-and-Orientation-aware Network for Real-time 3D Hand Mesh Reconstruction},
author={Xu, Hao and Wang, Tianyu and Tang, Xiao and Fu, Chi-Wing},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2023}
}