Abstract
One of the challenging problems in computer vision is hand segmentation, especially when the hands overlap with the face. There are many applications that require this type of segmentation, such as sign language recognition, action recognition and recognition of objects that hands interact with. Hand over face is a challenging scenario where faces are occluded by hands, that can be used to test the performance of hand segmentation methods. Not much work has been done on this topic. After analysis of related datasets for hand segmentation that include hands in front of or near to the face, we introduce our challenging public dataset for the hand-over-face segmentation problem. The new dataset contains 4384 annotated frames and includes color, depth, infrared streams recorded by Kinect. Additionally, hand(s) locations and shapes data using Leap Motion sensor, which is an infrared hand shape sensor, are included. We compare two leading semantic segmentation methods: SegNet [1] and RefineNet [12], to analyze the new dataset. Two experiments were executed: the first one for hand-background segmentation and the other one for right hand- left hand- background segmentation. RefineNet shows significantly better accuracy, 14% better than that of SegNet, on our new dataset. Nonetheless, the highest accuracy archived was 62.2%, demonstrating that VLM-HandOverFace(1) is a challenging dataset for the current state of the art.