Hyunchul Lim
Ruidong Zhang
Cheng Zhang
ABSTRACT
This paper presents D-Touch, a neck-mounted wearable sensing system that can recognize and predict how a hand touches the face. It uses a neck-mounted infrared camera (IR), which takes pictures of the head from the neck. These IR camera images are processed and used to train a deep-learning model to recognize and predict touch time and positions. The study showed D-Touch distinguished 17 Facial related Activity (FrA), including 11 face touch positions and 6 other activities, with over 92.1% accuracy and predict the hand-touching T-zone from other FrA activities with an accuracy of 82.12% within 150 ms after the hand appeared in the camera. A study with 10 participants conducted in their homes without any constraints on participants showed that D-Touch can predict the hand-touching T-zone from other FrA activities with an accuracy of 72.3% within 150 ms after the camera saw the hand. Based on the study results, we further discuss the opportunities and challenges of deploying D-Touch in real-world scenarios.
- Timo Ahonen, Abdenour Hadid, and Matti Pietikainen. 2006. Face description with local binary patterns: Application to face recognition. IEEE transactions on pattern analysis and machine intelligence 28, 12(2006), 2037–2041.Google Scholar
Digital Library
- Jonathan Aigrain, Michel Spodenkiewicz, Séverine Dubuiss, Marcin Detyniecki, David Cohen, and Mohamed Chetouani. 2016. Multimodal stress detection from multiple assessments. IEEE Transactions on Affective Computing 9, 4 (2016), 491–506.Google ScholarDigital Library
- Antonis A Argyros and Manolis IA Lourakis. 2004. Real-time tracking of multiple skin-colored objects with a possibly moving camera. In European Conference on Computer Vision. Springer, 368–379.Google ScholarCross Ref
- G Bally, J Müller, M Rohs, D Wigdor, and S Kratz. 2012. ShoeSense: a new perspective on hand gestures and wearable applications. In Proc. CHI, Vol. 12.Google Scholar
- Cigdem Beyan, Matteo Bustreo, Muhammad Shahid, Gian Luca Bailo, Nicolo Carissimi, and Alessio Del Bue. 2020. Analysis of Face-Touching Behavior in Large Scale Social Interaction Dataset. In Proceedings of the 2020 International Conference on Multimodal Interaction (Virtual Event, Netherlands) (ICMI ’20). Association for Computing Machinery, New York, NY, USA, 24–32. https://doi.org/10.1145/3382507.3418876Google ScholarDigital Library
- Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. IEEE transactions on pattern analysis and machine intelligence 43, 1(2019), 172–186.Google ScholarDigital Library
- Liwei Chan, Chi-Hao Hsieh, Yi-Ling Chen, Shuo Yang, Da-Yuan Huang, Rong-Hao Liang, and Bing-Yu Chen. 2015. Cyclops: Wearable and single-piece full-body gesture input devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3001–3009.Google Scholar
- Dong Chen, Xudong Cao, Liwei Wang, Fang Wen, and Jian Sun. 2012. Bayesian face revisited: A joint formulation. In European conference on computer vision. Springer, 566–579.Google ScholarDigital Library
- Tuochao Chen, Yaxuan Li, Songyun Tao, Hyunchul Lim, Mose Sakashita, Ruidong Zhang, Francois Guimbretiere, and Cheng Zhang. 2021. NeckFace: Continuously Tracking Full Facial Expressions on Neck-mounted Wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1–31.Google ScholarDigital Library
- Tuochao Chen, Benjamin Steeper, Kinan Alsheikh, Songyun Tao, François Guimbretière, and Cheng Zhang. 2020. C-Face: Continuously reconstructing facial expressions by deep learning contours of the face with ear-mounted miniature cameras. In Proceedings of the 33rd annual ACM symposium on user interface software and technology. 112–125.Google ScholarDigital Library
- Xiang’Anthony’ Chen. 2020. FaceOff: Detecting face touching with a wrist-worn accelerometer. arXiv preprint arXiv:2008.01769(2020).Google Scholar
- Jun Cheng, Can Xie, Wei Bian, and Dacheng Tao. 2012. Feature fusion for 3D hand gesture recognition by learning a shared hidden space. Pattern Recognition Letters 33, 4 (2012), 476–484.Google ScholarDigital Library
- Sungman Cho, Minjee Kim, Joonmyeong Choi, Taehyung Kim, Juyoung Park, and Namkug Kim. 2020. Implementation of Face-Touching Action Recognition System based on Deep Learning for Preventing Contagious Diseases. In Proceedings of the Korean Society of Broadcast Engineers Conference. The Korean Institute of Broadcast and Media Engineers, 630–633.Google Scholar
- Timothy F Cootes and Christopher J Taylor. 1992. Active shape models—‘smart snakes’. In BMVC92. Springer, 266–275.Google Scholar
- Timothy F Cootes, Christopher J Taylor, David H Cooper, and Jim Graham. 1995. Active shape models-their training and application. Computer vision and image understanding 61, 1 (1995), 38–59.Google Scholar
- Martin Côté, Pierre Payeur, and Gilles Comeau. 2006. Comparative study of adaptive segmentation techniques for gesture analysis in unconstrained environments. In Proceedings of the 2006 IEEE International Workshop on Imagining Systems and Techniques (IST 2006). IEEE, 28–33.Google ScholarCross Ref
- James Crowley, François Berard, Joelle Coutaz, 1995. Finger tracking as an input device for augmented reality. In International Workshop on Gesture and Face Recognition. 195–200.Google Scholar
- Trevor J Darrell, Irfan A Essa, and Alex P Pentland. 1996. Task-specific gesture analysis in real-time using interpolated views. IEEE Transactions on Pattern Analysis and Machine Intelligence 18, 12(1996), 1236–1242.Google ScholarDigital Library
- Martin de La Gorce, David J Fleet, and Nikos Paragios. 2011. Model-based 3d hand pose estimation from monocular video. IEEE transactions on pattern analysis and machine intelligence 33, 9(2011), 1793–1805.Google ScholarDigital Library
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.Google Scholar
- Nicole D’Aurizio, Tommaso Lisini Baldi, Gianluca Paolocci, and Domenico Prattichizzo. 2020. Preventing Undesired Face-Touches With Wearable Devices and Haptic Feedback. IEEE Access 8(2020), 139033–139043.Google ScholarCross Ref
- Martin Grunwald, Thomas Weiss, Stephanie Mueller, and Lysann Rall. 2014. EEG changes caused by spontaneous facial self-touch may represent emotion regulating processes and working memory maintenance. brain research 1557(2014), 111–126.Google Scholar
- Xiaofei He, Shuicheng Yan, Yuxiao Hu, Partha Niyogi, and Hong-Jiang Zhang. 2005. Face recognition using laplacianfaces. IEEE transactions on pattern analysis and machine intelligence 27, 3(2005), 328–340.Google Scholar
- Ryosuke Hori, Ryo Hachiuma, Hideo Saito, Mariko Isogawa, and Dan Mikami. 2021. Silhouette-Based Synthetic Data Generation For 3D Human Pose Estimation With A Single Wrist-Mounted 360° Camera. In 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 1304–1308.Google ScholarCross Ref
- Dong-Hyun Hwang, Kohei Aso, Ye Yuan, Kris Kitani, and Hideki Koike. 2020. Monoeye: Multimodal human motion capture system using a single ultra-wide fisheye camera. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 98–111.Google ScholarDigital Library
- Vimal Kakaraparthi, Qijia Shao, Charles J Carver, Tien Pham, Nam Bui, Phuc Nguyen, Xia Zhou, and Tam Vu. 2021. FaceSense: sensing face touch with an ear-worn system. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1–27.Google ScholarDigital Library
- Takeo Kanade. 1973. Picture processing by computer complex and recognition of human faces. Ph. D. Thesis, Kyoto University(1973).Google Scholar
- Michael David Kelly. 1970. Visual identification of people by computer. Number 130. Department of Computer Science, Stanford University.Google Scholar
- Yen Lee Angela Kwok, Jan Gralton, and Mary-Louise McLaws. 2015. Face touching: a frequent habit that has implications for hand hygiene. American journal of infection control 43, 2 (2015), 112–114.Google Scholar
- Ivan Laptev and Tony Lindeberg. 2001. Tracking of Multi-state Hand Models Using Particle Filtering and a Hierarchy of Multi-scale Image Features. In International Conference on Scale-Space Theories in Computer Vision. Springer, 63–74.Google ScholarCross Ref
- Hyunchul Lim, Yaxuan Li, Matthew Dressa, Fang Hu, Jae Hoon Kim, Ruidong Zhang, and Cheng Zhang. 2022. BodyTrak: Inferring Full-body Poses from Body Silhouettes Using a Miniature Camera on a Wristband. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1–21.Google ScholarDigital Library
- H. Lin, M. Hsu, and W. Chen. 2014. Human hand gesture recognition using a convolution neural network. In 2014 IEEE International Conference on Automation Science and Engineering (CASE). 1038–1043.Google Scholar
- Shang-Hung Lin, Sun-Yuan Kung, and Long-Ji Lin. 1997. Face recognition/detection by probabilistic decision-based neural network. IEEE transactions on neural networks 8, 1 (1997), 114–132.Google Scholar
- Mona Hosseinkhani Loorak, Wei Zhou, Ha Trinh, Jian Zhao, and Wei Li. 2019. Hand-Over-Face Input Sensing for Interaction with Smartphones through the Built-in Camera. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19). Association for Computing Machinery, New York, NY, USA, Article 32, 12 pages. https://doi.org/10.1145/3338286.3340143Google ScholarDigital Library
- David G Lowe. 1999. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, Vol. 2. Ieee, 1150–1157.Google ScholarCross Ref
- Tiffany L Lucas, Rachel Mustain, and Robert E Goldsby. 2020. Frequency of face touching with and without a mask in pediatric hematology/oncology health care professionals. Pediatric Blood & Cancer 67, 9 (2020), e28593.Google ScholarCross Ref
- Marwa Mahmoud, Tadas Baltrušaitis, and Peter Robinson. 2016. Automatic analysis of naturalistic hand-over-face gestures. ACM Transactions on Interactive Intelligent Systems (TiiS) 6, 2(2016), 1–18.Google ScholarDigital Library
- Marwa Mahmoud and Peter Robinson. 2011. Interpreting hand-over-face gestures. In International Conference on Affective Computing and Intelligent Interaction. Springer, 248–255.Google ScholarCross Ref
- Marwa M Mahmoud, Tadas Baltrušaitis, and Peter Robinson. 2014. Automatic detection of naturalistic hand-over-face gesture descriptors. In Proceedings of the 16th International Conference on Multimodal Interaction. 319–326.Google ScholarDigital Library
- Katsutoshi Masai, Yuta Sugiura, and Maki Sugimoto. 2018. Facerubbing: Input technique by rubbing face using optical sensors on smart eyewear for facial expression recognition. In Proceedings of the 9th Augmented Human International Conference. 1–5.Google ScholarDigital Library
- GRS Murthy and RS Jadon. 2009. A review of vision based hand gestures recognition. International Journal of Information Technology and Knowledge Management 2, 2 (2009), 405–410.Google Scholar
- Mark Nicas and Daniel Best. 2008. A study quantifying the hand-to-face contact rate and its potential application to predicting respiratory tract infection. Journal of occupational and environmental hygiene 5, 6 (2008), 347–352.Google ScholarCross Ref
- Aditya Shekhar Nittala, Anusha Withana, Narjes Pourjafarian, and Jürgen Steimle. 2018. Multi-touch skin: A thin and flexible multi-touch sensor for on-skin input. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Behnaz Nojavanasghari, Charles E Hughes, Tadas Baltrušaitis, and Louis-Philippe Morency. 2017. Hand2face: Automatic synthesis and recognition of hand over face occlusions. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 209–215.Google ScholarCross Ref
- Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep face recognition. (2015).Google Scholar
- Alex Pentland, Baback Moghaddam, Thad Starner, 1994. View-based and modular eigenspaces for face recognition. (1994).Google Scholar
- P Phillips. 1998. Support vector machines applied to face recognition. Advances in Neural Information Processing Systems 11 (1998), 803–809.Google Scholar
- Laura R Pina, Ernesto Ramirez, and William G Griswold. 2012. Fitbit+: A behavior-based intervention system to reduce sedentary behavior. In 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops. IEEE, 175–178.Google ScholarCross Ref
- Juma Rahman, Jubayer Mumin, and Bapon Fakhruddin. 2020. How Frequently Do We Touch Facial T-Zone: A Systematic Review. Annals of Global Health 86, 1 (2020).Google Scholar
- Siddharth S Rautaray and Anupam Agrawal. 2012. Real time hand gesture recognition system for dynamic applications. International Journal of UbiComp 3, 1 (2012), 21.Google ScholarCross Ref
- Siddharth S Rautaray and Anupam Agrawal. 2015. Vision based hand gesture recognition for human computer interaction: a survey. Artificial intelligence review 43, 1 (2015), 1–54.Google ScholarDigital Library
- Michael J Reale, Shaun Canavan, Lijun Yin, Kaoning Hu, and Terry Hung. 2011. A multi-gesture interaction system using a 3-D iris disk model for gaze estimation and an active appearance model for 3-D hand pointing. IEEE Transactions on multimedia 13, 3 (2011), 474–486.Google ScholarDigital Library
- Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497(2015).Google Scholar
- Helge Rhodin, Christian Richardt, Dan Casas, Eldar Insafutdinov, Mohammad Shafiei, Hans-Peter Seidel, Bernt Schiele, and Christian Theobalt. 2016. Egocap: egocentric marker-less motion capture with two fisheye cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–11.Google ScholarDigital Library
- Nicola D Ridgers, Melitta A McNarry, and Kelly A Mackintosh. 2016. Feasibility and effectiveness of using wearable activity trackers in youth: a systematic review. JMIR mHealth and uHealth 4, 4 (2016), e129.Google Scholar
- Hamada Rizk, Tatsuya Amano, Hirozumi Yamaguchi, and Moustafa Youssef. 2022. Smartwatch-based face-touch prediction using deep representational learning. In Mobile and Ubiquitous Systems: Computing, Networking and Services: 18th EAI International Conference, MobiQuitous 2021, Virtual Event, November 8-11, 2021, Proceedings. Springer, 493–499.Google Scholar
- Enver Sangineto and Marco Cupelli. 2012. Real-time viewpoint-invariant hand localization with cluttered backgrounds. Image and Vision Computing 30, 1 (2012), 26–37.Google ScholarDigital Library
- Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro. 2017. Learning bases of activity for facial expression recognition. IEEE Transactions on Image Processing 26, 4 (2017), 1965–1978.Google ScholarDigital Library
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014).Google Scholar
- Lawrence Sirovich and Michael Kirby. 1987. Low-dimensional procedure for the characterization of human faces. Josa a 4, 3 (1987), 519–524.Google Scholar
- Yi Sun, Xiaogang Wang, and Xiaoou Tang. 2014. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1891–1898.Google ScholarDigital Library
- Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1701–1708.Google ScholarDigital Library
- Orlando Motohiro Tanaka, Robert Willer Farinazzo Vitral, Giulia Yuriko Tanaka, Ariana Pulido Guerrero, and Elisa Souza Camargo. 2008. Nailbiting, or onychophagia: a special habit. American Journal of Orthodontics and Dentofacial Orthopedics 134, 2(2008), 305–308.Google ScholarCross Ref
- Cuong Tran and Mohan Manubhai Trivedi. 2011. 3-D posture and gesture recognition for interactivity in smart spaces. IEEE Transactions on Industrial Informatics 8, 1 (2011), 178–187.Google ScholarCross Ref
- Daniel Sáez Trigueros, Li Meng, and Margaret Hartnett. 2018. Face recognition: From traditional to deep learning methods. arXiv preprint arXiv:1811.00116(2018).Google Scholar
- Hans Van Kuilenburg, Marco Wiering, and Marten Den Uyl. 2005. A model based method for automatic facial expression recognition. In European conference on machine learning. Springer, 194–205.Google ScholarDigital Library
- Laurenz Wiskott, Norbert Krüger, N Kuiger, and Christoph Von Der Malsburg. 1997. Face recognition by elastic bunch graph matching. IEEE Transactions on pattern analysis and machine intelligence 19, 7(1997), 775–779.Google ScholarDigital Library
- John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, and Yi Ma. 2008. Robust face recognition via sparse representation. IEEE transactions on pattern analysis and machine intelligence 31, 2(2008), 210–227.Google ScholarDigital Library
- Xuhai Xu, Haitian Shi, Xin Yi, Wenjia Liu, Yukang Yan, Yuanchun Shi, Alex Mariakakis, Jennifer Mankoff, and Anind K Dey. 2020. Earbuddy: Enabling on-face interaction via wireless earbuds. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Liu Yun and Zhang Peng. 2009. An automatic hand gesture recognition system based on Viola-Jones method and SVMs. In 2009 Second International Workshop on Computer Science and Engineering, Vol. 2. IEEE, 72–76.Google ScholarDigital Library
- Junbo Zhang and Swarun Kumar. 2020. NoFaceContact: stop touching your face with NFC. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services. 468–469.Google ScholarDigital Library
- Ligang Zhang and Dian Tjondronegoro. 2011. Facial expression recognition using facial movement features. IEEE transactions on affective computing 2, 4 (2011), 219–229.Google ScholarDigital Library
- Ruidong Zhang, Mingyang Chen, Benjamin Steeper, Yaxuan Li, Zihan Yan, Yizhuo Chen, Songyun Tao, Tuochao Chen, Hyunchul Lim, and Cheng Zhang. 2021. SpeeChin: A Smart Necklace for Silent Speech Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1–23.Google ScholarDigital Library
- Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z Li. 2018. Single-shot refinement neural network for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4203–4212.Google ScholarCross Ref
- Shibo Zhang, Yuqi Zhao, Dzung Tri Nguyen, Runsheng Xu, Sougata Sen, Josiah Hester, and Nabil Alshurafa. 2020. NeckSense: A Multi-Sensor Necklace for Detecting Eating Activities in Free-Living Conditions. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 72 (June 2020), 26 pages. https://doi.org/10.1145/3397313Google ScholarDigital Library
Index Terms
- D-Touch: Recognizing and Predicting Fine-grained Hand-face Touching Activities Using a Neck-mounted Wearable
Recommendations
Space-time flexible kernel for recognizing activities from wearable cameras
AbstractRecognizing activities of daily living is useful for ambient assisted living. In this regard, the use of wearable cameras is a promising technology. In this paper, we propose a novel approach for recognizing activities of daily living using ...
Elderly Assistance Using Wearable Sensors by Detecting Fall and Recognizing Fall Patterns
UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable ComputersFalling is a serious threat to the elderly people. One severe fall can cause hazardous problems like bone fracture or may lead to some permanent disability or even death. Thus, it has become the need of the hour to continuously monitor the activities of ...
Multimodal Wearable Sensing for Fine-Grained Activity Recognition in Healthcare
State-of-the-art in-home activity recognition schemes with wearable devices are mostly capable of detecting coarse-grained activities (sitting, standing, walking, or lying down), but can't distinguish complex activities (sitting on the floor versus the ...
Comments