With progress and innovation in technology, wearable electronic tools, such as smartwatches, are growing in popularity and spreading among the new, and old, generations. However, current technology presents some limitations in terms of interaction, opening a path for improvement.

A research team led by Professor Mike Fraser, Mr Asier Marzo and Mr Jess McIntosh from the Bristol Interaction Group (BIG) at the University of Bristol, in cooperation with University Hospitals Bristol NHS Foundation Trust (UH Bristol), investigated the possibility of using ultrasound imaging to sense hand gestures and presented the paper at ACM CHI 2017, one of the world’s most important conferences on human-computer interfaces held in Denver, USA.

Hand gestures are recognised as a useful, yet intuitive manner to interact with smart devices, allowing control over devices’ functionalities. Example can be retrieved in the use of gesture to regulate the brightness of lights in a room, or to open/close doors and windows. There are several techniques that can lead to hand gesture recognition, but the common requirement, that can represent sometimes a strong limitation, is the placement of a sensor capable of detecting the movements.

However, the creation of wearable products such as smartwatches made it possible to overcome the restricting static nature of traditional tools, allowing the insertion of sensor in mobile devices to sense hand movement.

The new research suggests that ultrasonic imaging of the forearm could allow the devices to recognise hand gestures. Techniques involving ultrasonic imaging are already used in medicine, for example in pregnancy scans or monitoring of muscles and tendon movements, and their effectiveness led the researchers to recognise their potential use in the scanning of hand movements.

The team exploited image processing algorithms and machine learning to classify muscle movement as gestures. Further studies involved the research of the ideal place to locate the sensor in order to fully exploit the technique.

The findings of the study indicated a very high recognition accuracy, and importantly this sensing method worked well when implemented at the wrist, identifying an ideal opportunity for future wearable devices (eg. Smartwatches) to include this ultrasonic technique to sense gestures Jess McIntosh, PhD student in the Department of Computer Science and BIG Group, explained: “With current technologies, there are many practical issues that prevent a small, portable ultrasonic imaging sensor integrated into a smartwatch. Nevertheless, our research is a first step towards what could be the most accurate method for detecting hand gestures in smartwatches.”

Written by: Pietro Paolo Frigenti

Journal Reference: McIntosh, J., Marzo, A., Fraser, A., and Phillips, C. (2017). EchoFlex: Hand Gesture Recognition using Ultrasound Imaging. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ‘17). ACM, New York, NY, USA, 1923-1934. DOI: https://doi.org/10.1145/3025453.3025807