Facial recognition takes a new step in virtual reality
A face can unlock a smartphone, grant access to a secured building and speed up passport control at airports, confirming a person's identity for many purposes.
An international team of researchers from Australia, New Zealand and India has taken facial recognition technology to a new level, using a person's facial expressions to control objects in virtual reality without a handheld controller or touch panel.
In the first study of its kind in the world, conducted by the University of Queensland, experts in human-computer interaction used neural processing methods to detect a person's smile, frown and jaw clench, and used each expression to trigger specific actions in virtual reality environments.
"Virtual reality (VR) interfaces provide an immersive environment for interacting with the digital world. Most VR interfaces require physical interaction using handheld controllers, but other alternative interaction methods exist that can support different use cases and users. Interaction methods in VR are primarily evaluated in terms of their usability, but their differences in neurological and physiological effects remain less studied," the study states.
One of the researchers involved in the experiment, University of South Australia professor Mark Billinghurst, says the system was designed to recognize different facial expressions through an EEG headset.
“A smile was used to trigger the "move" command, a frown the "stop" command, and a jaw clench the "action" command, instead of a handheld controller performing these actions. Essentially, we capture common facial expressions such as anger, happiness and surprise and implement them in the virtual reality environment”
The researchers developed three virtual environments — happy, neutral and frightening — and measured each participant's cognitive and physiological state while they were immersed in each scenario.
By reproducing three universal facial expressions — smile, frown and clench — they explored whether changes in the environment elicit one of the three expressions, based on emotional and physiological responses.
"Three different environments with different interaction experiences were used: happy (catching butterflies), neutral (object selection) and scary (shooting zombies). Overall, we noticed the influence of interaction methods on gamma activity in the brain and on skin conductance. In some aspects of presence, facial expressions outperformed controllers, but controllers were found to be better than facial expressions in terms of usability," the scientists reported.
For example, in the happy environment, users were tasked with walking through a park and catching butterflies with a net. The user moved when smiling and stopped when frowning.
In the neutral environment, participants were asked to walk through a workshop to collect items scattered around. A clenched jaw triggered an action — in this case picking up each item — while the "start" and "stop movement" commands were triggered by a smile and a frown.
The same facial expressions were used in the frightening environment, where participants moved through an underground base to shoot zombies.
“Overall, we expected handheld controllers to perform better, since they are a more intuitive method than facial expressions, yet people reported feeling more immersed in the virtual reality controlled by facial expressions”
Professor Billinghurst said that relying on facial expressions in virtual reality is hard work for the brain, but gives users a more realistic experience: "I hope that after further research we can make it more user-friendly."
Beyond providing a new way to use virtual reality, this technology will also allow people with disabilities, including people with amputated limbs and patients with motor neuron disease, to interact in virtual reality without using their hands, so they will no longer need controllers designed for fully abled people.
The researchers argue that this technology can also be used to complement handheld controllers, where facial expression is a more natural form of interaction.
The study results were published in International Journal of Human-Computer Studies: "In this paper, alongside other traditional qualitative measures such as presence, affect and system usability, we explore neurophysiological effects — brain signals and electrodermal activity — of using facial expressions as an alternative interaction method for interacting with virtual reality interfaces. This form of interaction was also compared with traditional handheld controllers."
