...

A bit of context

Which one is better?

Evaluation tools

Inquiries (e.g. questionnaires, interviews)
- Distract users
- Sporadic and delayed measures
Behavioral measures (e.g. task completion time)
- Few information about user experience

Using physiological signals

Continuous measures
Non-disruptive
Various dimensions altogether

Fairclough, S. H. (2009). Fundamentals of physiological computing. Interacting with Computers
Mehta, R. K., & Parasuraman, R. (2013). Neuroergonomics: a review of applications to physical and cognitive work. Frontiers in Human Neuroscience

Physiological sensors

Breathing, cardiac activity, electrodermal activity

Practical sensors for (new) popular interactions

Toward more social presence

Short, J. et al. (1976). The Social Psychology of Telecommunications.

Ready?

Steady?

Title!

Leveraging human-computer interactions and social presence with physiological computing

Jérémy Frey

December 8, 2015

(sponsored by Martin Hachet® & Fabien Lotte™)

Measuring brain activity

ElectroEncephaloGraphy

Extracting meaningful info from EEG

Features: spectral, temporal, spatial

Dimensions for evaluating HCI

Frey, J., Mühl, C., Lotte, F., & Hachet, M. (2014). Review of the use of electroencephalography as an evaluation method for human-computer interaction. PhyCS '14

Estimating... visual comfort

Stereoscopic displays

First things first, how to create the illusion of depth? Simply by presenting a different image to each eye. While two identical images produce "flat" images, the greater the discrepancy between what is shown to your left and right eyes, the further (or closer) objects will appear.
And this is where problems start. In real life, when an object comes closer of goes further, your eyeballs rotate accordingly ("vergence") and your eyes focus onto it, as a camera lens would do ("accomodation"). But with stereoscopic displays, while eyeballs do rotate, eyes stay focus on a fixed plane, on the display. This create the vergence-accomodation conflict, one cause of discomfort that occurs with (nearly) all technologies. (NB: better explained with me doing funny faces while mimicking the situation).

How "3D" displays work

Discrepancy between left and right images: illusion of depth

Vergence-Accomodation Conflict

Fixed focal plan but eyeballs rotate: discomfort

Shibata, T., Kim, J., Hoffman, D. M., & Banks, M. S. (2011). The zone of comfort: Predicting visual discomfort with stereo displays. Journal of Vision

Is it possible to measure visual comfort continuously?

Estimating visual comfort in stereoscopic displays

Frey, J., Pommereau, L., Lotte, F., & Hachet, M. (2014). Assessing the zone of comfort in stereoscopic displays using EEG. CHI EA ’14
Frey, J., Appriou, A., Lotte, F., & Hachet, M. (2015). Estimating Visual Comfort in Stereoscopic Displays Using EEG: A Proof-of-Concept. INTERACT ’15
Frey, J., Appriou, A., Lotte, F., & Hachet, M. (2015). Classifying EEG Signals during Stereoscopic Visualization to Estimate Visual Comfort. Computational Intelligence and Neuroscience.

Protocol

3D objects at various depths
Comfortable (C) and no comfortable (NC) zones

Results: EEG

Event related potentials (ERP) around stimuli
Average across 28 channels and 12 subjects

Results: Classification

Use machine learning to discriminate C against NC
160 trials for training, 160 for testing
63% accuracy on average using single-trial

Use-case: adaptive systems

Tune parameters on the fly using a passive Brain-Computer Interface (BCI)

Zander, T. O., & Kothe, C. (2011). Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. J. Neural. Eng

Estimating... workload

Wobrock, D., Frey, J. et al. (2015). Continuous Mental Effort Evaluation during 3D Object Manipulation Tasks based on Brain and Physiological Signals. INTERACT ’15
Frey, J., Daniel, M., Hachet, M., & Lotte, F. Tools for electroencephalography-based evaluation of user experience. CHI '16

Step-by-step

Create an application
Make sure it's difficult
Teach the computer
Compare interaction techniques

A 3D maze to drive them crazy

Parameters:

Speed
Path's length
Change orientation

Validate difficulty levels

Ground truth with NASA-TLX [1] questionnaire (N=15)

"Was the task easy or demanding?"
"How much time pressure did you feel?"
"How successful do you think you were?"

[1] Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Human mental workload

Calibrating workload

N-back task [1,2]

Remembering a sequence of letters
Two conditions: easy vs hard

EEG features

5 frequency bands, between 1 and 40Hz
Spatial filters, 6 "virtual" channels from 32

Classification average: 87% (N=12)

[1] Mühl, C., Jeunet, C., & Lotte, F. (2014). EEG-based workload estimation across affective contexts. Frontiers in Neuroscience

Testing keyboard and touchscreen

Apply N-back training on a real task
Using same EEG features

Workload through EEG: results

Workload through EEG: (continuous) results

Even more

Attention & interaction errors

Attention: how much users notice external stimuli
Interaction errors: how much intuitive an interface is

Recap: EEG as an evaluation method for HCI

Visual comfort with stereoscopic displays
Framework for workload, attention, error recognition

Physiological computing:
better know ourselves and the others

The cute side

One half of the story

Round 2(P.M.) on December 9

Overwhelming EEG

How could we make EEG accessible to novices?

Teegi: Tangible EEG Interface

Frey, J., Gervais, R., Fleck, S., Lotte, F., & Hachet, M. (2014). Teegi: Tangible EEG Interface. UIST '14

Bonding with your avatar

How

Applications

TechFest IIT Bombay 2015

Going further

Give access to both low and high-level signals
Meaningful feedback
Multiple users
Open source components

Tobe: Tangible Out-Of-Body Experience

Gervais, R., Frey, J., Gay, A., Lotte, F., & Hachet, M. (2016). Tobe: Tangible Out-of-Body Experience. TEI ’16.

Take into account people's representations

In the field

We also made a pilot study in a control environment about multi-users interactions. People came by pairs of two. Each had her/his own Tobe. First they had to relax and reach cardiac coherence, a state during which heart rate varies accordingly to breathing. A hippy-like flower was blooming on Tobe's forehead accordingly to the partipant's relaxation state. During a second stage participants had to reach cardiac coherence together, i.e. to synchronize their hearts. To do so they could not see each other, nor talk to each other. They could only watch at the other's Tobe. Hence they only had breathing patterns to communicate. And it worked. Most participants managed to make bloom the shared flower displayed between their Tobes.
More than that, when we informally talked to them afterwards, we were surprised by the complex interactions that emerged from a so simple communication channel. Besides helping to synchronize, breathing patterns let some participants sense when the other was "in trouble". One even playfully tried to "trick" his partner by suddenly breathing in or out (it was a "he" who did that, obviously). All in all, this was very good surprise that came out of the project.

Multi-users relaxation

"I waited for you"
"I saw you had troubles"
"I tried to trick you"

Possible applications

Incoming Tobeegi

Instrumentation

New research leads

Increase presence with biofeedback

Physiological similarity-attraction effect

Frey, J. (2015). Heart Rate Monitoring as an Easy Way to Increase Engagement in Human-Agent Interaction. PhyCS '15

Recently I also setup a multi-users scenario that involves board game, physiological signals and remote sensing. Once again a proxy similar to Tobe is ideal to incarnate inner states. I would have many more comments to do here — new gameplay elements, social interactions, feedback acceptance, uber-techno... — but I'm not sure how much information I can leak before publication :D (AKA the peer-reviewing frustrations). One aspect that I will highlight right now, though, is how much I believe that giving an overt and symmetrical feedback is important both for acceptance and ethics. Users should be both aware and in control of the data they are giving away. Creating a tangible representation of the signals that are inferred from physiological sensors, shared alike by every person present, is one mean to enforce this goal.

Social interactions

Better user experience with symmetrical feedback?

Even though during my presentation I insisted mostly on the good aspects of my work, one should be aware of the pitfalls that lie around.

First and foremost, when dealing with signals as complex as EEG, it is important to ensure that the the claimes reguarding the measures are accurate. Tobe is a bad example in that sense, since with the current prototype high-level signals come from a very simplistic signal processing, far from the rigour of the first studies hereby presented. In the future machine learning and calibration must be added to the mix.
This directly leads to point number 2: we must acknowledge that EEG is not the ultimate brain imaging technique. It possesses a low spatial resolution and is prone (like, very) to artifacts such as muscular activity. It is far from a "plug and play" tool. Not only the hardware must be handled correctly, but on the software side, even with the best intentions, it's difficult to discard all biases. I hold on my trust when it comes to high-level API that obfuscates signal processing behind closed source.
Finally, on a broader scale, I personally seek applications that empower people. With great powers comes great responsibility etc., and I do not feel comfortable with some usages when it comes to neuromarketing, for example. The domain is still young and it could be the right time to establish some dos and don'ts about its end-uses. That is not to say that we should get paranoid about the technology, simply educated and cautious. Rest reassure, I'm an optimistic, greats things ahead!

Limitations to overcome

Ensure reliable measures
The flaws of EEG
Empowering, not objectifying

Conclusion

During my thesis I started by looking at physiological signals. I reviewed the different sensors that could be used, which metrics they could give. I showed how brain signals and EEG can be used to assess various dimensions of the user experience, paving the way for better human-computer interactions.
At the same time, I tried to find and build sensors that are more practical to use: I crafted various wearables, I evaluated the performance of a cheap and open hardware EEG amplifier while participating in its software integration and I implemented a pipeline that can measure remotely the heart rate of several users with off-the shelf webcams.
Technological advances lead to new usages, hence I looked at how physiological computing can increase the social presence of embodied agents and how it can foster new social interactions in everyday life. I participated in the creation of tangible avatars that let anyone learn about self and others, favouring wellbeing, enhancing communication between peers.
Technology is a mean, not an end. Looking back, I came to believe that physiological computing is an opportunity to enhance human-human interactions, the computers being there, in the background, only to support a novel medium. Pour des lendemains qui chantent.
The PhD is an adventure, fortunately I was not alone during this journey; I was lucky enough to spend 3 years in a nurturing environment, hence I wanted to take few seconds to acknowledge the persons that directly took part in my work — more to come in the manuscript!

Rolling Credits

Stereo marathon: Aurélien Appriou

Mad maze: Maxime Daniel

Disco ball: Maxime Duluc

Learning by doing: Stéphanie Fleck

The Designer: Alexis Gay

Twin PhD: Renaud Gervais

Overt Adviser: Martin Hachet

Good solder: Thibault Laine

Papart CEO: Jérémy Laviole

Covert Adviser: Fabien Lotte

Proof reader: Christian Mühl

Stereo ignition: Léonard Pommereau

CubTile master: Dennis Wobrock

Post-credits scene

Here is one final example of how physiological computing can make the world a better place (should have embedded cheesy musing in my slides). Between the moment I wrote my manuscript and my defense I visited for 2 months Musae Lab, in Montréal. I met a fabulous community there, and I participated in Hacking Health, a hackathon that tries to fill the gap between techies and healthcare. Long story short, I teamed up with brilliant minds and we combined physiological sensors with head-mounted displays (somehow the initial subject of my internship)... in order to promote meditation (did not see that one coming, although we reused pieces from Tobe).
Over a week-end we created a virtual and guided meditative experience, where the environment adapts to user's inner state. For example the campfire nearby grows and shrinks according to breathing — more about the initial project and the wonderful team here.
We had very positive feedback from the persons who tested our (very) hacky prototype. We will try to push further these ideas while publishing code and tutorials. When Science encounters the DIY community and personal initiatives, the leverage becomes unstoppable...
(Thanks to Alex for the pictures and the video!)

Breathe@Work

Thanks for your attention

Frey, J., Daniel, M., Hachet, M., & Lotte, F. Tools for electroencephalography-based evaluation of user experience. CHI '16
Gervais, R., Frey, J., Gay, A., Lotte, F., & Hachet, M. (2016). Tobe: Tangible Out-of-Body Experience. TEI ’16.
Frey, J., Appriou, A., Lotte, F., & Hachet, M. (2015). Classifying EEG Signals during Stereoscopic Visualization to Estimate Visual Comfort. Computational Intelligence and Neuroscience.
Wobrock, D., Frey, J. et al. (2015). Continuous Mental Effort Evaluation during 3D Object Manipulation Tasks based on Brain and Physiological Signals. INTERACT ’15
Frey, J., Appriou, A., Lotte, F., & Hachet, M. (2015). Estimating Visual Comfort in Stereoscopic Displays Using EEG: A Proof-of-Concept. INTERACT ’15
Frey, J. (2015). Heart Rate Monitoring as an Easy Way to Increase Engagement in Human-Agent Interaction. PhyCS '15
Frey, J., Gervais, R., Fleck, S., Lotte, F., & Hachet, M. (2014). Teegi: Tangible EEG Interface. UIST '14
Frey, J., Pommereau, L., Lotte, F., & Hachet, M. (2014). Assessing the zone of comfort in stereoscopic displays using EEG. CHI EA ’14
Frey, J., Mühl, C., Lotte, F., & Hachet, M. (2014). Review of the use of electroencephalography as an evaluation method for human-computer interaction. PhyCS '14