My PhD was funded by Freescale Semiconductor (now NXP) and supervised by Stephen Brewster and Frank Pollick in the Multimodal Interaction Group.
A car has three channels to reach a driver: ears, eyes and skin. At the time there was no principled account of when to use which, or how to make one alert feel more urgent than another without simply making it louder. I built every unimodal, bimodal and trimodal combination at three designed urgency levels and tested them in a driving simulator, measuring what drivers thought and what they did.
Urgency can be designed and read back
Perceived urgency matched designed urgency, and more urgent warnings produced quicker, more accurate responses. They were rated more annoying too, but annoyance rose more gently than urgency, which leaves room to make a warning clearly more urgent at a modest cost in irritation. Adding modalities had the same shape of effect. AutomotiveUI 2013, later given third place in the conference's 10-Year Impact Award.
The road changes the answer
The same cues behave differently depending on what is happening outside. Delivered with and without a lead car braking hard, responses were faster when the situation itself was critical, and driving behaviour improved when warnings were present. Visual warnings degraded in exactly the moment they were most needed, because the driver was already visually loaded. CHI 2014.
Speech Tactons
I designed tactile warnings that keep the rhythm of a spoken phrase and use roughness and intensity to carry urgency. Paired with speech they were rated more urgent and more effective than either channel alone, and stayed recognisable on their own. Best Paper Award, AutomotiveUI 2014.
Abstract cues against language
A first comparison of abstract pulses against speech, text and Speech Tactons across all modality combinations. In non-critical situations, abstract and visual cues were recognised faster. In critical ones, anything containing audio won, and language based warnings performed as well as abstract ones, so they can be preferred for being less annoying at no cost in response time. CHI 2015.
Handing control back
My last experiment looked at how a driverless car should say you may stop driving or you must start again. I wrote messages for plausible handover situations and tested them while participants played a game on a tablet, as people really would. Urgency came across and the messages were rated effective. The head-up display failed, because nobody was looking at the road.
Talks
- CHI Doctoral Consortium and AutomotiveUI Doctoral Colloquium, ACM (2014)
- SICSA HCI All Hands Meeting, University of Dundee (2014)
- Crossmodal Multisensory Day, University of Glasgow (2014)
- Pint of Science, Glasgow (2014)
- Freescale Future Fridays (2014) and AESIN, University of Warwick (2013)