Category Archives: hci

Personal and Ubiquitous Computing theme issue: Automotive user interfaces and interactive applications in the car

I’m thrilled to announce that the theme issue of Personal and Ubiquitous Computing entitled “Automotive user interfaces and interactive applications in the car” is now available in PUC’s Online First. I had the pleasure of serving as co-editor of this theme issue with Albrecht Schmidt, Anind Dey, and Susanne Boll.

The theme issue includes our editorial [1], and three papers. The first is by Tuomo Kujala, who explores scrolling on touch screens while driving [2]. The second is by Florian Schaub, Markus Hipp, Frank Kargl, and Michael Weber, who address the issue of credibility in the context of automotive navigation systems [3]. The third paper is co-authored by me, my former PhD student Alex Shyrokov, and Peter Heeman. We explore multi-threaded spoken dialogues between a driver and a remote conversant [4]. The three papers were selected in a rigorous review process from 17 submissions, by approximately 50 reviewers.

References

[1] Andrew L. Kun, Albrecht Schmidt, Anind Dey and Susanne Boll, “Automotive User Interfaces and Interactive Applications in the Car,” PUC Online First

[2] Tuomo Kujala, “Browsing the Information Highway while Driving – Three In-Vehicle Touch Screen Scrolling Methods and Driver Distraction,” PUC Online First

[3] Florian Schaub, Markus Hipp, Frank Kargl, and Michael Weber, “On Credibility Improvements for Automotive Navigation Systems,” PUC Online First

[4] Andrew L. Kun, Alexander Shyrokov, and Peter A. Heeman, “Interactions between Human-Human Multi-Threaded Dialogues and Driving,” PUC Online First

Video calling while driving? Not a good idea.

Do you own a smart phone? If yes, you’re likely to have tried video calling (e.g. with Skype or FaceTime). Video calling is an exciting technology, but as Zeljko Medenica and I show in our CHI 2012 Work-in-Progress paper [1], it’s not a technology you should use while driving.

Zeljko and I conducted a driving simulator experiment in which a driver and another participant were given the verbal task of playing the game of Taboo. The driver and the passenger were in separate rooms and spoke to each other over headsets. In one experimental condition, the driver and the other participant could also see each other as shown in the figure below. We wanted to find out if in this condition drivers would spend a significant amount of time looking at the other participant. This is an important question, as time spent looking at the other participant is time not spent looking at the road ahead!

We found that, when drivers felt that the driving task was demanding, they focused on the road ahead. However, when they perceived the driving task to be less demanding they looked at the other participant significantly more.

What this tells us is that, under certain circumstances, drivers are willing to engage in video calls. This is due, at least in part, to the (western) social norm of looking at the person you’re talking to. These results should serve as a warning to interface designers, lawmakers (yes, there’s concern [2]), transportation officials, and drivers that video calling can be a serious distraction from driving.

Here’s a video that introduces the experiment in more detail:

References

[1] Andrew L. Kun, Zeljko Medenica, “Video Call, or Not, that is the Question,” to appear in CHI ’12 Extended Abstracts

[2] Claude Brodesser-Akner, “State Assemblyman: Ban iPhone4 Video-Calling From the Road,” New York Magazine. Date accessed 03/02/2012

Further progress towards disambiguating the effects of cognitive load and light on pupil diameter

In driving simulator studies participants complete both visual and aural task. The most obvious visual task is driving itself, but there are others such as viewing an LCD screen that displays a map. Aural tasks include talking to an in-vehicle computer. I am very interested in estimating the cognitive load of these various tasks. One way to estimate this cognitive load is through changes in pupil diameter: in an effect called the Task Evoked Pupillary Response (TEPR) [1], the pupil dilates with increased cognitive load.

However, in driving simulator studies participants scan a non-uniformly illuminated visual scene. If unaccounted for, this non-uniformity in illumination might introduce an error in our estimate of the TEPR. Oskar Palinko and I will have a paper at ETRA 2012 [2] extending our previous work [3], in which we established that it is possible to separate the pupil’s light reflex from the TEPR. While in our previous work TEPR was the result of participants’ engagement in an aural task, in our latest experiment TEPR is due to engagement in a visual task.

The two experiments taken together support our main hypothesis that it is possible to disambiguate (and not just separate) the two effects even in complicated environments, such as a driving simulator. We are currently designing further experiments to test this hypothesis.

References

[1] Jackson Beatty, “Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources,” Psychological Bulletin, 276-292, 91(2)

[2] Oskar Palinko, Andrew L. Kun, “Exploring the Effects of Visual Cognitive Load and Illumination on Pupil Diameter in Driving Simulators,” to appear at ETRA 2012

[3] Oskar Palinko, Andrew L. Kun, “Exploring the Influence of Light and Cognitive Load on Pupil Diameter in Driving Simulator Studies,” Driving Assessment 2011

2011 opportunity for UNH CS students: multi-touch surface manipulation of geo-coded time series

When I think back to the recent BP oil spill in the Gulf of Mexico, the images that come to mind are of wildlife affected on beaches, idle fishing vessels, and a massive response that involved thousands of people across multiple states.

How can such a massive response be managed? There is no single answer. However, one thing that can help is to make data about various aspects of the disaster, as well as the response effort, accessible to those conducting the response activities. This is the role of the Environmental Response Management Application (ERMA). ERMA is a web-based data visualization application. It visualizes geo-coded time series, without requiring users to know how to access specialized databases, or overlay data from these databases on virtual maps. ERMA was developed at UNH, under the guidance of the Coastal Response Research Center (CRRC).

Nancy Kinner is the co-director of the UNH Coastal Response Research Center. Building on Nancy’s experiences with ERMA, she and I are interested in exploring how a multi-touch table could be used to access and manipulate geo-coded time series.

Seeking UNH CS student

To further are effort, we are seeking a UNH CS student interested in developing a user interface on a multi-touch table. The interface would allow a human operator to access remote databases, manipulate the data (e.g. by sending it to Matlab for processing) and display the results on a virtual map or a graph. This work will be part of a team effort with two students working with Nancy on identifying data and manipulations of interest.

What should the user interface do?

The operator should be able to select data, e.g. from a website such as ERMA. Data types of interest include outputs from various sensors (temperature, pressure, accelerometers, etc.). Data manipulation will require some simple processing, such as setting beginning and end points for sensor readings. It will also require more complex processing of data, e.g. filtering.

What platform will be used?

The project will leverage Project54’s Microsoft Surface multi-touch table. Here is a video by UNH ECE graduate student Tim April introducing some of the interactions he has explored with the Surface.

What are the terms of this job?

We are interested in hiring an undergraduate or graduate UNH CS student for the 2011-2012 academic year, with the possibility of extending the appointment for the summer of 2012 and beyond, pending satisfactory performance and the availability of funding. The student will work up to 20 hours/week during the academic year and up to 40 hours a week during the summer break.

What are the required skills? And what new skills will I acquire?

Work on this ream-project will require object-oriented programming that is necessary to control the multi-touch table. You will explore the application of these skills to the design of surface user interfaces as well as experiments with human subjects – after all we will have to systematically test your creation! Finally, you will interact with students and faculty from at least two other disciplines (civil/environmental and electrical/computer engineering), which means you will gain valuable experience working on multi-disciplinary teams.

Interested? Have questions, ideas, suggestions?
Email me.

2011 Cognitive Load and In-Vehicle Human-Machine Interaction workshop

I’m thrilled to announce the 2011 Cognitive Load and In-Vehicle Human-Machine Interaction workshop (CLW 2011) to be held at AutomotiveUI 2011 in Salzburg, Austria. I’m co-organizing the workshop with Peter Heeman, Tim Paek, Tom Miller, Paul Green, Ivan Tashev, Peter Froehlich, Bryan Reimer, Shamsi Iqbal and Dagmar Kern.

Why have this workshop? Interactions with in-vehicle electronic devices can interfere with the primary task of driving. The concept of cognitive load helps us understand the extent to which these interactions interfere with the driving task and how this interference can be mitigated. While research results on in-vehicle cognitive load are frequently presented at automotive research conferences and in related journals, so far no dedicated forum is available for focused discussions on this topic. This workshop aims to fill that void.

Submissions to the workshop are due October 17. Topics of interest include, but are not limited to:

– Cognitive load estimation in the laboratory,
– Cognitive load estimation on the road,
– Sensing technologies for cognitive load estimation,
– Algorithms for cognitive load estimation,
– Performance measures of cognitive load,
– Physiological measures of cognitive load,
– Visual measures of cognitive load,
– Subjective measures of cognitive load,
– Methods for benchmarking cognitive load,
– Cognitive load of driving,
– Cognitive overload and cognitive underload,
– Approaches to cognitive load management inspired by human-human interactions.

For a detailed description of workshop goals take a look at the call for papers.

Augmented Reality vs. Street View for Personal Navigation Devices

Personal navigation devices (PNDs) are ubiquitous and primarily come in three forms: as built-in devices in vehicles, as brought-in stand-alone devices, or as applications on smart phones.

So what is next for PNDs? In a driving simulator study to be presented at MobileHCI 2011 [1], Zeljko Medenica, Tim Paek, Oskar Palinko and I explored two ideas:

Augmented reality PND: An augmented reality PND overlays route guidance on the real world using a head-up display. Our version is simulated and we simply project the route guidance on the simulator screens along with the driving simulation images. Augmented reality PNDs are not yet available commercially for cars.
Street-view PND: This PND uses a simplified version of augmented reality. It overlays route guidance on a sequence of still images of the road. The images and overlay are displayed on a head-down display. Google Maps Navigation runs on smart phones and can be used with street view.

The following video demonstrates the two PNDs.

Our findings indicate that augmented reality PNDs allow for excellent visual attention to the road ahead and excellent driving performance. In contrast, street-view PNDs can have a detrimental effect on both. Thus, while further research is clearly needed, it might be best if navigation with a street view PND was handled by a passenger and not by the driver.

References

[1] Zeljko Medenica, Andrew L. Kun, Tim Paek, Oskar Palinko, “Augmented Reality vs. Street Views: A Driving Simulator Study Comparing Two Emerging Navigation Aids,” to appear at MobileHCI 2011

Zeljko Medenica advances to candidacy

Last week my PhD student Zeljko Medenica advanced to candidacy. Zeljko plans to create a driving performance measure that would be sensitive to short-lived and/or infrequent degradations in driving performance. In previous driving simulator-based studies [1, 2] we found that glancing away from the road is correlated with worse driving performance. Importantly, this is true even when performance averages over the length of the entire experiment are not affected. Thus, Zeljko plans to explore the use of cross-correlation in creating a new, highly sensitive driving performance measure.

Zeljko’s PhD committee includes Paul Green (UMTRI), Tim Paek (Microsoft Research), Nicholas Kirsch (UNH) and Tom Miller (UNH). Thanks to all for serving!

References

[1] Andrew L. Kun, Tim Paek, Zeljko Medenica, Nemanja Memarovic, Oskar Palinko, “Glancing at Personal Navigation Devices Can Affect Driving: Experimental Results and Design Implications,” Automotive UI 2009

[2] Zeljko Medenica, Andrew L. Kun, Tim Paek, Oskar Palinko, “Augmented Reality vs. Street Views: A Driving Simulator Study Comparing Two Emerging Navigation Aids,” to appear at MobileHCI 2011

PhD and MS position at the University of New Hampshire exploring in-car human-computer interaction

A PhD and an MS position are available in the Project54 lab at the University of New Hampshire. The lab is part of the Electrical and Computer Engineering department at UNH. Successful applicants will explore human-computer interaction in vehicles.

The Project54 lab was created in 1999 in partnership with the New Hampshire Department of Safety to improve technology for New Hampshire law enforcement. Project54’s in-car system integrates electronic devices in police cruisers into a single voice-activated system. Project54 also integrates cruisers into agency-wide communication networks. The Project54 system has been deployed in over 1000 vehicles in New Hampshire in over 180 state and local law enforcement agencies.

Research focus

Both the PhD and the MS student will focus on the relationship between various in-car user interface characteristics and the cognitive load of interacting with these interfaces, with the goal of designing interfaces that do not significantly increase driver workload. Work will involve developing techniques to estimate cognitive load using performance measures (such as the variance of lane position), physiological measures (such as changes in pupil diameter) and subjective measures (such as the NASA-TLX questionnaire).

The work will utilize experiments in Project54’s world-class driving simulator laboratory which is equipped with two research driving simulators, three eye trackers and a physiological data logger. Laboratory experiments will be complemented by field deployments in law enforcement agencies such as the New Hampshire State Police, which operates over 300 police cruisers. Project54 has deployed a state-wide data update infrastructure for the New Hampshire State Police which allows remote updates to in-car experimental software and remote collection of experimental data.

Appointment

The PhD student will be appointed for four years, and the MS student for two years. Initial appointments will be for one year, starting between June and September 2011. Continuation of funding will be dependent on satisfactory performance. Appointments will be a combination of research and teaching assistantships. Compensation will include tuition, fees, health insurance and academic year and summer stipend.

How to apply

For application instructions, and for general information, email Andrew Kun, Project54 Principal Investigator at andrew.kun@unh.edu. Please attach a current CV.

Talk at SpeechTEK 2010

On Tuesday (August 3, 2010) I attended SpeechTEK 2010. I had a chance to see several really interesting talks including the lunch keynote by Zig Serafin, General Manager, Speech at Microsoft. He and two associates discussed, among other topics, the upcoming release of the Windows 7 phone and of the Kinect for Xbox 360 (formerly Project Natal). We also saw successful live demonstrations of both of these technologies.

One of Zig’s associates to take the stage was Larry Heck, Chief Scientist, Speech at Microsoft. Larry believes that there are three areas of research and development that will combine to make speech a part of everyday interactions with computers. First, the advent of ubiquitous computing and the need for natural user interfaces (NUIs) means that we cannot keep relying on GUIs and keyboards for many of our computing needs. Second, cloud computing makes it possible to gather rich data to train speech systems. Finally, with advances in speech technology we can expect to see search move beyond typing keywords (which is what we do today sitting at our PCs) to conversational queries (which is what people are starting to do on mobile phones).

I attended four other talks with topics relevant to my research. Brigitte Richardson discussed her work on Ford’s Sync. It’s exciting to hear that Ford is coming out with an SDK that will allow integrating devices with Sync. This appears to be a similar approach to ours at Project54 – we also provide an SDK which can be used to write software for the Project54 system [1]. Eduardo Olvera of Nuance discussed the differences and similarities between designing interfaces for speech interaction and those for interaction on a small form factor screen. Karen Kaushansky of TellMe discussed similar issues focusing on customer care. Finally, Kathy Lee, also of TellMe, discussed her work on a diary study exploring when people are willing to talk to their phones. This work reminded me of an experiment in which Ronkainen et al. asked participants to rate the social acceptability of mobile phone usage scenarios they viewed in video clips [2].

I also had a chance to give a talk reviewing some of the results of my collaboration with Tim Paek of Microsoft Research. Specifically, I discussed the effects of speech recognition accuracy and PTT button usage on driving performance [3] and the use of voice-only instructions for personal navigation devices [4]. The talk was very well received by the audience of over 25, with many follow-up questions. Tim also gave this talk earlier this year at Mobile Voice 2010.

For pictures from SpeechTEK 2010 visit my Flickr page.

References

[1] Andrew L. Kun, W. Thomas Miller, III, Albert Pelhe and Richard L. Lynch, “A software architecture supporting in-car speech interaction,” IEEE Intelligent Vehicles Symposium 2004

[2] Sami Ronkainen, Jonna Häkkilä, Saana Kaleva, Ashley Colley, Jukka Linjama, “Tap Input as an Embedded Interaction Method for Mobile Devices,” TEI 2007

[3] Andrew L. Kun, Tim Paek, Zeljko Medenica, “The Effect of Speech Interface Accuracy on Driving Performance,” Interspeech 2007

[4] Andrew L. Kun, Tim Paek, Zeljko Medenica, Nemanja Memarovic, Oskar Palinko, “Glancing at Personal Navigation Devices Can Affect Driving: Experimental Results and Design Implications,” Automotive UI 2009

Albrecht Schmidt visit to UNH

Last month (April 16) Albrecht Schmidt visited UNH and the Project54 lab. Albrecht gave an excellent talk introducing some of the research problems in pervasive computing and specifically touching on the latest results from his lab, which were just published at CHI 2010 [1, 2]. I was especially interested in the work on helping users find the last place of interest on a map quickly. Albrecht and colleagues track the user’s gaze and when the user looks away, they place a marker (or gazemark) on the map. When the user looks back at the map she can start where she left off: at the place of the marker. Clearly this could be very useful when looking at GPS maps in a car. In such a situation the driver has to keep going back and forth between the map and the road and you want to minimize the time spent looking at the map (the road being the more important thing to look at!). The gazemarks introduced by Albrecht’s group may help. It would be interesting to conduct a driving simulator study with gazemarks.

After the talk Albrecht spent about an hour with students from the Project54 lab and those in my Ubicomp Fundamentals course. This was a more intimate setting for conversations about Albrecht’s research. Finally, Project54 staff and students spent a couple of hours discussing Project54 research with Albrecht – our work on handheld computers, on driving simulator-based investigations of in-car user interfaces and our budding efforts in multi-touch table interaction.

I am grateful to the UNH Provost’s Office for helping to fund Albrecht’s visit through a grant from the Class of 1954 Academic Enrichment Fund.

References

[1] Dagmar Kern, P. Marshall and Albrecht Schmidt, ” Gazemarks: gaze-based visual placeholders to ease attention switching,” CHI 2010

[2] Alireza Sahami Shirazi, Ari-Heikki Sarjanoja, Florian Alt, Albrecht Schmidt, and Jonna Häkkilä, J. “Understanding the impact of abstracted audio preview of SMS,” CHI 2010