Archives

Towards disambiguating the effects of cognitive load and light on pupil diameter

Light intensity affects pupil diameter: the pupil contracts in bright environments and it dilates in the dark. Interestingly, cognitive load also affects pupil diameter, with the pupil dilating in response to increased cognitive load. This effect is called the task evoked pupillary response (TEPR) [1]. Thus, changes in pupil diameter are physiological measures of cognitive load; however changes in lighting introduce noise into the estimate.

Last week Oskar Palinko gave a talk at Driving Assessment 2011 introducing our work on disambiguating the effects of cognitive load and light on pupil diameter in driving simulator studies [2]. We hypothesized that we can simply subtract the effect of lighting on pupil diameter from the combined effect of light and cognitive load and produce an estimate of cognitive load only. We tested the hypothesis through an experiment in which participants were given three tasks:

  • Cognitive task with varying cognitive load and constant lighting. This task was adapted from the work of Klingner et al. [3]. Participants listened to a voice counting from 1 to 18 repeatedly. Participants were told that every sixth number (6, 12, and 18) might be out of order and were instructed to push a button if they detected an out-of-order number. This task induced increased cognitive load at every sixth number as participants focused on the counting sequence. A new number was read every 1.5 seconds, thus cognitive load (and pupil diameter) increased every 6 x 1.5 sec = 9 seconds.
  • Visual task with constant cognitive load (assuming no daydreaming!) and varying lighting. Participants were instructed to follow a visual target which switched location between a white, a gray and a black truck. The light reaching the participant’s eye varied as the participant’s gaze moved from one truck to another. Participants held their gaze on a truck for 9 seconds, allowing the pupil diameter ample time to settle.
  • Combined task with varying cognitive load and lighting. Participants completed the cognitive and visual tasks in parallel. We synchronized the cognitive and visual tasks such that increases in cognitive load occurred after the pupil diameter stabilized in response to moving the gaze between trucks. Synchronization was straightforward as the cognitive task was periodic with 9 seconds and in the visual task lighting intensity also changed every 9 seconds.

Our results confirm that, at least in this simple case, our hypothesis holds and we can indeed detect changes in cognitive load under varying lighting conditions. We are planning to extend this work by introducing scenarios in which participants drive in realistic simulated environments. Under such scenarios gaze angles, and thus the amount of light reaching participants’ eyes, will change rapidly, making the disambiguation more complex, and of course more useful.

References

[1] Jackson Beatty, “Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources,” Psychological Bulletin, 276-292, 91(2)

[2] Oskar Palinko, Andrew L. Kun, “Exploring the Influence of Light and Cognitive Load on Pupil Diameter in Driving Simulator Studies,” Driving Assessment 2011

[3] Jeff Klingner, Rashit Kumar, Pat Hanrahan, “Measuring the Task-Evoked Pupillary Response with a Remote Eye Tracker,” ETRA 2008

Bryan Reimer visit to UNH

It was my great pleasure to host Bryan Reimer at UNH. Bryan is Research Scientist at the MIT Age Lab as well as Associate Director of the New England University Transportation Center. His research focuses on the measurement and understanding of human behavior in dynamic environments, such as in cars.

Bryan spent time in the Project54 lab discussing various aspects of driving simulator and field studies. He then gave a thought-provoking talk reviewing results from multiple studies exploring driver workload and distraction. I expecially enjoyed his discussion of physiological measures that can be used to estimate workload. E.g. Bryan has found that heart rate is a robust estimate of workload and is often more useful than the often-used measure of heart rate variability. Bryan also discussed work on validating driving simulator results through field studies. His data indicate that driving simulator results can be used to predict relative changes in workload measures under different situations in real-life driving. However, the actual values of the measures collected in simulator and field studies often differ significantly.

For more pictures visit Flickr.

Co-chairing AutomotiveUI 2010

On November 11 and 12 I was at the AutomotiveUI 2010 conference serving as program co-chair with Susanne Boll. The conference was hosted by Anind Dey at CMU and co-chaired by Albrecht Schmidt.

The conference was successful and really fun. I could go on about all the great papers and posters (including two posters from our group at UNH [1,2]) but in this post I’ll only mention two: John Krumm’s keynote talk and, selfishly, my own talk (this is my blog after all). John gave an overview of his work with data from GPS sensors. He discussed work on prediction of where people will go, his experiences with location privacy and with creating road maps. Given that John is, according to his own website, the “all seeing, all knowing, master of time, space, and dimension,” this was indeed a very informative talk ;) OK in all seriousness, the talk was excellent. I find John’s work on prediction of people’s destination and selected route the most interesting. One really interesting effect of having accurate predictions, and people sharing such data in the cloud, would be on routing algorithms hosted in the cloud. If such an algorithm could know where all of us are going at any instant of time, it could propose routes that overall allow efficient use of roads, reduced pollution, etc.

My talk focused on collaborative work with Alex Shyrokov and Peter Heeman on multi-threaded dialogues. Specifically, I talked about designing spoken tasks for human-human dialogue experiments for Alex’s PhD work [3]. Alex wanted to observe how pairs of subjects switch between two dialogue threads, while one of the subjects is also engaged in operating a simulated vehicle. Our hypothesis is that observed human-human dialogue behaviors can be used as the starting point for designing computer dialogue behaviors for in-car spoken dialogue systems. One of the suggestions we put forth in the paper is that the tasks for human-human experiments should be engaging. These are the types of tasks that will result in interesting dialogue behaviors and can thus teach us something about how humans manage multi-threaded dialogues.

Next year the conference moves back to Europe. The host will be Manfred Tscheligi in Salzburg, Austria. Judging by the number of submissions this year and the quality of the conference, we can look forward to many interesting papers next year, both from industry and from academia. Also, the location will be excellent – just think Mozart, Sound of Music (see what Rick Steves has to say), and world-renowned Christmas markets!

References

[1] Zeljko Medenica, Andrew L. Kun, Tim Paek, Oskar Palinko, “Comparing Augmented Reality and Street View Navigation,” AutomotiveUI 2010 Adjunct Proceedings

[2] Oskar Palinko, Sahil Goyal, Andrew L. Kun, “A Pilot Study of the Influence of Illumination and Cognitive Load on Pupil Diameter in a Driving Simulator,” AutomotiveUI 2010 Adjunct Proceedings

[3] Andrew L. Kun, Alexander Shyrokov, Peter A. Heeman, “Spoken Tasks for Human-Human Experiments: Towards In-Car Speech User Interfaces for Multi-Threaded Dialogue,” AutomotiveUI 2010

Report on ECE Graduate Seminar

During the 2009-2010 academic year I taught a new version of the UNH ECE Graduate Seminar (ECE 900), a course I first introduced in the fall of 2002. The primary aim of the previous version of the course was to expose our graduate students to research and development conducted at other institutions. Thus, the course consisted of eight invited talks per semester, given by engineers, scientist and other professionals, and covering a range of topics of interest to ECE students. 

Starting with the 2009-2010 academic year the primary aim of the course has become to introduce graduate students to the general tools of scientific research. I championed this new aim for the course and I’m grateful that my faculty colleagues gave me an opportunity to share my excitement about scientific research with our graduate students. 

The course had three main aspects: 

  • Lectures on performing scientific research. My lectures introduced students to the steps of scientific research, from formulating problems, to proposing hypotheses and conducting experiments.
  • Research talks. Each student attended at least 15 talks. Most of these were held at UNH and the speakers were exceptional. At the same time, I encouraged the students to recommend talks that we can attend at other institutions. The result: trips to MIT, BU and WPI.
  • Research proposal. At the end of the two semester sequence each student submitted a short research proposal and gave a presentation on the same. The proposals were developed over the two semesters, with students working individually and in groups. I provided feedback throughout the year on different segments of the proposal. 

In an informal survey at the end of the academic year most students indicated that they liked the new version of the ECE Graduate Seminar and that they thought it was useful. All of the students thought that learning about the tools of science is useful and the majority also indicated that their technical writing skills improved due to this course. These responses are certainly encouraging. 

I will be teaching ECE 900 again during the 2010-2011 academic year. Based on my experiences reported here, as well as those with my Fundamentals of Ubicomp course, I plan to implement two changes: 

  • Accelerate proposal development. I will move up the due date for the final research proposal to sometime early in the second semester. The accelerated schedule should help build excitement for learning about science. It will also give us time at the end of the year to discuss how other researchers approach scientific work. Finally, it will help with student participation in the course, which is the subject of the second change I intend to implement.
  • Increase student participation. While I encouraged student participation throughout the semester, the results were not always stellar. By accelerating the proposal development process I hope to provide students with discussion topics that they feel comfortable talking about. I also intend to ask students to hold multiple formal presentations in class. One assignment that students can expect next semester: create a 15 minute presentation about a research topic of your choice, based on a research video posted online.

Talk at SpeechTEK 2010

On Tuesday (August 3, 2010) I attended SpeechTEK 2010. I had a chance to see several really interesting talks including the lunch keynote by Zig Serafin, General Manager, Speech at Microsoft. He and two associates discussed, among other topics, the upcoming release of the Windows 7 phone and of the Kinect for Xbox 360 (formerly Project Natal). We also saw successful live demonstrations of both of these technologies.

One of Zig’s associates to take the stage was Larry Heck, Chief Scientist, Speech at Microsoft. Larry believes that there are three areas of research and development that will combine to make speech a part of everyday interactions with computers. First, the advent of ubiquitous computing and the need for natural user interfaces (NUIs) means that we cannot keep relying on GUIs and keyboards for many of our computing needs. Second, cloud computing makes it possible to gather rich data to train speech systems. Finally, with advances in speech technology we can expect to see search move beyond typing keywords (which is what we do today sitting at our PCs) to conversational queries (which is what people are starting to do on mobile phones).

I attended four other talks with topics relevant to my research. Brigitte Richardson discussed her work on Ford’s Sync. It’s exciting to hear that Ford is coming out with an SDK that will allow integrating devices with Sync. This appears to be a similar approach to ours at Project54 – we also provide an SDK which can be used to write software for the Project54 system [1]. Eduardo Olvera of Nuance discussed the differences and similarities between designing interfaces for speech interaction and those for interaction on a small form factor screen. Karen Kaushansky of TellMe discussed similar issues focusing on customer care. Finally, Kathy Lee, also of TellMe, discussed her work on a diary study exploring when people are willing to talk to their phones. This work reminded me of an experiment in which Ronkainen et al. asked participants to rate the social acceptability of mobile phone usage scenarios they viewed in video clips [2].

I also had a chance to give a talk reviewing some of the results of my collaboration with Tim Paek of Microsoft Research. Specifically, I discussed the effects of speech recognition accuracy and PTT button usage on driving performance [3] and the use of voice-only instructions for personal navigation devices [4]. The talk was very well received by the audience of over 25, with many follow-up questions. Tim also gave this talk earlier this year at Mobile Voice 2010.

For pictures from SpeechTEK 2010 visit my Flickr page.

References

[1] Andrew L. Kun, W. Thomas Miller, III, Albert Pelhe and Richard L. Lynch, “A software architecture supporting in-car speech interaction,” IEEE Intelligent Vehicles Symposium 2004

[2] Sami Ronkainen, Jonna Häkkilä, Saana Kaleva, Ashley Colley, Jukka Linjama, “Tap Input as an Embedded Interaction Method for Mobile Devices,” TEI 2007

[3] Andrew L. Kun, Tim Paek, Zeljko Medenica, “The Effect of Speech Interface Accuracy on Driving Performance,” Interspeech 2007

[4] Andrew L. Kun, Tim Paek, Zeljko Medenica, Nemanja Memarovic, Oskar Palinko, “Glancing at Personal Navigation Devices Can Affect Driving: Experimental Results and Design Implications,” Automotive UI 2009

Return visit to Budapest University of Technology and Economics (BUTE)

On June 7 and 8, 2010 I visited the Budapest University of Technology and Economics (BUTE) for the second time in ten months. As with my last visit I went to discuss the BUTE-CEPS exchange program.

During this visit I met six people who have been involved in organizing different aspects of the exchange program. My host was Eszter Kiss, the Program Director of the Information Center for Engineering Programs in English (ICEPE). For UNH/CEPS students, staff and faculty, she is the Hungarian face of the exchange program. Eszter and I primarily talked about the fact that, starting in 2011, UNH ECE exchange students will spend the spring semester in Budapest. Other CEPS students will remain on the fall-in-Budapest schedule.

Eszter organized two meetings for me with BUTE leaders. The first one was with Dr. Peter Moson, Vice-Rector for International Relations (the Vice-Rector position at BUTE is equivalent to the Vice President position at a US university). Ildiko Varga, the head of the BUTE Erasmus and Exchange Office was also present at this meeting. Dr. Moson expressed his full support for a vibrant relationship between BUTE and CEPS. On a personal note it was great to see Dr. Moson who I met during his visit to UNH last year. It was also nice to talk to Ms. Varga who went to graduate school and taught mathematics at Purdue.

The second meeting organized by Eszter was with Dr. Gabor Stepan. Dr. Stepan, a member of the Hungarian Academy of Sciences (HAS), is the Dean of the BUTE Faculty of Mechanical Engineering, the ICEPE’s parent unit. Dr. Stepan expressed his full support for the BUTE-CEPS exchange program. Again on a personal note, it was exciting for me to visit the BUTE Faculty of ME where my father received his BS ME a long time ago. Dr. Stepan also spent some time telling me about BUTE’s history, including facts and anecdotes about BUTE’s Nobel-prize winning alumni.

While the meetings with Drs Stepan and Moson and with Ms. Varga primarily dealt with the overall BUTE-CEPS relationship, I also had a chance to work on issues related to UNH ECE directly with the BUTE unit that hosts ECE students. Specifically, Dr. Moson introduced me to Dr. Balint Kiss, the person in charge of the English language education at the Faculty of Electrical Engineering and Informatics. This is the BUTE unit that hosts UNH ECE exchange students and Dr. Kiss will be my primary contact in determining courses for our students to take while at BUTE. The meeting with Dr. Kiss was also an opportunity to catch up with Dr. Peter Arato. Dr. Arato, who is also a HAS member, has strong ties to the UNH ECE department having collaborated extensively with UNH ECE professor Andrzej Rucinski.

In addition to all these productive meetings I had a chance to give a talk to BUTE students interested in the exchage program. Seven prospective students attended, several of them interested in coming to the UNH ECE department – I hope we’ll see them here soon.

I would like to thank Eszter Kiss for organizing my visit (on very short notice). I would also like to thank the BUTE faculty, staff and students who took time to meet with me. Finally, I would like to acknowledge the UNH ECE Department and the CEPS Dean’s office who jointly funded this visit.

For pictures about my trips to Budapest visit my Flickr page.

Visit to FTW, Vienna

On June 4, 2010 I visited the Telecommunications Research Center Vienna (FTW). My host was Peter Froehlich, Senior Researcher in FTW’s User-Centered Interaction area of activity. Peter and I met at the CHI SIG meeting on automotive user interfaces [1] that I helped organize.

Peter and his colleagues are investigating automotive navigation aids and are currently preparing for an on-road study. I’m happy to report that this study will utilize one of our eye trackers. My visit provided an opportunity for us to discuss this upcoming study and how the eye tracker may be useful in evaluating the research hypotheses. Part of this discussion was a Telecommunications Forum talk I gave – see the slides below:

I want to thank Peter and his colleagues at FTW for hosting me and I’m looking forward to our upcoming collaboration. I also want to thank FTW for providing funding for my visit.

References

[1] Albrecht Schmidt, Anind L. Dey, Andrew L. Kun, Wolfgang Spiessl, “Automotive User Interfaces: Human Computer Interaction in the Car,” CHI 2010 Extended Abstracts

Albrecht Schmidt visit to UNH

Last month (April 16) Albrecht Schmidt visited UNH and the Project54 lab. Albrecht gave an excellent talk introducing some of the research problems in pervasive computing and specifically touching on the latest results from his lab, which were just published at CHI 2010 [1, 2]. I was especially interested in the work on helping users find the last place of interest on a map quickly. Albrecht and colleagues track the user’s gaze and when the user looks away, they place a marker (or gazemark) on the map. When the user looks back at the map she can start where she left off: at the place of the marker. Clearly this could be very useful when looking at GPS maps in a car. In such a situation the driver has to keep going back and forth between the map and the road and you want to minimize the time spent looking at the map (the road being the more important thing to look at!). The gazemarks introduced by Albrecht’s group may help. It would be interesting to conduct a driving simulator study with gazemarks.

After the talk Albrecht spent about an hour with students from the Project54 lab and those in my Ubicomp Fundamentals course. This was a more intimate setting for conversations about Albrecht’s research. Finally, Project54 staff and students spent a couple of hours discussing Project54 research with Albrecht – our work on handheld computers, on driving simulator-based investigations of in-car user interfaces and our budding efforts in multi-touch table interaction.

I am grateful to the UNH Provost’s Office for helping to fund Albrecht’s visit through a grant from the Class of 1954 Academic Enrichment Fund.

References

[1] Dagmar Kern, P. Marshall and Albrecht Schmidt, ” Gazemarks: gaze-based visual placeholders to ease attention switching,” CHI 2010

[2] Alireza Sahami Shirazi, Ari-Heikki Sarjanoja, Florian Alt, Albrecht Schmidt, and Jonna Häkkilä, J. “Understanding the impact of abstracted audio preview of SMS,” CHI 2010

At the 2009 fall NIJ CommTech TWG meeting

On Wednesday and Thursday, Oskar Palinko, Mark Taipan and I participated in the NIJ CommTech Technical Working Group meeting. On Wednesday I gave the presentation below reporting on our lab’s progress.

Project54 Status Report, October 2009
View more presentations from Andrew Kun.

On Thursday we participated in the meeting’s demo session. We demonstrated the advantage of using voice commands to control a police radio over using the radio’s buttons. We used a single-computer driving simulator and a radio setup. Of course the first driving simulator experiment we published investigated this effect [1]. We also demonstrated accessing a remote database using the Project54 system running on a Symbol handheld computer. We expect that, once we get approval from the NH State Police to deploy such devices (NHSP is responsible for data access for all officers in the state), they will be a big hit with local departments.

One of the many people we had a chance to talk to at the TWG meeting is Gil Emery, Communications Manager at the Portsmouth, NH PD. Gil was interested in the handhelds and we may be able to work with him on using these handhelds as cameras that allow tagging pictures on the spot and then using a cellular network to transmit them to headquarters. This work would build on Michael Farrar’s MS thesis research.

You can see pictures from this event of Flickr.

References

[1] Zeljko Medenica, Andrew L. Kun, “Comparing the Influence of Two User Interfaces for Mobile Radios on Driving Performance,” Driving Assessment 2007

Visiting Budapest University of Technology and Economics (BUTE)

After my trip to Automotive UI 2009 I flew to Budapest, Hungary. The UNH College of Engineering and Physical Sciences has an exchange program with BUTE and I went to promote this program to BUTE students. I also got a chance to meet two people responsible for implementing the program “on the ground” in Budapest, Eszter Kiss and Máté Helfrich. Eszter is the person who looks after the UNH students (and many others from all over the world) from the time they arrive in Budapest, so I was very happy to meet her and express UNH’s gratitude for all of her efforts.

Eszter organized a talk in which I presented some of the reasons why a semester at UNH would be beneficial to BUTE students (see the slides). The discussion that followed my presentation was excellent, with students asking questions about many aspects of the exchange program, as well as a new summer internship program. The discussion was in Hungarian, which was fun, as I don’t use this language for work very much :)

You can see more pictures about my visit on Flickr.