Author Archives: Andrew

Augmented Reality vs. Street View for Personal Navigation Devices

Personal navigation devices (PNDs) are ubiquitous and primarily come in three forms: as built-in devices in vehicles, as brought-in stand-alone devices, or as applications on smart phones.

So what is next for PNDs? In a driving simulator study to be presented at MobileHCI 2011 [1], Zeljko Medenica, Tim Paek, Oskar Palinko and I explored two ideas:

Augmented reality PND: An augmented reality PND overlays route guidance on the real world using a head-up display. Our version is simulated and we simply project the route guidance on the simulator screens along with the driving simulation images. Augmented reality PNDs are not yet available commercially for cars.
Street-view PND: This PND uses a simplified version of augmented reality. It overlays route guidance on a sequence of still images of the road. The images and overlay are displayed on a head-down display. Google Maps Navigation runs on smart phones and can be used with street view.

The following video demonstrates the two PNDs.

Our findings indicate that augmented reality PNDs allow for excellent visual attention to the road ahead and excellent driving performance. In contrast, street-view PNDs can have a detrimental effect on both. Thus, while further research is clearly needed, it might be best if navigation with a street view PND was handled by a passenger and not by the driver.

References

[1] Zeljko Medenica, Andrew L. Kun, Tim Paek, Oskar Palinko, “Augmented Reality vs. Street Views: A Driving Simulator Study Comparing Two Emerging Navigation Aids,” to appear at MobileHCI 2011

Towards disambiguating the effects of cognitive load and light on pupil diameter

Light intensity affects pupil diameter: the pupil contracts in bright environments and it dilates in the dark. Interestingly, cognitive load also affects pupil diameter, with the pupil dilating in response to increased cognitive load. This effect is called the task evoked pupillary response (TEPR) [1]. Thus, changes in pupil diameter are physiological measures of cognitive load; however changes in lighting introduce noise into the estimate.

Last week Oskar Palinko gave a talk at Driving Assessment 2011 introducing our work on disambiguating the effects of cognitive load and light on pupil diameter in driving simulator studies [2]. We hypothesized that we can simply subtract the effect of lighting on pupil diameter from the combined effect of light and cognitive load and produce an estimate of cognitive load only. We tested the hypothesis through an experiment in which participants were given three tasks:

Cognitive task with varying cognitive load and constant lighting. This task was adapted from the work of Klingner et al. [3]. Participants listened to a voice counting from 1 to 18 repeatedly. Participants were told that every sixth number (6, 12, and 18) might be out of order and were instructed to push a button if they detected an out-of-order number. This task induced increased cognitive load at every sixth number as participants focused on the counting sequence. A new number was read every 1.5 seconds, thus cognitive load (and pupil diameter) increased every 6 x 1.5 sec = 9 seconds.
Visual task with constant cognitive load (assuming no daydreaming!) and varying lighting. Participants were instructed to follow a visual target which switched location between a white, a gray and a black truck. The light reaching the participant’s eye varied as the participant’s gaze moved from one truck to another. Participants held their gaze on a truck for 9 seconds, allowing the pupil diameter ample time to settle.
Combined task with varying cognitive load and lighting. Participants completed the cognitive and visual tasks in parallel. We synchronized the cognitive and visual tasks such that increases in cognitive load occurred after the pupil diameter stabilized in response to moving the gaze between trucks. Synchronization was straightforward as the cognitive task was periodic with 9 seconds and in the visual task lighting intensity also changed every 9 seconds.

Our results confirm that, at least in this simple case, our hypothesis holds and we can indeed detect changes in cognitive load under varying lighting conditions. We are planning to extend this work by introducing scenarios in which participants drive in realistic simulated environments. Under such scenarios gaze angles, and thus the amount of light reaching participants’ eyes, will change rapidly, making the disambiguation more complex, and of course more useful.

References

[1] Jackson Beatty, “Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources,” Psychological Bulletin, 276-292, 91(2)

[2] Oskar Palinko, Andrew L. Kun, “Exploring the Influence of Light and Cognitive Load on Pupil Diameter in Driving Simulator Studies,” Driving Assessment 2011

[3] Jeff Klingner, Rashit Kumar, Pat Hanrahan, “Measuring the Task-Evoked Pupillary Response with a Remote Eye Tracker,” ETRA 2008

Zeljko Medenica advances to candidacy

Last week my PhD student Zeljko Medenica advanced to candidacy. Zeljko plans to create a driving performance measure that would be sensitive to short-lived and/or infrequent degradations in driving performance. In previous driving simulator-based studies [1, 2] we found that glancing away from the road is correlated with worse driving performance. Importantly, this is true even when performance averages over the length of the entire experiment are not affected. Thus, Zeljko plans to explore the use of cross-correlation in creating a new, highly sensitive driving performance measure.

Zeljko’s PhD committee includes Paul Green (UMTRI), Tim Paek (Microsoft Research), Nicholas Kirsch (UNH) and Tom Miller (UNH). Thanks to all for serving!

References

[1] Andrew L. Kun, Tim Paek, Zeljko Medenica, Nemanja Memarovic, Oskar Palinko, “Glancing at Personal Navigation Devices Can Affect Driving: Experimental Results and Design Implications,” Automotive UI 2009

[2] Zeljko Medenica, Andrew L. Kun, Tim Paek, Oskar Palinko, “Augmented Reality vs. Street Views: A Driving Simulator Study Comparing Two Emerging Navigation Aids,” to appear at MobileHCI 2011

Presentation at the 2011 Emergency Responders Workshop

Yesterday I participated in the work of the 2011 Emergency Responders Workshop (pdf) organized by WisDOT, CVTA and GLTEI. The workshop had two major goals. One was to provide a sampling of state-of-the-art technologies used by emergency responders. The other was to begin charting a path toward developing advanced technologies. Participants from emergency responder agencies, industry and academia discussed their vision for future technologies as well as barriers to progress.

My presentation focused on pervasive (or ubiquitous) computing for law enforcement. I encouraged participants to ask the following question:

“What should be the focus of R&D efforts targeting percom technologies for emergency responders?”

CVTA President Scott McCormick (in picture below) and WisDOT’s John Corbin led the meeting superbly – thanks to both for including me in this effort.

For more pictures from the event visit Flickr.

2011 opportunities for UNH CS students: multi-touch surface interaction

I am seeking UNH CS students (individuals or teams) interested in developing a user interface on a multi-touch table. The interface would allow a human operator to control a fleet of unmanned aerial vehicles (UAVs). This project will part of a collaborative effort with WPI on creating a fleet of UAVs. Students at WPI will focus on building the UAVs. Students at UNH will work on communication issues (with Professor Nicholas Kirsch) and on user interface issues (with me).

What should the user interface do?

The operator should be able to view and manipulate data sent out by the UAV fleet. Data types of interest include images, video, sounds and outputs from various sensors (temperature, pressure, accelerometers, etc.). Data manipulation will require some simple processing, such as setting beginning and end points for sounds, zooming images, etc. It will also require more complex processing of data, e.g. filtering.

What are the data sources?

Eventually, the data will come from UAVs. However, as a first step, data will be generated through games, similarly to work done by Jatin Matani and Trupti Telang. Thus, we might utilize cell phones to get images, webcams to get video, and Arduino boards to generate sensor data (e.g. temperature).

What platform will be used?

The project will leverage Project54’s Microsoft Surface multi-touch table. Here is a video by UNH ECE graduate student Tim April introducing some of the interactions he has explored with the Surface.

Is this a job, a project, or something else?

CS students would be able to use this effort as a senior project (details to be worked out with appropriate CS faculty). An independent study might also be a possiblity. Finally, I am interested in hiring students for academic year and/or summer jobs.

Can CS and ECE students collaborate?

Collaboration is not a requirement. However, some aspects of this work might benefit from the involvement of one or more UNH ECE students. E.g. ECE students can work on some of the data processing aspects of the projects, as well as on creating data sources (e.g. deployment of wireless sensor networks). I am actively recruiting ECE students for multi-touch projects and you are welcome to talk to your friends in ECE.

What are the required skills? And what new skills will I acquire?

For CS students, work on this project will require object-oriented programming that is necessary to control the multi-touch table. You will explore the application of these skills to the design of surface user interfaces as well as experiments with human subjects – after all we will have to systematically test your creation!

Interested? Have questions, ideas, suggestions?
Email me.

2011 Senior Project topics: multi-touch surface interaction

I am seeking students (individuals or teams) for two senior projects. Both projects would leverage a multi-touch surface to create a natural user interface for pervasive computing applications.

Pervasive computing problems and ideas are often introduced using videos. An excellent exampe is the Microsoft Health Future Vision video (download, watch on YouTube).

Let’s focus on three themes from the video that are relevant to the senior projects: interactions with multi-touch interfaces, interactions with tangible user interfaces, and data manipulation/fusion. Multi-touch surfaces appear throughout the video: in Sabine’s home, in the doctor’s office, and in the hospital lobby. Several of the multi-touch interfaces, such as Sabine’s remote control, and her virtual wallet (used in the lobby), are tangible interfaces. Finally, Dr. Kemp manipulates/fuses data when interacting with Alex (patient in bed) and especially during the meeting with Sabine and Wei Yu.

The two senior projects will leverage Project54’s Microsoft Surface multi-touch table. Here is a video by UNH ECE graduate student Tim April introducing some of the interactions he has explored with the Surface.

With all this in mind, here are the specifics on the two proposed projects.

Project 1: Mobile data fusion

This project will explore fusing data, such as images, video, sounds and outputs from various sensors (temperature, pressure, accelerometers, etc.). Data fusion will require some simple processing, such as setting beginning and end points for sounds, zooming images, etc. It will also require more complex digital signal processing of data, e.g. windowing and filtering (topics covered in ECE 714). Consequently, work on this project will focus on data processing as well as object-oriented programming that is necessary to control the multi-touch table.

This project will be tied to a collaborative effort with WPI on creating a fleet of UAVs. Thus, eventually, the data to process and display on the multi-touch will come from the UAVs. However, as a first step, data will be generated through games, similarly to work done by Jatin Matani and Trupti Telang.

Project 2: IR wallet

The Microsoft Surface uses infrared illumination and cameras to recognize interactions with its surface. It can also recognize 2D barcodes if they are visible in the IR part of the spectrum. The “IR wallet” project would result in a tangible user interface, similar to Sabine’s virtual wallet, that can display 2D barcodes in IR. These in turn will be picked up by the Microsoft Surface. Work on this project will focus on microcontroller-based design (e.g. with an Arduino board) and object-oriented programming for the Surface.

Interested? Have questions, ideas, suggestions? Email me.

Bryan Reimer visit to UNH

It was my great pleasure to host Bryan Reimer at UNH. Bryan is Research Scientist at the MIT Age Lab as well as Associate Director of the New England University Transportation Center. His research focuses on the measurement and understanding of human behavior in dynamic environments, such as in cars.

Bryan spent time in the Project54 lab discussing various aspects of driving simulator and field studies. He then gave a thought-provoking talk reviewing results from multiple studies exploring driver workload and distraction. I expecially enjoyed his discussion of physiological measures that can be used to estimate workload. E.g. Bryan has found that heart rate is a robust estimate of workload and is often more useful than the often-used measure of heart rate variability. Bryan also discussed work on validating driving simulator results through field studies. His data indicate that driving simulator results can be used to predict relative changes in workload measures under different situations in real-life driving. However, the actual values of the measures collected in simulator and field studies often differ significantly.

For more pictures visit Flickr.

2011 CEPS Study Abroad Information Session

UNH’s College of Engineering and Physical Sciences (CEPS) has several exchange programs, one of which allows our students to study a semester at the Budapest University of Technology and Economics (BUTE). On Saturday (3/5/11) I had a chance to introduce the ECE perspective of the Budapest exchange program to an impressively large (80+) group of students and parents. My presentation was part of the CEPS Study Abroad Information Session organized by Bob Henry, CEPS associate dean for academic affairs, and his staff.

My favorite part of the information session was the student panel. Bob Henry invited five students who spent time abroad (three in Budapest and two at Heriot-Watt University in Scotland) last fall to answer questions from the audience. As in panels I’ve seen in the past, the students gave the exchange programs glowing recommendations. However, this year Bob Henry added something I haven’t seen in the past: the parents of a student panelist were on hand to answer questions from their perspective. This strikes me as an excellent way to make the parents of prospective study-abroad students feel comfortable with the program, and prepare them for the inevitable stress of their children moving to another continent for a few months.

Several UNH ECE students were at the information session and I hope they take advantage of the opportunity to study in Budapest. Carol Perkins and four other ECE juniors are currently in Budapest and, according to Carol, they are adjusting very well.

For more pictures from the event visit Flickr.

Pervasive Computing and Communications Group at UNH

Three UNH ECE faculty, Nicholas Kirsch, Tom Miller and me, have formed the Pervasive Computing and Communications Group. Our group investigates a wide range of topics from human-computer interactions to the physical networks that make sharing information effortless.

Nick joined UNH ECE in the fall of 2010. His interest is in spectrally efficient wireless communications systems utilizing software defined radios. Tom and I have been involved with the Project54 system since 1999. Tom’s interest is in embedded system design and application development and on human-computer interaction with mobile devices. My primary interest is in human-computer interaction for mobile devices and for desktop multi-touch devices.

PhD and MS position at the University of New Hampshire exploring in-car human-computer interaction

A PhD and an MS position are available in the Project54 lab at the University of New Hampshire. The lab is part of the Electrical and Computer Engineering department at UNH. Successful applicants will explore human-computer interaction in vehicles.

The Project54 lab was created in 1999 in partnership with the New Hampshire Department of Safety to improve technology for New Hampshire law enforcement. Project54’s in-car system integrates electronic devices in police cruisers into a single voice-activated system. Project54 also integrates cruisers into agency-wide communication networks. The Project54 system has been deployed in over 1000 vehicles in New Hampshire in over 180 state and local law enforcement agencies.

Research focus

Both the PhD and the MS student will focus on the relationship between various in-car user interface characteristics and the cognitive load of interacting with these interfaces, with the goal of designing interfaces that do not significantly increase driver workload. Work will involve developing techniques to estimate cognitive load using performance measures (such as the variance of lane position), physiological measures (such as changes in pupil diameter) and subjective measures (such as the NASA-TLX questionnaire).

The work will utilize experiments in Project54’s world-class driving simulator laboratory which is equipped with two research driving simulators, three eye trackers and a physiological data logger. Laboratory experiments will be complemented by field deployments in law enforcement agencies such as the New Hampshire State Police, which operates over 300 police cruisers. Project54 has deployed a state-wide data update infrastructure for the New Hampshire State Police which allows remote updates to in-car experimental software and remote collection of experimental data.

Appointment

The PhD student will be appointed for four years, and the MS student for two years. Initial appointments will be for one year, starting between June and September 2011. Continuation of funding will be dependent on satisfactory performance. Appointments will be a combination of research and teaching assistantships. Compensation will include tuition, fees, health insurance and academic year and summer stipend.

How to apply

For application instructions, and for general information, email Andrew Kun, Project54 Principal Investigator at andrew.kun@unh.edu. Please attach a current CV.