Mastodon Visualization Decision Making Laboratory
VDM
lab

Toward Objective Evaluation of Working Memory in Visualizations: A Case Study Using Pupillometry and a Dual-Task Paradigm


Cognitive science has established widely used and validated procedures for evaluating working memory in numerous applied domains, but surprisingly few studies have employed these methodologies to assess claims about the impacts of visualizations on working memory. The lack of information visualization research that uses validated procedures for measuring working memory may be due, in part, to the absence of cross-domain methodological guidance tailored explicitly to the unique needs of visualization research. This paper presents a set of clear, practical, and empirically validated methods for evaluating working memory during visualization tasks and provides readers with guidance in selecting an appropriate working memory evaluation paradigm. As a cas study, we illustrate multiple methods for evaluating working memory in a visual-spatial aggregation task with geospatial data. The results show that the use of dual-task experimental designs (simultaneous performance of several tasks compared to single-task performance) and pupil dilation can reveal working memory demands associated with task difficulty and dual-tasking. In a dual-task experimental design, measures of task completion times and pupillometry revealed the working memory demands associated with both task difficulty and dual-tasking. Pupillometry demonstrated that participants’ pupils were significantly larger when they were completing a more difficult task and when multitasking. We propose that researchers interested in the relative differences in workin memory between visualizations should consider a converging methods approach, where physiological measures and behavioral measures of working memory are employed to generate a rich evaluation of visualization effort.

Paper Info

Padilla et al. 2019 download

Preprint

  • Free PDF
OSF

Open Science Framework

  • Supplementary Materials
APA Citation

Padilla, L., Castro, S., Quinan, P., Ruginski, I. & Creem-Regehr, S. (2019, October). Toward Objective Evaluation of Working Memory in Visualizations: A Case Study Using Pupillometry and a Dual-Task Paradigm. IEEE transactions on visualization and computer graphics.

Bibtex

@article{2019_vis_pupillometry, title = {Toward Objective Evaluation of Working Memory in Visualizations: A Case Study Using Pupillometry and a Dual-Task Paradigm},
author = {Lace Padilla and Spencer Castro and P. Samuel Quinan and Ian Ruginski and Sarah H. Creem-Regehr},
journal = {IEEE transactions on visualization and computer graphics},
year = {2019} }

Links

Medium

Blog post on this topic

  • How do we know when a visualization is good? Perspectives from a cognitive scientist