Are there any studies of users getting jaded of dialog requests?

Question

There are permission request boxes such as this one, or requests from other users as per online social networking sites. Are there any studies that show people becoming jaded of dialogue boxes of any sort?

Answer 1

Yes, there is extensive research on this question available in the literature on usable security, and they have found that users do indeed become jaded. This is an important challenge for making security usable: sometimes developers think, if in doubt, just throw in another warning dialog; but the research literature shows that this is a bad idea.

References. Here are some examples of work in the literature:

• S. Motiee, K. Hawkey, and K. Beznosov. Do Windows Users Follow the Principle of Least Privilege? Investigating User Account Control Practices. Symposium on Usable Privacy and Security (SOUPS), 2010.

  • Shows that many (about half of) users will click through a bogus Windows UAC dialog box, even when it is illegitimate and triggered by malware.

• N. Good, R. Dhamija, J. Grossklags, S. Aronovitz, D. Thaw, D. Mulligan, and J. Konstan. Stopping spyware at the gate: A user study of privacy, notice and spyware. Symposium On Usable Privacy and Security (SOUPS), 2005.

  • Finds that users will often ignore or click through warning dialogs that interrupt users from their primary task.

• R. Bohme and S. Kopsell. Trained to accept? A field experiment on consent dialogs. CHI, 2010.

  • Shows that users who repeatedly see a warning dialog box will pay less and less attention to it, the more often they see it. This effect carries over to other similar warning dialog boxes.

• R. Bohme and J. Grossklags. The Security Cost of Cheap User Interaction. New Security Paradigms Workshop (NSPW), 2011.

  • Explains the problems with users becoming "jaded". The technical term is habituation: users become habituated to the warning dialog boxes and start paying less attention to them, the more often they see them. This is a fundamental phenomenom in usable security. This paper suggests that we think about human attention as a scarce resource, which must be conserved. In other words, system designers should realize that they have a very limited "attention budget" from users, and they must carefully choose how they will spend this modest budget, to avoid triggering habituation effects.

• J. Sunshine, S. Egelman, H. Almuhimedi, N. Atri, and L. F. Cranor. Crying Wolf: An Empirical Study of SSL Warning Effectiveness. USENIX Security Symposium, 2009.

  • Documents habituation effects in browser warning dialogs related to SSL certificate errors. The title alludes to the problem: if the browser keeps crying wolf, and nothing bad seems to happen, then users stop paying attention to the browser's warnings. The consequence for system design is that we must be careful to avoid crying wolf.

• S. Egelman, L. F. Cranor, and J. Hong. You’ve been warned: An empirical study of the effectiveness of web browser phishing warnings. CHI 2008.

  • Documents habituation associated with browser phishing warnings. This just adds to the extensive evidence that this problem is fundamental and applies in a broad variety of settings where runtime warnings are used.

• L. Cranor. A Framework for Reasoning About the Human in the Loop. Usability, Psychology and Security 2008.

  • Provides a theoretical framework for system designers, for how to think about things in a way that will help you avoid the problem -- or at least understand the problem in a systematic way, so you can think through the consequences of various design options. This is a lovely piece of work, which builds upon Wogalter's Communication-Human Information Processing (C-HIP) model from warning science (e.g., how to design warnings for nuclear power plant operators or on road signs) as well as a deep understanding of human behavior.

Answer 2

Microsoft conducted a lot of testing into the User Account Control (UAC) between Windows Vista (when it was introduced) and Windows 7. They have a post about this on their msdn blogs:

We have learned from our customers participating in the Customer Experience Improvement Program, Windows Feedback Panel, user surveys, user in field testing, and in house usability testing that the benefit of the information provided by the UAC consent dialog decreases substantially as the number of notifications increases. So for the general population, we know we have to present only key information to avoid the reflex to “answer yes”.

^{(Emphasis mine)}

The article goes on to discuss some of their AB testing that they went through when making decisions on how to handle permissions dialog messages in Windows 7.

In making our choice for the default setting for the Windows 7 beta we monitored the behavior of two groups of regular people running the M3 build. Half were set to “Notify me only when…” and half to “Always Notify.” We analyzed the results and attitudes of these people to inform our choice. This study, along with our data from the Customer Experience Improvement Program, Windows Feedback Panel, user surveys, and in house usability testing, informed our choice for the beta, and informed the way we want to use telemetry from the beta to validate our final choice for the setting.

A key metric that came out of the study was the threshold of two prompts during a session. (A session is the time from power up to power down, or a day, whichever is shorter.) If people see more than two prompts in a session they feel that the prompts are irritating and interfering with their use of the computer. In comparing the two groups we found that the group with the “Always Notify” setting was nearly four times as likely to have sessions with more than two prompts (a 1 in 6.7 chance vs a 1 in 24 chance). We gathered the statistic for how many people in the sample had malware make it onto their machine (as measured by defender cleaning) and found there was no meaningful difference in malware infestation rates between the two groups. We will continue to collect data during the beta to see if these results hold true in a much broader study.

^{(Emphasis mine)}

Unfortunately they haven't published their test results (as far as I can see), so as with any study that references data that isn't published may need to be taken with a grain of salt. However the article (and the links to previous articles mentioned within that post) make very interesting reading.

Answer 3

Dialog boxes are needed. In the old HCI thinking, a user interface is essentially a dialog between a human, and a computer.

However, handling of the dialog boxes can become habitual. The typical example is Word or any similar document editor: when you close an unsaved document, usually, you don't want to save it. In those exceptional times when you actually want to save it, you're prone to click "don't save" anyway.

Therefore, it is a recommended best practice to avoid opening dialog boxes often. Provide forgiveness instead of asking for confirmation: provide a way so that the user can undo. If you look at document editors on iOS (Pages, or OmniOutliner), they don't have confirmation dialogs at all, and I guess the Lion version of Pages doesn't have it either.

Also, the users are sometimes not able to comprehend the situation in question. Like, in this case, I bet an average user doesn't really know, what "Microsoft Windows Component Publisher" means. There is no meaningful action in this case from the view of the user.

In other cases, the dialog box doesn't tell the user what can he do, what would be the result of each of the choices, and how would the user benefit from those possible futures. Always provide the right context, and make the options always clear from the users' perspective.

The users are always trying to achieve a goal. If the goal is to remove an application, they do whatever it takes to remove the application, even if a dialog box tells them it also removes their whole windows installation, publishes their nasty pictures on the internet, and sell their soul on eBay.

Together with this, and the habituality of choosing a default option, it's rather discouraged to do dialog boxes most of the time.

I'll grab the studies for you, and edit this answer in a moment, just couldn't find it at Nielsen (useit.com - albeit I'm sure it's there), but I have a few UX books lying around here.

Edit: Unfortunately, I didn't find the usual psychologic journal stuff about this,as this has been a known fact since the eighties, and even writers from the nineties were too lazy to cite these studies correctly.

• A case study from the Nielsen-Norman group is here
• General guidelines for error pages from Nielsen is here, and there is a course where they tell you about Dialogue Design
• Raskin was famously a big enemy of dialog boxes (look at the Canon Cat), and he describes these problems in The Humane Interface, Chapter 2-3: Locus of Attention
  • He also cites Baars, Bernard J. A Cognitive Theory of Consciousness (Cambridge, U.K: Cambridge University Press, 1988) and Card, Stuart K., Thomas P. Moran, and Allen Newell . The Psychology of Human-Computer Interaction (Hillsdale, NJ: Lawrence Erlbaum Associates, 1983), although I have no idea what these writings contain, and I have no access to them
• Chapter Five of Norman's classic Design of Everyday Things deals with errors, but it does deal with it in the physical environment, as the whole book tries to use the physical environment as a metaphor for the virtual one.
• Jeff Johnson talks about interface issues in Designing with the Mind in Mind, for example in Chapter 6, _Pop-up message in error dialog box", but he doesn't tell about habits.

I could also quote the Apple Human Interface Guidelines (or iOS guidelines) and perhaps also the Windows Interface Guidelines, but they are about practical issues. Also I grabbed two other UX books and I could grab a 90s edition of SEPA (Software Engineering: A Practitioners' Approach), where I remember there was the 10 commandment of user interfaces, that was an often-quoted UI rule-of-thumb guideline, but I don't think I'd find its reference.

All in all, it seems this is something that we understood for 30 years, and everyone is just too lazy to back up their claim. Perhaps it is this way because everyone knows from their private life how easy is to make such mistakes, and it is the most easily tested thing in experimental settings.

As for what to do instead of error boxes, read Raskins' book, or Norman's book. As for looking at how an alert-less application flow looks like,look at Google Docs or Lion / iOS 5 versions of Pages.

Sorry, I couldn't find it, it seems even I just remember that this is how it should be done and these are the reasons, but I can't find in my library where did it actually come from.