Testing OK/Cancel buttons

Question

Related: OK/Cancel on left/right?

I have been looking in to whether OK/Cancel should be on left or right, and after looking at lots of views on each side it seems there are good reasons for both. I don't want to start talking about that here, as it has already been covered in great detail in the question above.

As I am trying to make the UX the best it can be, all I really CARE about is the user, and I want to place the buttons wherever I need to so that it's as usable as possible. With this in mind I'd like to perform some UX tests.

What kind of tests should be performed? Shall I blank the buttons and take a poll on which button should be which? Should I remove the Red/Green colouring and ask users to perform the tasks with haste and see which generates more errors?

I'm open to suggestions on the best way to test this and I hope the results can help us all!

Answer 1

Don’t do a poll. What people say they like doesn’t always correspond to what they do. If you’re going to take the blank-button approach, put users in a situation where they need to click OK (and another situation where they need to click Cancel) and force them to guess which one to click.

It’s probably better to have a more realistic UI which has the labels. However, I wouldn’t rely on measuring error rates alone. The truth is users very rarely click the wrong button in a dialog of a desktop app no matter which button is left and right. That’s one reason debate on this topic has yet to be resolved –one way doesn’t blatantly out-perform the other. In addition to measuring error rates, also measure response time to the millisecond. If users are the least bit confused by the button positions, it will show up as a slight hesitation. This saves you from having to unrealistically rush your users (but it’s okay to provide realistic pressure). Also, efficient user performance is its own usability benefit apart from accuracy.

For the test, try to come up with a task where the choice between OK and Cancel is naturalistic but you can tell definitively when the user is right or wrong (just in case there are some accuracy issues). For example, you could have a spell-check feature that goes through a body of text and sequentially presents via a dialog box each word that may need correction. By design, some words need to be corrected (user should select OK) and some should not (user should select Cancel). Correct and incorrect spellings should be obvious to anyone with reasonably good education in English in order to minimize variance.

Try to recruit users with various levels of Apple OSX versus other platform experience, and record each user’s experience with Apple and others (how often and how recently), and use that information as covariates in your analysis.

Analyze your results with inferential statistics. If no one on the team is comfortable doing inferential statistics, then get someone who is. Because button position may not make much difference, select a difference in accuracy and response time that is worth caring about and determine the sample size you’ll need to reliably detect that difference by performing a statistical power analysis using the variance from a few pilot-test users. Be prepared to run a lot of users.

Publish your results, preferably in a peer-review journal. I think we’d all want to know the answer.

Answer 2

Do you have to put 'OK' and 'Cancel' on the buttons?

One of the problems with OK/Cancel in dialogs (and similarly, but worse, Yes/No) is that the user has to refer back to the original question to understand what the buttons will actually do. This is probably more of an issue than whether the OK or Cancel is on the left or the right. For example:

• Are you sure you want to discard your changes? [OK / Cancel]
• Are you sure you want to save and overwrite this file? [OK / Cancel]

In these examples the meanings of 'OK' and 'Cancel' are more or less the opposite of each other.

Even worse it the occasionally seen negative question:

• Are you sure you don't want to save your changes? [Yes / No]

A better solution is to but the verb for the action that will actually performed on at least one of the buttons, if not both. For example:

[Save / Cancel], [Delete / Cancel], [Assign / Cancel]

When the 'second' option is more than just cancelling the first, I'd be inclined to label both buttons in this way. For example:

• You haven't titled your email. Do you want to send it anyway? [Send / Keep Editing]

Now the options will generally make sense on their own terms, without reference to the original question.

Answer 3

I would perform a user test, and see if the user encounters any problems at the stage they work with the dialog. There are other research methods, but they may not be reliable.

Directly asking a user where they'd expect to see something is fraught with danger. Users are not usually conscious of the decisions they make traversing an interface and their responses could be subject to cultural and linguistic preferences (the answer 'OK/Cancel' looks more "correct" in text than 'Cancel/OK' because it's more common, and typical convention to put longer words towards the end of English constructions).

A/B testing can work, but only if you know what you're measuring against. Did users click 'cancel' more on the right because that's where they were expecting it? Or did they actually expect an 'OK' button and you just made them lose their work? You can track time to completion (so you can see evidence of users being confused), but at this scale of design, the times involved might be slight and seeing a statistically significant variation would require large sample sizes.

Ultimately, I think there's no real substitute to speaking with real users in a test environment.

Answer 4

A good argument for placing the "Ok" button on the right is made over at UX Movement, why ok buttons in dialog boxes work best on the right.