Tag Info

In addition to the HEART metrics (Happiness, Engagement, Adoption, Retention, Task Success) we also measure 1) new user adoption by whether or not training costs decreased and 2) existing user adoption by whether help calls decreased. I'll try to answer your follow up question and to do so you must first understand the personality of the user's you're ...

In Why You Only Need to Test with 5 Users Jakob Nielsen suggests: The best results come from testing no more than 5 users and running as many small tests as you can afford. However, rather than focusing on the number of users, it might be better to focus on the number and quality of the tasks: Usability test tasks are so critical that some people ...

Check out Skype's Screen Sharing feature (it's free). I'm planning on using Skype for this very purpose in a couple months. I'll be writing a "Skype Screen Sharing with Steve" blog post to try to encourage participation. I'm thinking the invitation should be up front about the fact that you're looking for feedback and are curious about uncovering ...

Read Steve Krug's Rocket Surgery Made Easy. It explains in plain English how you can do usability testing on a shoestring budget (or for free!) by using a common sense approach. It's been known for years that usability testing can dramatically improve products. But with a typical price tag of $5,000 to $10,000 for a usability consultant to ...

I had this problem as well a short while ago. What I did was write out a list of tasks and have participants record a screencast of themselves doing them, while thinking out loud. Later on when you watch the screencast, if there are parts that you would want them to re-do, you could ask them to do a second round. What is the advantage for you in ...

A few possibilities come to mind: Heuristic analyses (requires 5-6 fellow HCI practitioners in order to be comprehensive) Validation against personas or behavioural goals (may not be reliable) GOMS / KLM analyses - basically your first bullet point (reliable, but doesn't uncover issues with IA or broken interface metaphors) Comparative assessment ...

There are no absolute 'fundamental' metrics. All metrics should be decided on a per-project basis, and be grounded in explicit business objectives. That way, designers will have explicit priorities and clients will understand the compromises we make to achieve them. So, for example, to look at some items of your list: What web browsers are most commonly ...

It's time to conduct a competitive heuristic evaluation. Your question has described most of the solution, so I'll talk about how you put those together. First, if you haven't yet, write up the main use cases for your IDE. If your use cases are too broad ("an experienced developer will maintain ~100k lines of Python code"), then you add in additional ...

I don't have a reference for this, but I would think that these numbers would depend on the size of the user base. For qualitative testing you need to have "typical" users. So if you have 3 roles your users can take you need at least 3 users - one for each role. In reality you'd want more than one, but this is your absolute minimum. For quantitative ...

Something else to consider is where the result of your usability report is going. How much work can the people downstream from you fix - and what will the effect of those fixes be? Say I run a test with fifteen people. After the first three I've spotted problems A B C. By the end of the fifteen I've also spotted problems D E F G. The problem is that the ...

It depends what your goals are, and how you assess the different risks in what you're testing. Some things to consider: How do your usage patterns break down? Are the majority of your users coming in on mobile devices? Then you should probably focus more of your testing resources on that breakpoint. You could keep the same number of participants - but run ...

You still can do it with the same number of participants as you would do with a single test -- it just takes longer to test each user because you're showing them more material (i.e. three breakpoints instead of just one). However, you should "shuffle" and randomize the order in which you test each breakpoint so you don't skew the results due to familiarity ...

It depends on your specific case. The key factors are how strongly qualified your incoming traffic is, what the risk to value proposition is, and what obstacles occur in your conversion funnel. Attrition rates upwards of 70% are probably normal - sometimes per step in your funnel. You're going to lose a bunch to gotchas which the user didn't see before ...

I recently read an interview with Jonathan Ive (Head Apple designer) and a few paragraphs struck me as cutting to the heart of this issue: Q: Do consumers really care about good design? A: One of the things we’ve really learnt over the last 20 years is that while people would often struggle to articulate why they like something - as consumers we ...

Actually you are using humans for the evaluation of your novel UI, ain't you one? :-) I'm sure there are a few more humans with you that might be helpful for this task. The problem with you and the other humans around is that you know things about your UI that would slant the results of a naive user test. To overcome the IRB restrictions and the slant thing ...

Ethnography has been 're-calibrated' for Computer-Mediated Communication and related complex cultural phenomena, because anthropology, sociology and psychology (to name some) have never stop analysing them. Technologically mediated environments prevent researchers from directly observing research participants. The online environment requires ...

In academic (computer science) circles this is known as affective computing (link to Wikipedia article). From a computer science perspective, the challenge is first to characterise the user's behaviour/emotion while they are experiencing/using the system, and then classify this feature vector (a list of numbers describing the behaviour) to belong to a ...

My intuition says that you should have separate tests for the different break points. If we only consider two scenarios (small-mobile, large-desktop) then you are targeting two distinct input devices (touch vs mouse). You may find that your tests look different for touch devices, than for desktop devices (the tasks may differ, or priorities change depending ...

A few other question ideas: How frequently users come to the site (several times a day?). Once a week? What percentage of users contribute the content? Do contributors usually contribute to one topic or is there a lot of variety? (e.g. Jack adds comments only to posts related to environment, Kate comments to posts about cognitive science and gardening). ...

For quantitative testing, it's possible to be more explicit about the effect of the sample size on your results but the number of users you need depends on the particular tests or analyses you are considering (examples could be determining the proportion of participants successfully completing a task, estimating the average time-on-task, comparing two ...