I'm trying to reconcile what I feel is a bit of a contradiction in LeanUX.

  • test, validate, and iterate early
  • reduce the documentation load

Does your organization user/usability test in an agile process? If so, when and how?

In our org, we do a lot of unmoderated remote testing. This requires a relatively complex, shareable prototype be created (typically Axure or Invision, sometimes HTML).

This isn't entirely bad, as in the process of creating the prototype, we do do a lot of internal iterating and design tweaking. However, it seems counter to bullet point 2...as these prototypes tend to be rather large documents in terms of creating time and effort (not to mention maintenance if they are also being used as wireframes for dev).

Question: Are those two bullet points contradictory or is the issue that I am interpreting them incorrectly?


An alternative question that may make more sense to answer: Does one normally unmoderated usability testing in an Lean UX process?

  • Unmoderated testing is the leanest way to get a lot of feedback ... Commented Dec 22, 2015 at 0:46
  • @plainclothes what kind of artifact do you usually test with unmoderated testing?
    – DA01
    Commented Dec 22, 2015 at 3:08
  • Clickable prototype, low fi HTML, or better if we have it. Just depends on the project. Commented Dec 22, 2015 at 3:31

2 Answers 2


TLDR: We ask this question "What's the smallest increment of work that can be tested?

We go and test that using the quickest solution that'll get us answers. Which means products like Axure and even Invision are often considered as too time consuming to use unless we're dealing with a heavy animation/interaction-based solution that is difficult to convey to people using static UI.

Breakdown of our process

We built our process off ideas from the book Inspired: How to Create Products Customers Love.

There are 4 phases

  1. Opportunity Assessment - Research on perceived user problem, go/no go decision
  2. Product Discovery - Iterative ideation and concept testing
  3. Product Execution - Outline stories and work with Devs for implementation
  4. Release - Staged (beta) or incremental release, get post-release feedback

Majority of the testing happens during the product discovery phase. We try to go for broad stroke checks at the beginning. So test group sizes are super small... 3-5. We may start off with a quick test with internal staff (support, on-boarding and sales guys). Then we schedule remote screen sharing sessions with users. (We deal with a complex software, it's usually easiest to pick up nuisances when you can hear the user.) Since we're there with the user, we frequently do a combo session with a mental model interview followed by A/B type concept tests.

The concept tests are as low fidelity as we can get away with. Typically this is just an opened omnigraffle file. I read out the question, click through pages of screens and collect their feedback directly on said file. You save a lot of time if you don't have to export and upload images, then tweak stuff in a separate tool. (Not saying tools like Axure and Invision don't have their place in our workflow. They do for stuff with more complex screen interactions. They're often not necessary.)

The most time consuming aspect of this is probably user recruitment. We're pretty lucky in having a very active user base who are super happy to help with testing. We then use a tool like youcanbook.me to indicate free blocks for testing and have our users sign up for the blocks.

I believe another term for this is dual-track scrum if you want to do a deeper dive into the area.

Update: About your question on unmoderated tests. We tried it in the past and stopped. Unmoderated tests require you to know a lot more about the question at hand. If the questions asked are completely off mark, the unmoderated test data will be completely wasted. With a moderated test, you have much more flexibility to adjust on the fly. Flip side with a moderated, you have to be there to run the test, which limits the number you can do. But, numbers aren't necessary for early testing, just the user's gut feelings.

  • Ah, so that's interesting. Sounds like your testing artifacts are essentially "screen sharing" and your data is really just feedback rather than a scripted test. Is that correct? Do you fit these tests into the same sprint as dev, or is this outside of sprint work?
    – DA01
    Commented Dec 21, 2015 at 21:53
  • That is correct. I have an evernote bullet list that acts as the intro script. The specific task-based questions are added as annotations directly on the omnigraffle file. Same goes for capturing of user feedback. We record the screen sharing session and notes goes into the graffle. Dual track has design operate several sprints ahead of the main dev sprint. You can say this is work outside of the main dev sprint.
    – nightning
    Commented Dec 21, 2015 at 22:00
  • I'm interested in the A/B concept tests. Do you literally ask the same question for each concept? Would that bias participants' feedback? What type of feedback are you getting after you show them a static image/page?
    – erik_lev
    Commented Dec 22, 2015 at 7:52
  • @lineplay I swap the presentation order for each user, but yes I ask the exact same question. Let say we have a resource forecasting report. I may ask the manager, there's a new piece of work that needs to be done. How would you determine who to assign it to? They would think aloud in attempting the task. The original view may not show additional details they need. When they bring it up, I prompt them for what info they would like on hand. Then if I have another screen prepped I'll show them that and ask them to continue with the task. I then repeat with concept B. Then rate A vs B.
    – nightning
    Commented Dec 22, 2015 at 19:04
  • @nightning I've done something similar; however, I only show participants one concept. I fear that their exposure to concept A will bias their response to concept B. I prefer interactive prototypes because it's hard to get a valid answer about which concept is easier to use based on how people think they could use, say, a static image.
    – erik_lev
    Commented Dec 23, 2015 at 6:43

The magic with lean UX is that you can test everytime you need. It means everytime you have a doubt in your development process.

You don't need to organize long and complex users testing sessions with a full complex prototype done in Axure. It is the lean UX force.

You can test just a specific feature, or interface with a sketched prototype done with free and simple apps (like Pop for example).

Invision's app provides good tools also like the Liveshare or the online comments wich can help you to make fast & short user tests.

Lean UX is a method which allowed failure. You have to see your sprints like experiences, so as in a science lab, you make assumptions, try, test, failed or success and then you go one.

  • I don't see anything specific here, though...which tends to be what I read elsewhere too. Do you do user testing the same sprint as development? How do you prep a prototype (even with something like Pop), schedule a test group, perform the moderated tests, and re-incorporate that in one sprint?
    – DA01
    Commented Dec 21, 2015 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.