Quantifiable benefits of wireframing and prototyping?

Question

I'm trying to come up with a few sources for quantifiable benefits of wireframing and prototyping, but I am having trouble finding results. I've seen plenty of "reduces requirements phase, improves quality, etc." but I am looking for studies with a few more numbers.

EDIT: I've updated the title of the question to better reflect what I'm asking. I'm not necessarily looking for a traditional ROI calculation of wireframing and prototyping, but I am looking for quantifiable benefits that are a direct result. Studies that can display, for example, "30% (arbitrary number) of reduced rework from a direct result of implementing wireframes and prototypes" is what I am in search of.

I am using some metrics from the "ROI of UX" question, such as "50% of developers time is spent in rework", which helps my case, but specific results directly from wireframing/prototying are desired.

EDIT 2:

I'm getting a lot of "advice answers" that are ultimately not the goal of this question. Most of us here at UX.SE understand prototypes and the general benefits. I am looking for quantifiable data, preferably derived from scientific analysis.

Here is an excerpt I found that is that is the type of information I'm looking for:

"In an experiment conducted at UCLA some development teams used conventional development methodologies while others employed prototypes in the software development process (with no particular emphasis on the interface). …Code of the final systems produced by prototyping groups was only about 40 percent as large as that of their counterparts, possibly at a cost in generality of design. Finally, the prototyping groups accomplished their task with 45 percent less effort than the other groups."

http://eprints.cs.vt.edu/archive/00000179/01/TR-89-42.pdf - page 5.

Edit 3:

Another great example of the information I am seeking. This is from the book "Prototyping: A Practitioner's Guide" by Todd Zaki Warfel:

Consulting company in the UK switched from requirements-oriented process to a prototyping-oriented process. The change in process resulted in the following benefits:

• Time and effort required to produce the prototype and 16-page supplemental document is less than half required for the 200-page specification document

• Estimates for build time and cost have become 50% more accurate

• Request for clarification by the development team has been reduced by 80%

• The amount of rework and bug fixes post-launch has been reduced to 25% of similar previous projects

Answer 1

The straight-out answer to that is pretty simple:

It's in the nature of a prototype to be beneficial.

If it's not beneficial, then you're doing it wrong...
If it's not beneficial, then it's not prototyping by definition...

When you start prototyping, you need to know why you are prototyping. You need to know what the benefits will be. If you don't know the purpose of the prototyping, then drop it. ... ... ... or read up on the topic ;-)

---

Updated second part of answer

I pointed you to Scott Overmyer's paper (from 2002), "Revolutionary vs. Evolutionary Rapid Prototyping: Balancing Software Productivity and HCI Design Concerns". This paper is cited as a reference to how prototyping will "Reduced time and costs" at Wikipedia.

The paper was unavailable before, but the author has now uploaded the paper on academia.edu.

In short, the paper is comparing revolutionary prototyping and evolutionary prototyping and advocates a hybrid solution of these methods (remember that the paper was written in 2002, so the state of prototyping approaches has developed since then).

Five case studies are analyzed (and the author emphasizes that it's hard to find such studies in the literature).

Case study #1: Revolutionary prototyping. The prototyping effort represented approximately 6% of the total software development effort of 10 person years. 5 iterations gave great insight to several aspects of the system. Delivery was a success.

Case study #2: An extremely large software development effort with 1200-1500 graphical and tabular displays. Data was collected from user trials and was used as input to a system simulation model, resulting in more accurate performance prediction for the overall system architecture.

Case study #3: A field study of 48 "Fortune 1000" companies. It concludes that revolutionary prototyping is the "silly empty shell" concept of information system development. Chicago bank's failure to successfully employ revolutionary prototyping for requirements definition is used as an example. In that case, end users spent 250 hours on average developing each of six revolutionary prototypes. The system developers spent between 75-225 hours for each prototyped application to build the operational system. This constitutes redundant effort of between 30 and 90 percent per prototype.

(Note: This case study was carries out in 1987. Before the Windows area and before Internet area. I'm not sure if these number are still valid.)

Case study #4: Evolutionary prototyping performed in 12 cycles of design, implementation and evaluation. Each cycle lasting about 2 weeks for a total of 2 years total project duration. During that time, 412 modules were produced from 32,745 lines of code taking 2613 hours (65 person-weeks) of effort. One interesting figure is that 12,957 lines of code were discarded during the effort, with approximately 1/6th of that amount discarded during the final optimization phase.

Case study #5: Experiment with students from 1984. The prototyping (and specifying) effort took place over a period of 11 weeks.

Conclusion drawn from these five cases:

It is asserted that revolutionary rapid prototyping is a more effective manner in which to deal with user requirements-related issues, and therefore a greater enhancement to software productivity overall.

Evolutionary prototyping can be used to address system performance issues, however, it is unclear whether this technique is superior to architectural modeling and simulation combined with man-in-the-loop simulation via rapid prototyping

For interactive systems it will likely benefit from revolutionary rapid prototyping for user requirements in combination with evolutionary prototyping for software requirements and system development.

Answer 2

I think the best way is to look at the amount of time and effort spent in sprint cycles with and without these assets, but it is very difficult to do a study on this because I don't think many organizations collect this type of information, or if they do then there is no particular reason to make it available to people. Personally you also have to weigh this up against the cost of creating and maintaining an extra set of assets.

Alternatively, you could look at the current effort/estimate for the development team, and speculate on the potential of doing more UX related work in reducing the development cost and also training efforts that are applicable for your organization. I think there has already been plenty written about the ROI of UX, but it is time that people start framing this in the context of their own organization.

Answer 3

Although the article below from J. Nielsen doesn't mention ROI, wireframing or prototyping, its conclusion is applicable to your question: 5 users or so help to discover ~80% of the usability issues.

Now it becomes a matter of product development process: testing the prototypes and iterating until the UX is right, before burning development resources, will increase the ROI of UX.

http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Following up on Edit 3

According to your example, you should take a look at product and project management key metrics.

Answer 4

I'm very curious as to why you are looking for this data.

I doubt that you'll ever find quantifiable results for ROI on wireframing/prototyping because the ROI for these activities is measured based on work that you don't do.

Let's take a simple example of creating a single web form or dialog box. In order to measure the ROI for creating a wireframe of this form or dialog, you'd have to do the following:

1. Create the wireframe/mockup
2. Decide that you don't like the wireframe/mockup
3. Go ahead and implement the form/dialog anyway, even though you know you don't want it
4. Measure the difference between development time for #3 and the time it took to do #1.

No one in their right mind would do this. I doubt there are any studies on this topic because the benefits are glaringly obvious to anyone with any real world experience.

Wireframing and prototyping is part of the software design process. It's no different than working with diagrams and blueprints before manufacturing a physical object. No one ever asks for the ROI on designing a physical object before manufacturing it :).

Answer 5

The quantifiable part would be the 'time and materials cost'.

To build something, it has to be prototyped. That could be a sketch on one end of the extreme, or manipulating the final materials on-site would be the other end.

So, for example, if I were to build a house I need to start with the foundation.

With a defined prototype process: napkin sketches of the property, some sketches of the house on said property. If nothing else was done, we at least have a pretty good idea of where the foundation should go.

Without a defined prototype process: Dig a hole, build foundation. Is this a good spot? No? Fill in the hole, dig a new hole, build another foundation. Is this a good spot?

Point being that to complete a product, this process has to happen. And less of that process that has to happen with the more costly final product/materials, the cheaper it will be.

To put it in code context, 3 quick wireframes on the whiteboard to help narrow the solution down to one option is going to be a lot cheaper than having 3 developers use actual code to come up with 3 different solution to compare.

Answer 6

The Nielsen Norman group should be able to help you with that. A quick search on their site returned these articles:

http://www.nngroup.com/articles/usability-roi-declining-but-still-strong/ http://www.nngroup.com/articles/when-high-cost-usability-makes-sense/

...among others. Return of investment in usability studies has a measurable impact, as does testing with prototypes. Would the combination of both data sets be enough to quantify a ROI of rapid prototyping, or are you also looking for opportunity cost data?

Answer 7

It seems to me that (even though it is not your intention) your question is essentially asking "what is the benefit of planning using wireframes/prototyping relative to anything else?" The problem with this question is that part of anything else will include both no planning and equivalents to wireframes/prototyping. There is no way to make a statistically (or financially) sound comparison in such a scenario.

In other words, you should be comparing x and y. I.e. wireframing vs no planning. I.e. wireframing vs flowcharts.

If the point of your question was to ask the benefit of planning in more general terms, you should look far beyond wireframing (as most other answers suggest). If you want to single out specific techniques like wireframing, you need to also define the alternatives to which you wish to compare them.

Answer 8

Am going off-topic to get some more valid reasoning for all the comments that has been mentioned here.

Nari Gandhi, an Architect is said so worked without an office or drawing - in similarity where it means he had used more of "Visual Scoping" or "Seamless adjustment" to guide his idea and pruned it with some good guesses (may be he drew smaller thumbnails or sketches to have a feeler), in that case it means the ROI (time in scheming things) of his products are seen as living artifacts - where a client is happy living in it, without ever having a traditional route of drawing practice (may be organic with his method - but had a final product) - The result! But ever so, Clients are always in dark without knowing what is going to shape up. Risky proposition.

"Nari worked without an office and rarely made any drawings for any of his projects. Nari spent a lot of time on his sites and worked closely with the craftsmen and often participated in the construction process himself" as extracted from Wikipedia.

If you got to superimpose this with UX practice - I could still stay just like how this Architect would team up with the workers, artisan and the site itself - the UX Guy had to collaborate with other team members.

This is really good if its Architecture, since you would see tangible outcomes - but in UX you can build direct coding but still had to depend on other human resources - more often since the product is intangible and lots of intricacy like speed, server, database and know-less technical knowledge slows down or practically impossible for a UX lead to run through it by himself.

All said - Wireframing or Mocking-up is a way to find result, but not in itself a result! You can chose not to follow this route and take another route where direct coding may find you still results as in case with Nari Gandhi.

I do not have a direct research point to your question, but a qualitative measure would certainly say that Time spent without a basic groundwork or proto is anybody's guess - which is Time consumption (Quantifiable) & lots of back-forth looping or ending up in a maze, unless ever its so collaborative effort.