There are some differences yes, and I have witnessed them myself. Using high fidelity mock-ups makes users think that the system they are using is fully functioning. That makes them think that system is not working properly and they rate usability to be less good.
Rudd, Stern and Isensee wrote on their article Low vs. high-fidelity prototyping debate:
When customers see a high-fidelity prototype, it often appears that the product is ready. If the prototype is much better than the product that they are using currently, they may demand it immediately.
On the other hand high fidelity prototypes can produce better feedback from the users.
In general, the user can get a feel for how the product will operate and, therefore, can make informed recommendations about how to improve its user interface.
Sauer and Sondegger studied how prototype fidelity and design aesthetics affect user behaviour, subjective evaluation and emotion in usability testing. In their paper they compared paper prototype, computer prototype and fully operational product. Product in their tests were two aesthetically different kinds of mobile phones.
For the attractiveness rating of the appliances, an interesting interaction between prototype fidelity and design aesthetics was observed. While there was no difference in ratings across different fidelity levels for the highly aesthetic mobile phone, the moderately aesthetic phone was rated lower on attractiveness for the original appliance than for the reduced fidelity prototypes.
[...]
This suggests that some compensatory activity on the part of the user took place since neither the paper prototype nor the computer-based prototype was aesthetically refined (e.g., lacking colour and shape of the reference appliance). Users may have mentally anticipated of what the real appliance might look like and employed this mental picture as a basis for their rating.
Prototype fidelity didn't seem to have statistical effect on subjective or objective usability but more appealing design was rated to have better usability.
The results showed no association between objective performance parameters and subjective usability evaluation. While there was a clear preference of users for the more aesthetic appliance because of higher attractiveness ratings and higher perceived usability, this was not paralleled by better objective usability of that appliance. This suggests that perceived usability may be more strongly associated with attractiveness ratings than objectively measured usability parameters.
Walker, Takayama and Landay studied differences between paper and computer prototypes – with high and low fidelity in both cases – in testing web prototypes. They found some differences between designs but the main find was that fidelity affects on the types of usability problems found.
Users made significantly more comments about computer than about paper prototypes but there were no differences in the number of usability issues. This suggests participants were more verbose on computer but the computers make it no easier to find usability problems. In practice, the additional comments from the computer condition may help interpret and solve usability problems.
[...]
The types of usability issues found were significantly different between low- and high-fidelity conditions. A limitation of the data analysis technique is that it does not allow us to attribute the difference to any particular type of issue.
Should you use low or high fidelity prototypes in testing. As always, it depends. If want to catch major usability problems - in information architecture and basic user interactions – early on and iterate on those, use low fidelity prototypes. When you want to test aesthetic look and feel of the product – something that might result in small adjustments – use high fidelity prototypes.
