Hopefully I've found the right corner of StackExchange...
Coming to my point, a few information are needed. The following paragraph is nothing more than an introduction, but might be important for my actual question.
I've implemented a Location-Based Service, which aims to highlight citizen engagement opportunities using GeoFences (a GeoFence is defined as an virtual area/barrier, which triggers a certain action). A scenario might look like the following:
A user, interested in "Public discussions" and "Sport events" (which receipts are donated to benevolent institutions) is walking around his hometown. Once the user enters a GeoFence, which corresponds to his interests, he receives a notification on his mobile device including additional information. The user can tab an on the notification to get further information or to vote/like this project/event.
The mobile application is part of my master thesis, which concentrates on the topic, whether a location based service as a mobile application can facilitate the user's engagement.
For this reason, a user study should take place.
Coming to my question: What would be the best way to evaluate my application or the usage of my application? I do not want to concentrate on the actual user interface (or usability), but on the user's experiences and thoughts.
Is there a Lab-based study the preferred method or is it better, to evaluate the application usage by a Field-based study?
I've already searched for some references and what I've found is nicely summarized by X.Sun and A.May:
A lab experiment is recommended when the focus is on the user interface and device-oriented usability issues. In such cases, a well-designed lab study should provide the validity required, while being easier, quicker, and cheaper to conduct. However, the results suggest that a field experiment is better suited for investigating a wider range of factors affecting the overall acceptability of mobile services, including system functions and impact of usage contexts. Where open and relaxed communication is important (e.g., where participant groups are naturally reticent to communicate), this is more readily promoted by the use of a field study.
As stated by the authors, a Lab-based study will be a good way, if I plan to concentrate on usability, but as already said: This is not my focus.
To give pros/cons for both methodologies:
Lab-Based-PRO
- Environment, known and unknown variables can be controlled
- Wrong app-usage can be prevented
- Higher internal validity
Lab-Based-Cons
- User might feels controlled
- No real-world experiences. Location has to be mocked/faked. User will have the device in their hand for the whole time. In a field study, the user might get the mobile out of his pocket once he would get a notification.
- Lack of external validity
Field-Based-PRO
- Real-World experiences
- Realistic behavior of user AND
Field-Based-Cons
- Much more complex than Lab-Based
- Needs more time for preparation
- User have to walk around the city
- No chance to answer questions or solve problems
Coming to my end: From my perspective, it seems that a field-based study is more valuable as the user uses the app in real-world conditions, but of course the Lab-based study has some advantages, too. I think that, although my application is a location-based service, a lab-based study would not be possible...faking the location is possible without bigger problems so it would not be a exclusion criterion.
It would be great to get to know your concerns and thoughts. So..what do you think?