Lab-Based vs. Field-based study for mobile (location-based) application

Question

Hopefully I've found the right corner of StackExchange...
Coming to my point, a few information are needed. The following paragraph is nothing more than an introduction, but might be important for my actual question.

I've implemented a Location-Based Service, which aims to highlight citizen engagement opportunities using GeoFences (a GeoFence is defined as an virtual area/barrier, which triggers a certain action). A scenario might look like the following:
A user, interested in "Public discussions" and "Sport events" (which receipts are donated to benevolent institutions) is walking around his hometown. Once the user enters a GeoFence, which corresponds to his interests, he receives a notification on his mobile device including additional information. The user can tab an on the notification to get further information or to vote/like this project/event.

The mobile application is part of my master thesis, which concentrates on the topic, whether a location based service as a mobile application can facilitate the user's engagement.
For this reason, a user study should take place.

Coming to my question: What would be the best way to evaluate my application or the usage of my application? I do not want to concentrate on the actual user interface (or usability), but on the user's experiences and thoughts.
Is there a Lab-based study the preferred method or is it better, to evaluate the application usage by a Field-based study?
I've already searched for some references and what I've found is nicely summarized by X.Sun and A.May:

A lab experiment is recommended when the focus is on the user interface and device-oriented usability issues. In such cases, a well-designed lab study should provide the validity required, while being easier, quicker, and cheaper to conduct. However, the results suggest that a field experiment is better suited for investigating a wider range of factors affecting the overall acceptability of mobile services, including system functions and impact of usage contexts. Where open and relaxed communication is important (e.g., where participant groups are naturally reticent to communicate), this is more readily promoted by the use of a field study.

X. Sun and A. May, “A comparison of field-based and lab-based experiments to evaluate user experience of personalised mobile devices,” Adv. Human-Computer Interact., vol. 2013, 2013.

As stated by the authors, a Lab-based study will be a good way, if I plan to concentrate on usability, but as already said: This is not my focus.
To give pros/cons for both methodologies:
Lab-Based-PRO

• Environment, known and unknown variables can be controlled
• Wrong app-usage can be prevented
• Higher internal validity

Lab-Based-Cons

• User might feels controlled
• No real-world experiences. Location has to be mocked/faked. User will have the device in their hand for the whole time. In a field study, the user might get the mobile out of his pocket once he would get a notification.
• Lack of external validity

---

Field-Based-PRO

• Real-World experiences
• Realistic behavior of user AND

Field-Based-Cons

• Much more complex than Lab-Based
• Needs more time for preparation
• User have to walk around the city
• No chance to answer questions or solve problems

---

Coming to my end: From my perspective, it seems that a field-based study is more valuable as the user uses the app in real-world conditions, but of course the Lab-based study has some advantages, too. I think that, although my application is a location-based service, a lab-based study would not be possible...faking the location is possible without bigger problems so it would not be a exclusion criterion.

It would be great to get to know your concerns and thoughts. So..what do you think?

Answer 1

This would seem to be a somewhat confusing and broad-reaching question.

You mention in passing something about the app collecting money for charity.

In that case you have your answer:

you would do A/B testing, and the result of "most money gained" is your result.

Unfortunately I can't see any value or relevance at all in "lab testing"; indeed I can't even see how it could be achieved in this case?? (I may be missing something.)

A/B testing of apps (and web sites) is, well, a standard, the "normal" procedure to ultimately determine what works in apps / www. This is a commonplace in every sort of site dealing with money from google and amazon down; indeed many tools (baas, web systems, APIs, etc) have the concept of A/B testing built-in.

BTW you mention the nomenclature "geo-fences". The type of app you describe is totally commonplace, you can download and use perhaps 1000s (10,000s?) of such apps exactly like this from the app store / google play today. I don't see any reason you'd introduce a "new term" you know?

"The mobile application is part of my master thesis, which concentrates on the topic, whether a location based service as a mobile application can facilitate the user's engagement" I don't know how academia "plays" on this topic, but, would this be a better approach: since what you describe is an old idea with a vast number of implementations, including any number of household-name implementations, would some sort of "survey" approach be better? So, you could garner information from the well-known dotcoms that do things like this? Indeed, perhaps you could just pick a specialised area: for instance recently I was using "zillow", you could look in to the question you pose JUST in the real-estate-app market. I found it really interesting the different ways that the main rivals in that field (this was in the US) use location: I'd be really interested to read some sort of analysis of the results of those guys and what one can learn from it.

I hope these broadly general comments help in some way.

{Just BTW, I don't think anyone says "geolocation!" any more .. it's just your location, what other location would it be?! :) }

Answer 2

First, you need to nail down a specific research question. That will have an impact on what type of study you do. Some things can't be studied well in the field or they won't yield enough internal validity. Others can't be studied in the lab well or they won't have enough external validity.

Second, you can do a controlled empirical study in a big area. It doesn't have to be in a lab. You probably have a spot an campus with diverse shops, sports fields, etc where you could give the participant instructions to walk around to certain places for about 20min. Or you could walk around with the participant and follow a script. Then you could ask probing questions at the right time about their experience. Another big advantage to lab study is you are much more likely to actually get a response. There will be a lot of missing data in a field study.

Be comfortable knowing you can't accurately measure everything in one study. In academia you would usually have multiple lab and multiple field studies published over multiple papers. Since you probably only have to do one for your master's, just pick one and report what that type of method reveals and note what needs to be studied in future.

Third, take a look at some human factors methods. They might help you structure your procedure. For example, situational awareness studies and mind wandering studies specialize in how and when to administer probes during a task and life in general. Obviously you may not care about those theories but the methods might be helpful.

Fourth, you should consult with your thesis committee members. It's their job to help you work through these problems. Further, they will ultimately be the ones that pass or fail you so you probably want them to help you choose so they are happy.