Our users use the app in a very private and natural context that is not really reproducible in a lab. What is the best way to learn about our users and how they use our app? Invite them for interviews? Cultural probes? In-app surveys?
Self-reporting methods (interviews, surveys) usually give less-accurate information than direct observation in users' own context. But, as you point out, you don't always have access to that context.
Diary studies involve self reporting too, but it happens more in the moment than surveys and interviews. In some studies, you specify that users write an entry at pre-set times (e.g., every Mon, Wed, Fri at noon), and in some users create entries on the fly (e.g., every time they use your app).
Check out YouTube for some overview videos about diary studies. See also Nielsen/Norman, UXPA, and visit the Googles. It's a really cool research method that's not performed very often. It takes some effort to plan and execute, but it gives a unique look into users' lives.
There is no specific method to test what is the best way to learn from your users.
What you can do is test various methods and see which ones provide better results.
Whatever you do, start with small samples so you keep the focus and the impact to your users (eg: requesting attention and time to your users to fill a survey) is minor.
And, above all, setup a criteria and see which methods are used to reach that criteria (some Google search will help on this).
You can check more info about this here and I quote:
List some characteristics of the target users you want to obtain information from. This can be things that you know off the top of your head like “active app users.” For example, when my team wanted to validate whether or not a feature should be worked on higher in the priority list, I translated that into screening only users who had touched that feature within the previous 30 days. They were the most relevant and highly active users to gather insights and feedback from.
If the pool was too small, I would expand the criteria to include a larger date range of activity or include broader criteria. I don’t recommend trying to broaden your scope too large, though, because you’ll start interacting with people who do not have much to say about what you’re trying to discover.