I disagree with you regarding the idea that web applications don't have an end goal or target for users to achieve or complete. Everything in your system should have a purpose. And every purpose can probably be measured.
To answer your question though, I wanna go ahead and acknowledge that engagement as a calculation is very broad, vague, and changes per application. Engagement on Pinterest is probably very different from engagement on Visual.ly. So to measure "engagement", you measure its parts. And to measure its parts, you must define them. (Also, I'm strictly talking about drawing conclusions from data, not user feedback.)
Let's take an example like Wolfram|Aplha.
W|A is a computation engine and doesn't really have typical pages. You enter a term/phrase and it "computes" results for you, and the types of data returned change depending on the context of the query.
Since users don't have a typical end or path (search > results > item), how do you measure whether users are engaging with the results, strictly using analytics? Well, I'd start with these questions:
How can the user interact with the results?
Mostly, users can digest the information on screen, but they do have options (some premium), such as: Enlarge, Download, Customize & Save, Copy, Enable Interactivity, and Clip n' Share.
Also, they can change the query through suggested alternatives (relate or disambiguated); and expose more of less information, if applicable;
How can we measure those interactions?
Just a few of things you could track:
1. How many clicks is each option getting?
2. How long between one option being clicked and another being clicked?
3. How long is a user staying on each query before leaving?
4. Does the user conduct another query? How long between queries?
5. Is this user a returning visitor?
Here's a glossary for Google analytics and should help to understand the language Google uses to define these types of metrics.
What could these measurements mean?
Taken as a whole, the data could be telling you: if users come back (#5); how they interact with the results they receive (#1); if those interactions are appropriate (#1, #2); and if the results are relevant (#3, #4).
I'll note here that I wrote "what could these measurements mean" on purpose. Analytic data needs to be taken very much in context, and that context can be warped easily. So, relying on analytics without following up your assumptions with users can lead you down a wrong path.
Louis Rosenfeld talks about this in more detail in this interview.
Interestingly, Kissmetrics notes that when used wrong, time on site can lead you in the wrong direction. This reinforces the idea that data is contextual, having different uses and meanings depending on the context.
The takeaway from this, for me at least, is that engagement is uniquely defined per product and is a constantly moving target. It requires combining lots of different metrics to abstract an idea of what's happening and those metrics will be different depending on what you're trying to measure. And having very specific user goals and tasks will help you define the metric you need.