23

Scenario

We have a soon to launch app which includes a rewards system. These rewards are real money, so a direct Rewards Balance: $xxxx should suffice (in theory). Basically put, rewards are a real payment for some task completion

However, our first tests shows that there are 2 parts to the story. Since these rewards may decrease there are 3 scenarios: Balance grows, balance shrinks, Balance stays the same

In the first scenario, tests obviously shown users have a very good reception for the amount shown in $.

However, on the second scenario (balance shrinks) the friction went to the roof, based on loss aversion bias. In turn, this lead to test groups become static and stop doing any tasks at all.

For third scenario, we didn't find negative signs (at least nothing important) in relation to the $ amount shown, but we found a curious threshold pattern: after some sum, they froze, which means loss aversion started to play even before decreasing.

Proposed Solution

So we decided to go with points. So far, test users didn't react well or bad, they seemed quite uninterested. The task completion was more or less on the above mentioned threshold pattern, maybe slightly higher, but this happened for all 3 cases: growing, shrinking, stalling. On the brighter side, this allows us for a marginal profit

Current conclusions

While showing money amounts has a lot of friction, it has more tasks performed until that friction stage. Points have a lot less friction, but they're not enticing enough to make users perform more tasks.

Question(s)

Since we're close to pack this up, times are really tight for more testing, so my question is: is there any study showing that either points method or $ method is definitely better?

Assuming points is the way to go, is there a way that works better than others to avoid the above mentioned problems? For example: 1 point = 1$ OR 10 points = 1$ OR 1 point =$10

or ranges that may lead to less money for users but a higher completion rate since they will need to get to the next step for a higher sum. Example: 50 to 99 points achieved = $50 . 100 to 199 points achieved = $100

EDIT: Just in case it is of help for the answer, sums may go from a few dollars (well, another currency which has more figures, but for the sake of example) to 5 figures in extreme cases. Not likely, but possible. However, most amounts will be on the 3-4 figures range

EDIT 2: This is not a gamification model. This is real money for real tasks performed. Due to very strict NDA, I can't provide more info, but as a totally unrelated example: let's say the client wants users to find yellow, blue or red flowers with 5 to 8 petals. So many users will send flowers. The client will want only pne yellow flower with 5 petals, one with 6, one with 7, one with 8. Same for red and blue. Everything else will be rejected.

But then we'll find there are many users sending yellow flowers with 5 petals, many users sending yellow flowers with 6 petals and so on. We have an algorithm in place for this , but for the sake of example, let's say client will approve only the first flower of each type.

Due to this huge waiting time, we need to reflect tasks were performed and as long as the specs have been followed, temporarily approved, or users won't have any way to follow their progress and see the result of their efforts.

So we're using temporary figures, and clearly specify these figures are subject to final review, so we have no problem with the process itself, as most answers are pointing out, the problem is with the figures: money amount or points amount, with the problems we've found for each one

  • 4
    What activity leads to the user's balance shrinking? – Andre Dickson Sep 28 '16 at 19:00
  • 6
    @AndreDickson: in simple terms, the user has to do a task which after approval reflects on an increases of their balance. However, there are 3rd parties that have up to 60 days to review and claim, and eventually reject the task. Since the payment will not be issued after that 60 days security period, it's not problem for us, but the user will see their balance shrink. Also, we predict the amount of disputes will be really high, at least 25% according to our projections, so this balance issue is of paramount importance – Devin Sep 28 '16 at 19:22
  • 1
    another user case for shrinking: repeated tasks. The task was already performed by another user, which we can't know until 3rd party review. 3rd party will approve only one task, so the other repeated tasks will need to reflect a discount on their balances. – Devin Sep 28 '16 at 19:24
  • 27
    @devin if the balance shrinks due to third-party review, can't you have two balances - the confirmed total and the "pending review" total - and highlight review rejections to train users to perform better? – moopet Sep 29 '16 at 7:57
  • 4
    @KiranLinsuain, we're considering paying with chickens – Devin Sep 29 '16 at 15:30
28

I previously designed/tested an application similar to FanDuel/DraftKings that included both points and cash as a reward. With the points system, the user could accumulate points - lose them or have them stay the same. At a later time, we also included cash as a reward system as well. (Users could choose cash or points as their reward) From a testing perspective, this allowed us to analyze which reward system drove user behavior in a more positive manner.

What we found was that in a direct comparison of user behavior, cash maintained a higher percentage of daily active users. With points, a high percentage of users would utilize the product for 1-2 weeks then abandon their account. The users appeared to be desensitized/unenthusiastic as it relates the point reward system. Even when outlining the value of a point as it relates to cash - it still wasn't compelling enough.

There are downsides to both approaches...which you've clearly found through testing. We also experienced that the loss of cash is a friction point and even sometimes an abandonment catalyst. A good way to solve for that is to pad the loss with some sort of incentive. Meaning...hey, looks like you lost some here - but here's a bonus of xyz amount. Or here's what you can do to get back in the positive.

From that experience, it became clear to me that providing value to the user through the cash reward system was the way to go. You just have to work through the friction points. Whereas with the point system, the users aren't enthusiastic from the beginning of the experience. From that standpoint takes a lot more work to show them where the value exists when using your application.

Hope this helps.

  • Was the $ system opt-in? Maybe only enthusiastic user noticed the option. – Christian Strempfer Sep 29 '16 at 7:23
  • @ChristianStrempfer, very interesting point. Since users were testing a prototype they didn't signup or opt-in, we just gave them the accounts to use, there might be something there, thank you! :) – Devin Sep 29 '16 at 15:32
  • Many point based real life currency systems usually have each point be worth a penny or less. People like having bigger numbers when it comes to money, so 10000 points feels better than $10, even if it took 2-3 hours or more to earn it. Its all about the reward feeling, 100 feels better than 10, even if 100 points is worth 10 pennies. Its also an easy way survey sites hide the fact that you only earn a few dollars an hour on average. – Ryan Sep 29 '16 at 17:57
11

I don't think the response to a shrinking money balance can be attributed to loss aversion. Your users are not being faced with a choice between certain small losses and less likely large losses. Additionally, because the balance can shrink by any degree at any time they can't really compare the proportions of gains to the proportions of losses. They can only react to loss as it happens. I think the response you observed in your research is more intuitive than loss aversion. People simply have a visceral reaction to losing money. For example, spending a lot of money has been found to cause pain.

If users stopped performing tasks it means they effectively stopped using your product. I would speculate that users stopped using your product because they had a negative experience (losing money) and believed they would have more negative experiences in the future. If they hold such a belief it means the product hasn't done a good enough job of educating them on what will and won't happen. This means they form their own pessimistic beliefs about what the future holds i.e. more losses.

Consider creating a provisional balance that has secondary prominence to the main balance. The provisional balance would allow the user to keep track of potential earnings i.e. earnings that have the potential to be erased. If the user has a clear understanding of the fluctuations that can take place with a provisional balance, they may not have a strong negative reaction to losses of this balance. Since you've already tested that losses to a points based balance does not affect task performance I would consider representing this balance as points. Since money has been shown to produce higher task performance you want your main balance to be represented as money instead of points. Once the 'security period' has expired you can convert points in the provisional balance to money in the main balance.

I think you should also consider adjusting your 'security period'. The 60 day 'security period' means that users have to wait quite a long time before they will receive their rewards. Keep in mind that users will compare payouts from your site to other activities where they are being paid to perform like jobs and services rendered. Some people fare well at delaying gratification while some don't. I would consider shortening the 'security period' to a 30 day period at most, since most adults have been 'trained' to wait 30 days for a salary. This would help to mitigate users who may be defecting from the product because they can't stand the wait between completing a task and receiving a payout.

  • 1
    very nice answer, +1. Although I didn't mention them, we have noticed most of your points, but there are external situations that prevent us from using the best options and go for the best possible. In this case, the 3rd party is a governmental agency and the 60 days period is required by law, so nothing we can do about it (but after 60 days the user will start to see 15 days period payments). Your second paragraph is something we considered and right now playing around it , but we're stuck on user's behavior and interaction with any of the proposed rewards methods – Devin Sep 28 '16 at 22:06
4

Your situation reminds me of BitCoin Mining pools. Users do "work" for "points" that can be exchanged for "cash". BitCoin has the same problem, where work can be rejected by the network.

Slushpool, a BitCoin Mining Pool, shows several values of your rewards.

  • Confirmed Reward
  • Unconfirmed Reward
  • All Times Reward

This makes the current state of your rewards clear. All Time Rewards will never decrease. Confirmed Reward only goes down when a user spends/withdraws/etc points.

The Unconfirmed Rewards balance will fluctuate naturally. Either by being transferred to Confirmed Rewards or by being rejected. Since it is labeled "Unconfirmed" your users should be more understanding when a dispute occurs and a reward is removed. (Your users know that the work needs to be accepted, not just submitted, for a reward right?)

3

It might be worth trying to combine both approaches. Users should have an incentive to do tasks, because the tasks will eventually lead to a provable, visible amount of money. On the other hand, you want them to experience a loss in rewards as less tragically.

So, let the users gain rewards in points, but allow converting points to money in a well-documented manner. (E.g., something like "You can convert all points above 1000 at any time to money at a rate of 10:1") Give them an incentive to work for the next cash-out.

This way users don't feel that reward earned or lost are directly touching their purse, but they still have the knowledge, that points will eventually translate to hard cash.

2

Personally, I would have said, and I feel, the same as what you discovered about interest in your app. Cash is king and I little care about "points". Cash creating higher interest might be what you really want: interest. If interest wanes as cash drops then maybe those are users you don't want.

So friction is what needs to be removed but I assume that's not what you can do. You might consider the cash drop to be a poke rather than a stab so they feel the nudge but not the pain.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.