11

Given the following two users in a Stack Exchange like system, both of whom we can reasonably deduce will exist in a successful, mature system:

  • User A, who just signed up to the site and has as yet no knowledge of the subject matter of the site
  • User B, who has been awarded the highest accolade for knowledge of any given tag (that would be a gold tag badge on Stack Exchange)

and the following two mature systems:

  • System 1 - is like Stack Exchange. Users vote on the actions of others and reputation is awarded for these votes. Each vote has the same reputation effect regardless of the reputation of the voting user
  • System 2 - is very similar, except the vote from User B would have significantly more impact on reputation - so for Stack Exchange specifically this could be X2 for a bronze tag badge, X5 for a silver and X10 for a gold if the vote was cast on a question with the relevant tag

To be clear on the effects of System 2, if User A was an expert in their field then their high quality answers would attract votes from users like User B and User A would benefit from the reputation boost. User B's expertise would have no bearing on the reputation gained from votes cast on their posts, only on the reputation gained by the targets of their votes.

What are the differences in the state of the user bases in these two systems? In particular does the weighting of votes in favour of proven users result in unforeseen problems for users, both new and established? Has this problem been examined at all by anyone?

  • 5
    Excellent question. YouTube comments is a good example for System 2. Comments from users with large subscriber base are automatically floated to the top, sometimes even above comments with significant votes. – nightning Feb 20 '15 at 22:42
  • @nightning similar but subtley different, in system two a users reputation would not effect any gains for themselves (more rep or listing position), it would effect how much credit they could give others – Toni Leigh Feb 20 '15 at 22:50
  • I see. Hmmm I actually don't think I've seen a system like this elsewhere. – nightning Feb 20 '15 at 22:57
  • @ToniLeigh what kind of problems do you think are unforeseen with system 2 for new users ? Direct impact, indirect impact? – Okavango Feb 20 '15 at 23:07
  • i'd be interested to know why Stack Exchange don't implement system 2, but I didn't want to ask 'why don't SE do this', ideally I'd like comparison between two established systems, or reasons from the owners of either system type as to why they made their choice, preferably with data from real operations. SE an ideal candidate for system 1 – Toni Leigh Feb 20 '15 at 23:16
11

There are quite a few ways to answer this question.

But I think you need to consider the purpose (user/business goals) behind the current system, and the pros/cons of many possible alternatives (your idea is just one of many possible variation).

Essentially, the current system is not rigorous - as the designers opted for simplicity instead. Adding complexity to any system has to be justified.

Nevertheless, there are quite a few interesting topics related to this, and I'll try to squeeze in as much as I can.

Voting strategies - A bit of background

Voting strategies have far reaching consequences on results.

The original Facebook strategy - a unipolar Like only - means no negative reinforcements to its users (a good thing given Facebook's nature, specifically in the short term). But the lack of a Dislike option (a bipolar strategy) means reduced quality control and effective filtering (a bad thing in the long term).

Similarly, each of the following rating strategies would yield different results (think of Amazon ratings):

  • Five stars - from 1..5;
  • Five stars - from -2 to 2 (so very bad, bad, neutral, good, very good).
  • Four starts - (very bad, bad, good, very good - no neutral).

You should expect (in the case of Amazon at least, where the assumption is that most purchases are not being returned) a positive skew if the second or the third systems are being used.

The rich are getting richer

Now your question refers to weighted votes - where some votes count more than others.

While subject to mathematical analysis, you could argue the following case:

  • Reputable users (type B) are likely to produce better content than new users (type A). This assumption may not be valid, but play along.
  • Such better content will be voted up more, by both reputable and non-reputable users.
  • It may be the case that reputable users have better appreciation towards good content (in the same way art critics have better 'notion' of what a good art work is). So they'll vote more for reputable users.
  • It may also be the case that reputable users use the site more often thus will vote more.
  • The net result of this would be that reputable users will get more reputation than new ones. Which will, over time, create inequality of 'earnings'.

Now SE can easily perform some analysis to determine whether the case above indeed happens, and whether the argument above stand.

There are other topics, like voting asymmetry (the weight and ratio between up and down votes), which I'm not going to get into here.

The NBA's draft lottery

Perhaps a good analogy for this would be the draft lottery in the NBA, compared to the non-draft system of the English Premier League. The former allows the smaller getting better, where the latter generally makes the better even better.

Already an issue on SE

SE already has issues with the 'rich getting richer' problem. Particularly on StackOverflow, where some key questions were asked years ago and answered by early joiners - these people have momentous reputation much due to timing rather than any other true substance. See this question for an example.

This issue defies the gamification prospect of the reputation system.

The point being made is that an opposite system to what you suggest might actually work better - where the reputable users' votes get less weight, creating altogether a more balanced leaderboard.

The social aspect

Social equality

So the implications of the chosen voting strategy on various goals is one argument. But in my view, a much more important principle is in play here - that of social equality.

There is a growing trend (in scientifically-driven cultures at least) away from hierarchies and towards social equality. This is a next to universal trend, affecting government policies, organisational structures and processes, and even design - where the socio-technical view is growing ever more dominant.

The reason for this is that there is a mounting research evidence demonstrating that equality within teams - and society in general - yields the most effective results.

In simple terms, there is an increasing recognition that the focus should be on collectives rather than the individuals they consist of.

Social inequality (where some individuals are considered 'more' than others) has costed people's life - like in quite a few past Korean aviation accidents (where co-pilots dared not to point out pilot mistakes), or within hospital teams (where mistakes were not shared and dealt with due to fear of superiors).

Within organisations, research has shown that the best decisions were reached when the debate was inequality-free. The notion of a CEO having higher voting weight than a team leader, or even the idea that the CEO is seen as someone who's views cannot be challenged, have proven poisonous sometimes.

This is in no way to suggest that weighting systems have no place. There are many cases where they do, for example, crew and passengers have much higher stakes in airborne safety than ground staff. But that's an affair quite remote from a group of people objectively ranking content on SE.

Equality with balanced diversity

So far, equality was defined as not a single person has more weight than another. But equality has many dimensions and in fact, a perfectly equal team will be completely dysfunctional.

Most research that deals with social equality (within the context of decision making) also asserts the importance of balanced diversity. In that sense, an 'all-yes' (total agreement) and 'all-no' (total disagreement) groups are bad at decision making. The best groups are made of members whose diversity is balanced across various dimensions (empathy and risk taking, for example).

In that sense, you can argue that reputation is one dimension that has to be balanced when it comes to the voting strategy on SE. And so, based on the theories above, if compared to new users more reputable users vote on something - their votes should count less; but if compared to new users less reputable users vote on something - their votes should count more.

A further improvement would be not to split users into A/B groups, but rather weigh each vote based on the ratio between the voter's reputation and that of all voters.

Such implementation, while in theory could yield the most 'socially valid results' is cumbersome and may not provide any real benefit to the site, given its goals.

Summary

So first I'd tried to give a taste into how various voting strategies could affect overall results, and that many options can be chosen and tuned to fit user and business goals.

Yet, I've used research into social equality to argue why reputable/new weighted votes may yield overall dubious results.

Lastly, balanced diversity suggests that some weighing system of votes may be worth considering, albeit its benefits are questionable.

  • i often think about the SO point whenever I compare the answer on a basic css or js question with it's 1000s of upvotes with my measly 125 rep on that site, another problem I think they have is that the quantity of questions is so vast that one can add several answers that don't even get noticed, even if they're really good! – Toni Leigh Mar 7 '15 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.