Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I've got an upcoming project where there can be literally thousands of submissions at any one given time that need to be manually reviewed before publication (or rejection). So, there needs to be multiple moderators to handle the load.

Each moderator sees a list (in tabular UI format) of submissions, and all necessary info is right there on the list for a moderator to make a decision (approve/deny). Each item can be manually approved or denied one by one, or bulk-approved/bulk-denied by clicking on check boxes and then click either the approve selected or deny selected button.

If we've got multiple moderators looking at a lot of the same pending work at any one given time, there's a 100% guarantee that we'll get moderation conflicts. For example, a moderator may have already denied a particular item a mere 100 milliseconds before another moderator tries to approve it.

So, what are some usability guidelines that would apply in this situation to avoid moderation conflicts?

We can't do constant polling because that's computationally expensive, and it would update each moderator's page too fast anyway, making moderation practically physically impossible. It looks like we'll have to give them feedback on which of their approve/deny attempts went through and which ones did not (and provide a reason), but that, I would imagine, would be very annoying.

We're thinking that a moderator pulls up only unmoderated submissions at the instance he/she loads his/her list, and then immediately assign him/her as the moderator of the submissions he/she pulled? This way, nobody else can claim moderation for an item, thus greatly reducing (or even entirely getting rid of) moderation conflict?

Are there any other suggestions on how I should tackle this?

share|improve this question
1  
No there is not a 100% guarantee that there will be moderation conflicts. That is only a guarantee if the developers don't address this properly. However, addressing it properly possibly takes more effort than addressing the possibility of conflicts. The way the SE sites do it is to inform you after you click your decision when your decision was pre-empted by someone else and when your decisions conflicted, also offering you the links to take action on that. –  Marjan Venema Feb 21 at 10:52

3 Answers 3

I've had to deal with this problem before when I was designing an app review system for my company.

Our solution is:

  1. A submission has an unclaimed, claimed, and reviewed state (reviewed state can be approved or rejected)
  2. Any reviewer can claim an unclaimed submission
  3. Once a submission is claimed, the status of the submission is changed. If another reviewer tries to claim the same submission, a message will display informing the reviewer the submission has been claimed. If reviewer refreshed the table list, the claimed submission will hide.
  4. Reviewer has the option to approve the submission or reject it. A claimed submission will need a time limit in case reviewer idles on the submission for what ever reason. An expired submission will go back to the unclaimed state. Your company can determine what that time limit is.
  5. A submission with a decision is in the reviewed state and you guys can create a table for log and tracking purpose and also to resurrect any rejected submission or kill any approved submission if an error was made.
share|improve this answer

First don't do any bulk process, doesn't matter how much work is it going to be, don't do that. Approving or rejecting things has to be done on a per case base.

With that out of the way, you have a few options.

You can have the interface or list of items on a page with an AJAX process that checks frequently for changes and updates the list constantly. If you go this way, you have to also add an AJAX process on the post itself to tell the person doing the revision that a change has happened and that the post has already been approved/rejected. But as you mention, this path takes a lot of energy from both, system and user.

Other option is to have an interface with a number of items to be moderated, but without showing them, that way, the moderator is going to click on something that is going to start the process, for instance a button saying something like "check first message on queue", then the moderator will be taken to a page where a message will appear and he can do the revision. This option also benefits from having an AJAX process checking for changes on that specific item/post/message.

Personally, I would go for option two since that relieves the server from a constant stream of requests from the list being loaded on different places and only leaves the requests for specific items.

Another way to reduce load on the server and conflicts on moderation, is to start with a random message, that way there are less chances of conflict. You can do this in a pure random fashion, or in reduced random way. With reduced randomness, the idea would be to consider only the first 20 messages of the queue to assign for moderation, that way, you still reduce conflict, but attend oldest post first.

One extra step that you may consider, is having a "second" revision, if a message receives two or more status changes (approve/reject) with at least one of them different from the rest, then it enters on a new, separate queue that has to be attended by a third moderator, that moderator would see a specific message about the post informing him about conflict on moderation. If a message has more than one status change, but all are the same, then they stay approved or rejected.

share|improve this answer

Look to bug/support/ticketing systems used primarily in software development for ways to solve this problem. Trello is a great one to look at, the premise of boards and states through drag and drop, Jira a popular bug tracking/task app for software development is another.

The flow would be something along the lines of:

  • Submission comes in and is set to "unassigned"
  • A moderator can go in and pick a submission to moderate, this would then assign the submission to the moderator and set it to, "assigned"
  • The moderator approves/denies the submission and the state is changed accordingly

Ideally your moderation aspect would be state aware. Moderators should be able to see submissions fade out when they've been taken in real-time. This would eliminate most cases of race-collissions occurring where a moderator approves a submission whilst someone else attempts the same thing.

There will always be the possibility of a race-collision (multiple users attempting the same action), but through a real-time interface and nice UI, you can eliminate this somewhat.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.