46

I have an app that enables users to exercise questions. The users have an option to report a faulty question. The problem is that the users use the option too frequently. The reviewer has checked the reports and it seems that the users don't read the questions. It is much easier for them to report a question than to read it closely. We can't change the questions. We are looking for a solution which will make the users more self-reliant, and avoid false reporting because of laziness.

  • 72
    Have you considered the questions to be at fault? Could they be awkwardly worded, needlessly long? This also sticks out: We can't change the questions - does that mean feedback is pointless? – domen Nov 18 '16 at 15:28
  • 6
    @prognoza: domen has a point - it would help if you explained what the option is for. When you say you cannot change the questions, do you at least have the ability to remove actual faulty questions, so the purpose of user feedback is to help identify and remove them? If not that, then what is the purpose of the user action? How is that purpose explained currently in the GUI? – Neil Slater Nov 18 '16 at 16:12
  • 31
    If you cannot change the question then why do you even have an option to report faulty questions? – paparazzo Nov 18 '16 at 22:35
  • 32
    What exactly does the reviewer do with the reports if they can't change the questions? Look at them and go "yep, that question is terrible" and then go grab a soda? – Zach Lipton Nov 18 '16 at 23:52
  • 5
    Some question are supposed to be hard, because they mimic the test which does require reading closely. The feedback isn't pointless, because sometimes, invalid data slips in. When I say the questions aren't changeable I mean that we can't lower the questions' complexity. – prognoza Nov 19 '16 at 17:11
85

If the problem is that "...report button is too convenient" and you can't change this (otherwise check Thomas' great answer) then you already have the solution implemented here on Stack Exchange network:

Ask them to write why they think the question is broken. Zach Lipton suggests to start only with this rule and implement the others only if required, I absolutely agree with him: if your community is disciplined enough then easier is better.

There will always be someone that fill with spaces, junk or try to circumvent your checks, your goal is to prevent this without bothering who is using that feature appropriately. Few validation rules you may want to apply:

  • Text should be, for example, at least 16 characters length. "Wrong" is not a valid reason. Don't forget they're not browsing random content but questions someone else prepared, they can't be just spam then a detailed reason should be provided.
  • Leading and trailing spaces do not count for the length. "It is wrong     " is not a valid reason.
  • Four or more repeated consecutive characters are not allowed (or they do not count for the length). "It is wrongggggg" is not a valid reason, also "It is     wrong" is not valid (a trick that reviewers may not note because HTML will collapse multiple spaces.)
  • Trailing punctuation does not count. "It is wrong!!!!!" is not a valid reason.

Do not try to make it easier counting for alphanumeric characters unless you're ready to deal with culture differences and text encoding. After you collected few statistics you may add few more rules (however do it after you analyzed current trend, it's useless to add unneeded complexity):

  • If an user wrongly flags more questions than average (+1 stddev) then he will need to double confirm and previous rules will be stricter (for example 32 characters). At first I wrote "...user flags more questions than average..." but Francisco Presencia edited this post changing the sentence to "...user wrongly flags more questions than average...", meaning is different but I think he is right and this addresses also Tim Grant's concerns about legitimate usage.
  • After a decent amount of rejected flags an user will be temporary banned from flagging.

According to your application's nature and content you may want (as suggested often for this kind of things) to add a gaming/competitive nature using score. Each declined flag will subtract two points, each accepted flag will add one point. When you reach zero you can't flag for one month.

Make it personal, when they submit a feedback remind them of their flagging history ("You already submitted 78 flags and 70 has been declined as wrong".) You may even have different wording when this ratio is above 50%. It's not exactly the same effect outlined in Effects of eye images on everyday cooperative behavior but it surely helps. I remember a nice case study (which I can't find in this moment) written by an high school teacher, he (obvisously some years ago) started to use his notebook in class to keep track of students' progress. He observed a drammatical improvment in their performance (and a drop in their excuses) even if, in practice, to use a computer or pen and paper isn't so different. To know there is a detailed report of your actions is a big deterrent.

Note: keep track of rejected flags and attempts to flag a question, they may indicate that a question is formally right but maybe poorly worded.

  • 52
    "If an user flags more questions than average" - That's not a valid criterion for laying extra requirements on a user. What if all their flags are valid? What if they are one of only a few users who ever flags bad posts? – Tim Grant Nov 18 '16 at 15:21
  • @TimGrant I agree, threshold shouldn't be moved blindly – Adriano Repetti Nov 18 '16 at 15:30
  • I agree with the principle of addressing the problem with test criteria like these, but outright rejection seems heavy-handed. Perhaps the types of filters you talk about could be used to automatically tag feedback responses, e.g. overpunctuated or allcaps or repeatedchar. Give respondents other tags to choose from: typo, poorwording, ambiguous. Maybe scoring would help, but only if you let the user know how their response fares through the filtering. Maybe even, on the backend, helpful respondents could be given "reputation" which loosens the restrictions. – Rich Nov 18 '16 at 18:21
  • 5
    I disagree about the length requirement. "Spam" is a perfectly valid description for an obvious spam post, for example, and adding an arbitrary length requirement will just lead to reasons like "Spam spam spam spam spam" – Pikamander2 Nov 18 '16 at 22:44
  • 1
    Zach, yes absolutely. I went little bit more far just because comparing with SE but smaller communities are probably easier to handle – Adriano Repetti Nov 19 '16 at 7:46
73

There are certainly options which make use of UI design and not only input validation. Don't get me wrong: input validation is also a good point, see the other answers. I just think it's not the only way.

Consider this as the starting point:

Equal

The buttons are sized equally, providing the impression that they can be used equally often.

As a first improvement, make it clear that these options are not equal:

Link

Next, add some distance from the submit button:

Distance

Also, changing the text from "Report faulty question" to "Provide feedback" could make it less obvious that it is possible to bypass the question:

Feedback

  • 11
    I like this solution, IMO it can't be used alone, because a lazy user will quickly identify and misuse even that link vanishing our effort, but it'll help/force new user to think twice before skipping an answer! – Adriano Repetti Nov 18 '16 at 15:33
  • 2
    Plus one from me good job I would do the exact same. – JonH Nov 18 '16 at 16:44
  • You could maybe even put it behind 2-3 clicks, making it more tedious to reach than the submit button. That way someone won't be able to spam it on every question without beeing annoyed. – HopefullyHelpful Nov 21 '16 at 12:17
  • 5
    And maybe you could make the "provide feedback" link move around the page, so when they mouse over the link it dodges out of the way before they can click on it. – David Gelhar Nov 21 '16 at 18:58
  • 1
    I think this is a terrible solution. Either you want feedback, then you make the feedback process as good/easy as possible. Or you don't want feedback, then you remove the option to give it. – Nobody Nov 22 '16 at 11:59
24

The fact that your users submit pointless feedback is good. The far harder problem in collecting feedback is getting them to do it at all.

Please improve the UX of your flag reviewers, not your flag submitters. Do they have tools to skip short feedback? Redundant feedback? Is the feedback grouped by category using even procedural language processing techniques?

Never get rid of user data just because they didn't format it well enough for you!

  • 2
    He already knows most user data coming in is no good. – Joshua Nov 21 '16 at 20:43
  • @Joshua and it's hard to just throw out the data that's no good. Better off to process it better. – djechlin Nov 22 '16 at 0:48
  • As you know there is no processing that can improve on complaints because the users could not be bothered to understand in the first place. – Joshua Nov 22 '16 at 2:09
  • @Joshua I disagree and it's wrong anyway. The problem is if 10% of complaints are good it's very hard to suppress total volume of complaint without suppressing useful ones. That being said, I mean, for comparison, Tinder has figured out how to get people to evaluate a prospective romantic partner in about 0.25 seconds. There are absolutely UX strategies to help review complaints quickly. Start by triaging them. Etc. – djechlin Nov 22 '16 at 2:12
  • 2
    I'm absolutely certain Tinder hasn't figured out how to get people to evaluate a perspective romantic partner in any way worth doing in about .25 seconds. – Joshua Nov 22 '16 at 4:56
9

I find it frustrating as a user when it's too hard to flag or report come content, so I think it's good to have it be convenient. On the other hand, as you point, it could increase the number of reports dramatically and that's not necessarily a good thing.

I would extend Adriano Repetti's solution like this:

  1. Have a mandatory "reason of report" dropdown and an optional comment text field - you don't need to say it's optional; just don't enforce any particular rule there.
  2. Internally, have a set of rules (minimum reason length, gibberish text filters etc) that decide if this report is something that should be looked into (a "good report") or not (let's cal it a "poor report").
  3. Good reports can have a weight of 1.0 and poor reports can have a weight of 0.25 so that 4 poor reports by different users are needed to produce any sort of action that usually is triggered by the report function of your app.
  4. Some users will report more than others. If a user is an outlier (reports way more than usual), lower their report weight to 0.1 and possibly show them a captcha.

This way you don't annoy most users, keeping it simple for them, and also reduce the amount of work done on your part.

  • 26
    Some users will report more than others. If a user is an outlier (reports way more than usual), lower their report weight to 0.1 and possibly show them a captcha.. Well only if he reported crap, that would be quite damageable to miss a super user that report quite often but always with good comment. – Walfrat Nov 18 '16 at 15:17
  • I guess you are right, but I feel it's more likely to be an offended justice warrior than a power user, depending on the community. I think you can take my comment not as a recipe to success but as something you can experiment with. Think Youtube vs StackExchange: the kind of "report happy" users is likely very different. – ecc Nov 18 '16 at 16:20
  • Yes of course I agree there are way more trolls than anything else :) – Walfrat Nov 19 '16 at 0:20
  • 2
    I so agree with your first paragraph. There are systems in the wild that initially seem to allow you to provide feedback about problems easily - but once you have compiled your feedback rebport, you need to upload it to some-service.com, and to do so you must first ceate an account with some-service.com, and to do so you have to go through the usual mail confirmation procedure, including possible spam and greylisting delays, and so on ... – Hagen von Eitzen Nov 19 '16 at 10:34
  • @ecc Why guess if you can check? If the reports are being looked at anyway there's data that says if they are crap or not, use that data. – Peter Nov 21 '16 at 17:01
2

Hello User XYZ,

We are aware that these questions can be quite difficult and involved; they were designed to be this way.

Before reporting this question please consider going through the checklist below:

  • Read the question at least 3 times; taking a 30 second break in between readings.
  • Move on to the next question and come back to this one later.
  • Additional suggestions...

Thank you

If you still wish to report this question then please click the link/button below

Report this question.

Upon clicking "Report this question" provide another pop-up with questions and dropdowns which requires them to validate their decision to report the question.

  • Users will read the checklist, think "I can't be bothered with that!" and then find "If you still wish to report this question..." with a handy button. I doubt that implementing this will have any material effect, I'm afraid. – Andrew Leach Nov 21 '16 at 23:46
  • 1
    As a user, I would probably not read those bullets. I'm way too impatient. – ecc Nov 22 '16 at 11:00
  • 2
    @ecc As a developer I would disable the button for 90 seconds so that the user either watches paint dry during that time or they can read the message and/or re-read the question instead. – MonkeyZeus Nov 22 '16 at 13:41
  • 1
    @MonkeyZeus that would infuriate me and I would stop reporting questions, possibly drop your website altogether for being annoying. That sounds like a terrible idea. You are not entitled to your users time, you should cater to them, which is why I think this kind of problem would benefit from making it easier to report, NOT harder, and then use automation/heuristic tools to filter out the trolls from the valuable reports. – ecc Nov 22 '16 at 14:01
  • 1
    @ecc Well, if your job or school grade is dependent upon answering the question and not giving up so easily then I am sure that your choice to automatically remove yourself from the test would be appreciated by everyone. Why waste the smart moderator's time with faulty reports by those that give up so easily? – MonkeyZeus Nov 22 '16 at 14:11

protected by Community Dec 12 '16 at 7:25

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.