I'm not 100% sure if this fits here, but I would guess it does...

I'm building an app where users can post and talk to each other anonymously. With that comes a lot of responsibility as I feel like some people will be tempted to post toxic content as their name isn't attached to it.

To prevent this, I have come up with a few solutions, and I wonder if you guys can give me your feedback whether you think this will have a positive or negative impact on the users that use the app (i.e will they be more inclined to post better content, or try and game the system and come up with creative ways to post worse content).

So here's what I have:

PerspectiveAPI uses machine learning to identify how toxic a sentence is. It's not 100% reliable but it works pretty well I'd say.

  1. Prevent users with an account age of less than 3 days from posting content that is considered more than 0.8 toxic score (according to Perspective)

For example:

You seem like a pretty dull person.

Has a toxic score of 0.78...

  1. My app has a reputation score (similar to this website, where the better content you post the higher score you have). Users can post any toxic score after 3 days but if the toxic score is greater than 0.6 or 0.7 notify them that if they post this they will lose 10 reputation, and ask them if they are still willing to post, or if they'd like to revise their post.

  2. Disallow users entirely from posting really bad words like the n-word.

What do you guys think of this? Is it too much? Too little? Do you think people would be more inclined or less inclined to post toxic content?

I want people to feel like they can post freely and openly, but within reason, of course.

Any feedback would be amazing! Thank you.

closed as too broad by RobbyReindeer, locationunknown, Andrew Martin, Shreyas Tripathy, Benny Skogberg May 24 '18 at 7:59

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Your question is very broad and you give little context about the "why" of your app. What's the point of these anonymous chats? Defining the point will help make your question more specific. And above all, it's against the guidelines to ask for opinions. – RobbyReindeer May 22 '18 at 13:31

This is a super interesting topic! I've recently thought about this as well - how best to moderate toxic behavior without having thousands of real, live humans scrolling through conversations (on a public forum, that is). For private conversations that's not really possible to use real people to moderate.

My take on the Perspective API: it could be good or bad. In the right environment with the right audience, it could foster positive conversations.

I think of Pinterest users vs. Facebook users. A Pinterest user is typically sharing something they like, in a platform that breeds positivity. Facebook is expected to be more open and a platform to share any and all opinions to challenge and spark a conversation - sadly negativity has become the norm (sort of).

So for Pinterest users they might actually like that the platform is promoting the already present positive conversations. On Facebook it could be viewed as suppression and a limitation to free speech.

Beyond the user base, there's also the age of the platform. Is this a new platform? Then creating this type of reputation scoring from the beginning could work beautifully! Users are mindful of what they post from the beginning, and those who are negative trolls against this type of big brother-like monitoring may just not use the platform at all. You could foster positive conversations from Day 1.

If this is already an established platform, you could turn away a lot of your user base if they have a habit of negative posts. Not necessarily bad though, as they might not be the most influential or profitable users you have.

Those are my initial thoughts, and I realize it's more of a two-sided answer...but hopefully it helps! Anyone else agree on these, or am I crazy?

  • Thank you this is really great! This is a new platform so I do get to mold the way the users interact with each other. – Jane Doe May 22 '18 at 15:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.