Previously, I discussed how many of Yahoo! Answers’ problems are endemic to its design. In other words, the problems it faces are, in large part, due to how it is designed and who uses it.
In this entry, I want to discuss the problems with moderation; specifically, how Community Moderation isn’t working, and why real-time human moderators are needed.
An Educated Guess At How Community Moderation Works
Let’s start by recapping how Community Moderation works. While Answers won’t divulge specifics, it’s fairly obvious, from what they will say, that it’s a weighting algorithm that works something like this:
- Person A, who has never before made an abuse report has, on a scale of 1 to 100, a weight of 30.
- Person B, who uses the reporting system out of spite, has a weight of 10.
- Person C, who is omnipotent and always gets reports correct, has a weight of 95.
- Suppose it takes a combined reporting weight of 200 for Community Moderation to remove a question or answer. (I believe this is about right, based on conjecture from questions I have reported).
- If 2 Person Cs and one Person B report a question, their combined weight is 200 (95 + 95 + 10), so the content is removed.
- If 2 Person Cs report the content, the content remains (95 + 95 = 190, 10 points shy of removal).
- If 7 Person As report the content, the content is removed (7 * 30 = 210).
- If 6 Person As report the content, the content remains (6 * 30 = 180, 20 points shy).
- If 16 Person Bs report the content, the content remains (16 * 10 = 160, 40 points shy).
This may be a bit too simplistic a model; again, I’m guessing based on what I’ve seen and what Answers has said.
For example, it may be that negative weights figure in. For example, maybe Person B, because he is so spiteful, has a weight of -10, rather than 10.
If so, take the case of two Person As reporting alongside a Person B. Now, it’s no longer enough to have a question removed, it’s 20 points shy; 95 + 95 + -10 = 180.
I have seen some answers that clearly should have been removed, based on the “Thumbs Down” rating they received; if an answer has seven “Thumbs Down” ratings, but its question only has three answers, it’s clear a lot of people reported the answer. When the answer is still around with many “Thumb Down” ratings, it suggests that when the wrong people report content, it actually makes it harder to get rid of the content.
Answers also states:
When a report is submitted to Yahoo!, Community Moderation first measures the contributions and reporting history – or ‘reputation’ – of both the reporter and the person being reported.
To accomplish that, it may well be that then questioner’s reputation weight is automatically added to the base weight of the question as the target weight.
For example, if I am brand-new, maybe it takes 230 weight points to remove my question, rather than 200; if my moderation weight is -30, maybe it only takes 170 points to remove my question, and it my weight is 95, it takes 295 points to remove my content.
As far as changing your weight, that’s more straightforward:
- Reporting content that was removed gives you, say, 3 points.
- Reporting content that was not removed costs you, say, 3 points.
- Having your content removed costs you, say, 10 points.
- Successfully appealing your removed content gives you, say, 13 points, but costs those who reported it, say, 15 points. (Although it’s worthy of noting, I know of no one who has ever successfully appealed content removal, myself included.)
Again, these are simple assumptions about the system, but I am certain the algorithm, whatever it actually is, isn’t too far off from these guesses.
Why Community Moderation Fails
The problem with a system such as this is twofold.
From a technical standpoint, it is far more punitive to new users than it is to older users, which is the exact opposite of the intent behind Community Moderation in the first place. By the same token, because its algorithm is not transparent, it inspires distrust among all users, even those using it properly.
More important, its very name is ironic: By weighting, in the parlance of my previous chicken coop / farm metaphor, all animals are equal, but some animals are more equal than others. That’s not “community moderation,” it’s meritocracy. And even though most people would agree with meritocracy, it’s the exact opposite of Answers’ stated desires — not to mention, it defeats the entire concept behind community moderation, which is equality.
Let me be very clear, before I proceed: Any moderation system that is going to work must give authority to only a portion of users. That certainly appears to be the end result of Community Moderation. The problem is that it’s a half-measure that actually makes moderation problems worse, not better.
Let’s address the inherit unfairness in a weighting system, especially seen within the intent of Community Moderation. Yahoo! Answers states its objectives are “to filter out specific targeting of one user by another” and “(promote) accurate reporters, giving them more influence.”
The problem is that these are mutually exclusive, but not intentionally.
If, by virtue of being a strong reporter, my votes for bad content weigh more heavily than the votes of some other user, I may not be intentionally targeting another user, but my say as to what the content of the site should be is significantly greater than the say of many others.
In other words, it has the same effect as targeting another user: My interpretation of the rules weighs heavier than the interpretation of another.
Nebulous Content Rules Rot The Whole Barrel
Which brings us to the root of Community Moderation’s failure: The Community Guidelines.
The guidelines are intentionally subjective. Given the breadth of users and the subjects being covered, by design, the guidelines must be fairly loose.
It’s easy to pass on concrete, basic rules, such as requiring people to use the question and answer format. That, for example, seems obvious enough. If I post this:
I really hate YouTube. Every time I try to post a video there, they delete it. I am making my own videos, and they don’t contain any copyrighted stuff, but YouTube deletes it anyway and never responds to my e-mails asking why.
That, clearly, is a rant. But is this any better?
I really hate YouTube. Every time I try to post a video there, they delete it. I am making my own videos, and they don’t contain any copyrighted stuff, but YouTube deletes it anyway and never responds to my e-mails asking why. Has anyone else had this problem?
Answers would probably claim that’s chatting, rather than ranting. But what about this?
I really hate YouTube. Every time I try to post a video there, they delete it. I am making my own videos, and they don’t contain any copyrighted stuff, but YouTube deletes it anyway and never responds to my e-mails asking why. Can someone please tell me how to get around this, or get in touch with Youtube?
Answers would probably consider that a legitimate question. But it’s really the same crap in a different wrapper, which doesn’t make it candy.
At some point, everything that seems objective actually comes down to a subjective judgment call. And since that’s the case — and the weighting behind Community Moderation only rewards the subjective interpretations of the rules shared by Answers itself — the effect is that there are, indeed, moderators on Answers.
Answers designed Community Moderation with a false assumption: That everyone, or at least a significant majority of users, would interpret the guidelines the same way; or, more accurately, that the rules aren’t subjective and therefore, misinterpretation is rightly punishable.
Were that assumption is true, then weighting is moot; in fact, it’s counterproductive. Because it is false, weighting Community Moderation actually achieves the opposite effect of its intent. Two premises, same outcome: That’s the dichotomy in Community Moderation.
I don’t want to get into a huge discussion here about group dynamics, but I will agree that as a rule, large groups tend to come to correct conclusions. There are startling cases where that is not true — witness the current financial crisis — but in aggregate, ask a large group to assess something, be it the number of jelly beans in a jar or whether someone’s question is shit — and they will come up with a consensus that is fair and accurate.
Were Community Moderation unweighted, this would be more obvious, transparent and fairer.
If the rule was simply, “If any five (or 10, or 20, or however many) people think your question or answer sucks, it’s gone,” then that would be true Community Moderation. Some quorum of persons, each a true peer, just gave you the boot. The community spoke. Don’t like it? Go find a different community, where only four people think you suck.
But because the system is weighted, it gives certain members — those who agree with Answers’ own interpretation of its rules — more of a say than those who do not share Answers’ interpretations. Maybe only two people need to think you suck in order for your content to get the boot, so long as they are the right two people.
This has the net effect of creating pseudomoderators. And while that’s directly contrary to Answers’ stated objectives, actual moderators are exactly what Answers needs.
What Answers Can Never Become: Self-Moderating Boards Examined
As I stated before, the strength of a forum is its replies, not its questions. Nothing drags down a forum like worthless topics, trolling and flaming.
By the same token, no board can be successful if there isn’t some sort of groupthink behind it. In fact, the more the case of groupthink applying to a board, the more efficiently it self-moderates.
These types of successful bulletin boards are either single-subject, limited-interest forums — such as the ASP.NET forums, which are very helpful — or have a given personality / mentality behind them, such as 4chan.org, and more notably, its /b/ board.
In both cases, the topics that get posted are largely ancillary to the replies they invoke, because by the nature of what these boards are, the content that will be posted in replies will have significant value.
In the case of the ASP.NET forum the topics, which tend to be technical problems, serve as segue to replies, which contain solutions; while the problem (topic) is needed to generate the solution (replies), it’s the replies that people visiting the forum want to have.
If no one ever answered the questions asked at the ASP.NET forum, or if the answers were consistently wrong, no one would stick around. By the same token, if the questions asked were largely worthless nonsense, no one would bother replying.
In the case of /b/, invariably, whatever someone posted to start a thread isn’t anywhere near as clever or provocative as what gets posted in replies. The topics are the straight men; the replies are the comics. Anyone who frequents /b/ knows he’s fodder for the cannon known as Anonymous.
Interestingly, both boards need only light moderation, but for different reasons. In the case of ASP.NET forums, about the worst content that gets posted is spam; you may see the rare flame or troll post, but most of the attraction of those forums are the ability to ask a programming question and get an informed opinion in reply.
Simply put, there is little to no incentive to troll or flame the board, because of its narrow appeal and the near total homogeneity of its users.
In the case of /b/, the basic concept is, “fend for yourself.” So long as what you are posting is not overtly criminal, such as child pornography, post whatever you want; and if you can’t stand the heat, get out of the kitchen.
That freedom, interestingly, also encourages a certain homogeneous user — Anonymous.
Answers wants something in the middle, and clearly, it can’t have a middle ground.
Because anyone can participate — because Answers wants everyone to participate –Â virtually any content is OK provided it doesn’t violate the question and answer format, and doesn’t hurt anyone’s feelings (more on this nonsense in a moment). As the YouTube example above shows, such content is often worthless, even when it follows those rules.
At the same time, the Community Moderation system, which is supposed to check bad content, actually disenfranchises those who are just starting out with reporting.
In theory, if the vast majority of Answers users disagreed with Answers’ assessment of the Community Guidelines, reported content that Answers values, and Answers restored that content, Community Moderation could actually be its antithesis — the majority of reporters, because they are not trusted, would promote content it doesn’t want by reporting it.
That’s the frustration most people feel with the Community Moderation system, which is what Answers is talking about when it says, “Due to its very nature, there is a great deal of interest in this system and how it’™s working.”
Answers would have you believe that the curiosity is academic. In truth, the problem is that because the exact algorithm is opaque, and because “bad” reporting is punitive, even long-time participants in Answers are fretful about reporting content that, in general estimation, is worthless.
In other words, Community Moderation actually makes the community distrustful and nervous, when the success of Answers depends entirely on an enthusiastic community with clear rules and specific appeal.
The Case For Real Moderators
Which begets the obvious question: If the effect of a weighted Community Moderation system is to create the next best thing to actual moderators, why doesn’t Answers simply create actual moderators?
I suspect legal reasons are partly the issue. If Yahoo! empowers someone to act on its behalf, even as a contractor, then Yahoo! is exposed to liability for that person’s actions. Therefore, any moderators would have to be vetted and managed, and take it from me, hiring and managing employees, especially casual employees, is a major pain in the ass.
But the majority of the problem is demonstrated in the fact that Yahoo! did not link the rankings or Top Contributor ratings to Community Moderation.
In fact, when Answers first rolled out Community Moderation a few years ago, it went out of its way to note that someone’s rating (points total) or status as a Top Contributor lends no weight whatsoever to Community Moderation.
It seems that may no longer be the case, if “contributions” means one’s ranks or ratings. But it was the case in the past, and Answers loudly proclaimed it because they wanted to encourage participation.
In other words, n00bs are welcome because l33t can’t shout them down. Just because you’re the biggest cock in the hen house, you don’t rule the roost; everyone else can crow and cluck.
This seems sensible prima facie, especially when Answers’ stated objective is to encourage participation.
But it’s flawed because it assumes that all participants will provide an equally valuable contribution — a conjecture that is laughable on its face, too.
If I walk into a Catholic Church and start handing out Planned Parenthood brochures, I’m asking for trouble, even if many people outside the church agree with me. If I prate on and on at a physics symposium about the force of a punch, they’re going to laugh me out of the room, and rightfully so.
People naturally congregate into homogeneous groups, based usually upon values, perspective and capabilities. They naturally take offense at the attempts of those outside that group to infiltrate or alter the group or its values.
One need only witness the furor over Proposition 8 to see this in action.
Yahoo! seems to believe that this dynamic can be offset by weighting the Community Moderation system. In other words, if they can find enough people who agree with their assessment of fairness and worthiness, then the tendency of similar people to congregate and impose groupthink will be counteracted by turning down their collective volume, or turning up the volume of outsiders.
Clearly, such a system is doomed to failure. For a forum to exist, it must have valuable content, and valuable content is produced by people who agree that the content is valuable.
You can’t have the “Animal Farm” scenario — where all animals are equal, but some animals are more equal than others — if your object is to have a truly inclusive community. In fact, you can’t have a universally inclusive community because it’s human nature to dislike people who aren’t like you.
Let The Roosters Crow
In other words, Answers wants it both ways. They don’t want l33t to have the ability to shout down n00bs, but they’ve created a weighting system that effectively makes their interpretation of the rules l33t, and all the participants who actually use their product, but don’t share their opinions of the rules, n00bs.
Common sense would dictate that persons who invest the time and effort to lead the community, and whose rating and rank indicate as much, should be empowered to lead the community in actuality.
In the real world, the biggest, baddest cock does rule the roost — at least, until a bigger, badder cock shows up.
That ought to be the case in Answers, and again, because of the weighting system for Community Moderation, it’s effectively the case.
But it is exceedingly clear Yahoo! does not share that assessment, nor have interest in acknowledging the realities that make such an assessment transparent. That, in turn, has led to the creation of a corrupt, fractious system that is universally distrusted, even by those who use it properly.
