Editing Encyclopedia:Historical archive/Policy/Approval mechanism (section)

===DWheeler's Proposal: Automated Heuristics===

It might also be possible to use some automated heuristics to identify
"good" articles.
This could be especially useful if the Wikipedia is being extracted to some
static storage (e.g., a CD-ROM or PDA memory stick).
Some users might want this view as well.
The heuristics may throw away some of the latest "good" changes, as long
as they also throw away most of the likely "bad" changes.

Here are a few possible automated heuristics:
* Ignore all anonymous changes; if someone isn't willing to have their name included, then it may not be a good change.  This can be "fixed" simply by a some non-anonymous person editing the article (even trivially).
* Ignore changes from users who have only submitted a few changes (e.g., less than 50). If a user has submitted a number of changes, and is still accepted (not banned), then the odds are higher that the user's changes are worthwhile.
* Ignore pages unless at least some number of other non-anonymous readers have read the article and/or viewed its diffs (e.g., at least 2 other readers).  The notion here is that, if someone else read it, then at least ''some'' minimal level of peer review has occurred.  The reader may not be able to identify subtle falsehoods, but at least "Tom Brokaw is cool" might get noticed.  This approach can be foiled (e.g., by creating "bogus readers"), but many trolls won't bother to do that.

These heuristics can be combined with the expert rating systems
discussed elsewhere here.  An advantage of these automated approaches
is that they can be applied immediately.

Other automated heuristics can be developed by developing
"trust metrics" for people.  Instead of trying to rank every article
(or as a supplement to doing so), rank the ''people''.  After all,
someone who does good work on one article is more likely to do good
work on another article.  You could use a scheme like
[http://www.advogato.org Advogato]'s, where people identify how much
they respect (trust) someone else.  You then flow down the graph to
find out how much each person should be trusted.
For more information, see
[http://www.advogato.org/trust-metric.html Advogato's trust metric information].
Even if the Advogato metric isn't perfect, it does show how a few
individuals could list other people they trust, and over time
use that to derive global information.
The
[http://www.advogato.org/code.html Advogato code]
is available - it's GPLed.

Another related issue might be automated heuristics that try to
identify likely trouble spots (new articles or likely troublesome diffs).
A trivial approach might be to have a not-publicly-known list of words
that, if they're present in the new article or diffs, suggest that the
change is probably a bad one.  Examples include swear words, and words
that indicate POV (e.g., "Jew" may suggest anti-semitism).
The change might be fine, but such a flag would at least alert someone
else to especially take a look there.

A more sophisticated approach to automatically identify
trouble spots might be to use learning techniques to
identify what's probably garbage, using typical text filtering and
anti-spam techniques such as naive Bayesian filtering
(see Paul Graham's "A Plan for Spam").
To do this, the Wikipedia would need to store deleted articles and
have a way to mark changes that were removed for cause
(e.g., were egregiously POV) - presumably this would be a sysop privilege.
Then the Wikipedia could train on "known bad" and "known good"
(perhaps assuming that all Wikipedia articles before some date, or
meeting some criteria listed above, are "good").
Then it could look for bad changes (either in the future, or simply
examining the entire Wikipedia offline).

:''A trivial approach might be to have a not-publicly-known list of words that, if they're present in the new article or diffs, suggest that the change is probably a bad one.''

:Why does it have to be not-publicly-known? [[User:Brianjd|Brian]][[User talk:Brianjd|j]][[Special:Contributions/Brianjd|d]] 10:22, 2005 Jan 29 (UTC)

:: I assume the idea is that if the list was known, it would be quicker for vandals to think of alternative nasty things to say.  But really we would want to be watching all the changes anyway, so that by watching bad edits that got past the filter one could update the list.  But really I think this naughtiness flagging is over complicated and not very useful.  More helpful might be a way to detect ''subtle'' errors: change in dates, heights, etc. that casual proofreaders would be less likely to pick up.  Rather than simply flagging the edit, maybe highlighting the specific change in the normal version of the article to warn all readers:  "Colombus discovered <span style="background-color:orange">Antarctica</span><span style="background-color:red">?</span> in 1492.". Reviewers would then be able to right click on the change and flag it as OK or bad, for example. [[User:Lbs6380|Luke]] 05:34, 15 Feb 2005 (UTC)