July 11th, 2008
There are two basic requirements of findings in order to call them scientifically obtained: they must be both reliable and valid.
Reliability refers to the stability or consistency of our finding over time. If we apply our test more than once, will we get the same result each time? If repeated applications of the test cluster closely, we say the result is reliable: every time I eat watermelon I enjoy it, so this is a reliable finding. Every time I take a certain online personality quiz I get a different result, so the finding is not reliable.
Validity refers to whether our results actually measure what we think they’re measuring. Checking the colour of your socks is not a valid manner of determining your intelligence, but scoring your results on a Stanford-Binet intelligence test probably is. (A finding cannot be valid unless it is reliable.)
And this is precisely why Wikipedia is not usable if your end is to claim your findings are scientific. It is neither reliable nor valid; more precisely, it may be reliable and valid, but we have no way of knowing this for sure. Entries can be changed at will by anyone, at any time. Therefore we cannot count on reliability. And there is no system of fact checking or peer review or editorial procedure which ensures that the information entered is correct; therefore we cannot count on validity. (Also, without reliability we can’t have validity anyway. We could stop at step one!)
Three arguments need to be dispelled here: One, you may say that just because it is possible for information to change or be wrong doesn’t mean that it in fact is either of these things. This is true – but so is the opposite. Since we have no reason to believe that reliability and validity are always preserved, we must be conservative and assume they have not been.
Two, you may say that many articles provide citations, which resolves questions of validity. But simply entering a citation does not mean your interpretation and summary of it in the body of the article are correct, or that in the time after you posted it that no one has altered the information without removing or changing the citation. Of course we can check the sources if we want to make sure, but this leads to the obvious conclusion that checking is in fact essential, otherwise how will we know if we have achieved reliability and validity? So we have not gained much, if anything, by using Wikipedia. We should have gone straight to the sources themselves in the first place.
Three, you may say no system is perfect and erroneous information finds it way into books and journals also. What differentiates these other sources from Wikipedia is a system of controls that minimizes error, and that seeks in good faith to preserve reliability and validity. Experts in methods are the gatekeepers to, for example, publish your findings in a journal. This system includes, for example, peer review. There is no comparable system for Wikipedia. It is true that informed people may correct errors as they come across them, but we can’t count on this informal version of The System to find each article (perhaps not even most articles). This is a problem of reliability. On the other hand, books and particularly academic journals are utterly beholden to The System (which brings problems of its own…). There is no natural selection of Wikipedia articles to weed out the stinkers. In fact, I suspect it is probably the case that the people best equipped to supervise and correct Wikipedia articles (such as academics and other highly experienced folks) simply don’t bother. Why would they? Not only is there no way to protect their work (meaning some jackass can just delete it any old time), but they know that other people know not to count on Wikipedia information – what is the incentive to put effort into it? If you’re an expert in a field, you’ll try to publish somewhere with credibility.
So what is Wikipedia good for? It is good for casual research when your interest is superficial enough that it doesn’t matter if it’s wrong or sort of wrong some (or more) of the time. It might be a good place to get oriented to a new problem, so you know what areas might be good to look into more seriously. It’s an interesting cultural phenomenon and should be studied in its own right for this reason.
What this means is that Wikipedia is extremely useful to the layperson who can’t be bothered with carrying out a literature review. And that’s fine! Most of us aren’t writing a thesis, we just want to find out some basics about the topic at hand. All that is required is assumption of the risks inherent in getting your information from a place with unknown reliability and validity. So if you’re serious about your research, if you really care about its accuracy and want to say your findings are scientific, then you need to ditch Wikipedia and get yourself a community membership to a university with online access to academic databases. And take a course in research methodology so you can intelligently assess what you read. That’s science.
Vex,
It seems to me that much of the important work of the academy is done in relation to new ideas, or at the very least newer perspectives on ideas.
Much ink is spilled in debating theories, for years, often decades before they become commonly accepted, and perhaps marginally “reliable”. Some of the commonly accepted science of the past, published in the peer reviewd articles of the time, is regarded as pure hokum today.
In my experience, a great many peer reviewed articles are little more than novel outgrowths of an authors sitz im leben, and sense of self-importance.
If entry level textbooks are the distillation of these articles, perhaps we can best akin Wikipedia to the bathtub gin equivalent — Slightly suspect, sometimes toxic, but most of the time, it gets you where you need to go.
~I.
P.S. Please blog about how you think the city should deal with the “homeless problem”.
I had to look up sitz im leben on Wikipedia, and if I understand it correctly (the entry was short), I agree! There’s this idea that capital-S Science related to capital-T Truth and both are somehow exempt from the influence of culture and place… maybe, but not necessarily, and as soon as you step just one inch outside of hard sciences it all goes to poo poo.
My entry here was clearly from a positivist, empiricist perspective. But there are other ways of knowing, other valid research methods.
And yes, I have recently come to realize how much distortion happens in the entry level texts in the service of simplification. Some things just don’t distill down very well!
Here’s what bugs me about those who share your attitude about Wikipedia: When having some kind of argument about a subject I will often look up something on Wikipedia to get a better grasp of it, and then point out the findings that Wikipedia has.
You, and others, then simply handwave that argument as utterly worthless because it came from Wikipedia.
I’ll then point out that what I quoted has a citation to a published scientific paper, and a link to that paper.
But it doesn’t matter, because it’s on Wikipedia and it’s untrustworthy, and “not science”.
Wikipedia articles are maintained by those who are experts in that subject. I’m sure Husband will soon be contributing to articles in his new field.
Any errors or defacement they do gain are quickly reversed — Wikipedia keeps copies of every single version of its pages so this is done with only a few clicks. I’ve done it myself for a few obvious defacements.
But even if 90% of wikipedia’s articles are accurate (the number is *far* higher, by the way) then the onus is on you to disprove what I’ve quoted, not to simply handwave it away as “yeah but that’s wikipedia”.
Also, if Wikipedia is wrong and you have a peer-reviewed paper that proves it’s wrong, you should get on Wikipedia and correct that article and cite the paper! Editing by everyone is not just a weakness of Wikipedia, it’s also a STRENGTH!
You’re right, it’s not appropriate to simply dismiss Wikipedia. It is indeed a source of much accurate information, and simply dismissing it because it’s from Wikipedia does not advance the discussion, whatever it is. THe website is highly useful for people like us and, in general, is probably accurate enough for the way we use it (mainly in casual conversation). The real objection is to using WIkipedia for real research (ie, academic or professional uses).
Are errors and defacements corrected quickly? I imagine the more egregious ones are. Many subtle inaccuracies or misinterpretations probably go entirely overlooked for who knows how long.
People maybe should correct Wikipedia if they know better, but do they? Some may, some may not. There is surely some selection bias at work here. I highly doubt wikipedia is worked on by all people in the know in their respective fields – some perspectives will get more air time and better research, and that may skew the type of information that is available.
Editing by everyone is not a strength when there is no accountability.
And please note, I did not just “hand wave” Wikipedia away. I wrote a good length explanation, with examples and even a couple of technical terms, explaining my position! That’s hardly dismissing without any grounds! Kindly don’t make sweeping generalizations about what I (and “everyone else” who “dismisses” wikipedia) do – particularly when you’re wrong and the evidence is right there on my blog!
I hope also you read the part where I wrote what I felt appropriate uses for Wikipedia are – did that part that sound like a hand waving dismissal? I also acknowledged in the post that it *may* be correct. The point I was making is we can’t *count on* it being correct, and therefore it is not useful for purposes requiring rigor of research. That means anything academic or professional.
It sounds like your real problem here is you’re sick of people dismissing your points simply because they were gleaned on wikipedia. While I agree in general that that sort of dismissing isn’t called for, maybe you should try upgrading your research sources so people can’t challenge them in that way.
The handwaving was done in previous discussions we’ve had, where I’ve pointed out some Wikipedia articles that backed me up and you waved your hand and went “pft, Wikipedia”.
Well, I shouldn’t have done that. Sorry.
I should have said, “Wikipedia – for whatever *that’s* worth!”
(By the way, Husband has already written a wikipedia article. He wrote it a couple years ago so I don’t know what shape it’s in these days, but anyway the original article was his. http://en.wikipedia.org/wiki/Intensive_short-term_dynamic_psychotherapy )