Controversial Finding or Established Fact?

How to decide what the facts really are
In replies to the Climate Change blogs in particular, many claims and counterclaims are made about what is true and what isn't. A small group of people claim that the entire anthropogenic global warming (AGW) "theory" is a hoax spread by a widespread conspiracy, and not a fact at all. A significant percentage (especially of American and Chinese people) have the impression that the scientific facts are still in dispute. The majority of the world population (especially of those who are climate scientists!) feel that the basic facts of global warming are now well established. How do you decide for yourself what is true or not?

How to know if it were a hoax spread by a conspiracy or not
There is a very simple test to see if it is possible for something which is untrue but widely believed -- a hoax - -- to result from a secret conspiracy. It is simply to ask: How many people would have to be in on it? If the answer is more than a dozen, you can safely forget it. That many people could never keep a secret, someone will talk. Or else they would never agree on what to do. It is impossible to coordinate exactly how to fake data and keep it a secret if more than one small group of people is involved.

Another test, if the above is not sufficient, is to ask what everyone involved gets out of it. If every person needed for the conspiracy to succeed doesn't clearly get a LOT of money out of it, or satisfy some equally important desire of theirs, they simply aren't going to be interested in taking the risk of being involved. Newspaper people are, for example, always looking for meaty stories to publish, and it is highly unlikely that all of them will keep quiet about a vast conspiracy unless a lot of money is in it for every one of them.

How to tell the difference between a controversial finding or an established fact
My impression is that a lot of people don't really know what is involved in deciding if a scientific report is considered a doubtful or controversial finding and what it takes for it to become accepted as an "established fact." As a scientist, this is something that I can answer. Since I am not sure what details you may or may not know, please skip over details you don't need.

You probably have already been told in sufficient detail about the scientist having a hypothesis, designing an experiment or observation to test it, etc. What you may not know is what is really required to have the results accepted by other scientists as established fact.

It is not as easy as it might seem. Just reporting it is not nearly enough. Knowing how scientific work is reported and judged by other scientists may help you to evaluate scientific findings yourself, as you must sometimes do to decide what the facts really are.

Reporting scientific findings
We'll start with what a scientist does after he finishes a segment of research. He may discuss his results with other scientists, informally or in presentations at meetings, but the work does not become an actual part of science until it is published in a peer-reviewed journal. The scientist therefore prepares a paper for publication in which to report his findings.

The paper must be organized in a particular format. All papers start with an Abstract, a paragraph to summarize the main points of the paper. The main body of the paper starts with an Introduction, which summarizes to the non-specialist what has been previously published on this subject, and sets out the reasons why the current research is needed. All facts stated must be backed up by references to published, peer-reviewed articles.

The Introduction is followed by a Methods section. In this section, the scientist reports the exact steps he took in doing the experiment or making the observation, analyzing the results, setting up a model, or whatever. The rule is that methods must be covered in sufficient detail that any competent scientist could exactly repeat the work. This also, of course, helps others evaluate whether the work was done well or not, and what the authors may have missed.

Results are reported next. Every single measurement may not be reported, but the appropriate averages and other statistics of the results are summarized. Individual measurements may be displayed in "scatter" plots that illustrate trends or correlations between different variables. Only results are presented in this section.

The Results section is followed by the Discussion, where the implications of the research can be detailed. Finally, the paper ends with a Bibliography containing the complete reference for every article mentioned in the paper.

Submission and review of the paper
The author sends the paper to the Associate Editor of the appropriate section of the chosen journal. This editor first determines if the paper meets the basic requirements of the journal and has sufficient merit to send out for review. If so, the editor selects two or three scientists who are experts in the same area that the paper covers.

Each expert, who does not know who the other reviewers are, performs a detailed, line-by-line review of the paper. A reviewer may suggest additional experiments that should be performed before the paper should be resubmitted. They may make any suggestions at all that they feel will improve the paper. They write up their review in detail and send it to the editor.

The editor makes the final decision on what action will be taken. This can include 1) rejection of the paper outright, 2) an invitation to redo or extend the work and resubmit it as a new paper, 3) acceptance for publication as long as the reviewers' suggestions are incorporated, or (4, very rarely) acceptance as written. The editor then sends a letter to the author, along with the written, anonymous reviews. If the paper is accepted with modification, the author can incorporate the suggestions by the reviewers, and resubmit the paper. The editor usually accepts the paper for publication at this point as long as it has been improved as required.

Even peer-reviewed, published articles require further confirmation
Important published findings are almost always confirmed by other researchers. No one believes anything important, anything new or controversial, until the results have been replicated, or confirmed in some other way, by scientists completely independent of the initial group. Scientists have a saying, "Extraordinary claims require extraordinary proof."

Scientists tend to be very skeptical people, always looking for the mistakes or omissions in other people's work. That is the very core of our training. Scientists get grants, after all, by identifying the mistakes or omissions in other people's work and applying to government agencies to do the work better. Scientists in their training spend a lot of time in "Journal Clubs" where they pick apart published papers, discussing every possible fault in the methods, results, conclusions, etc. You soon start to feel that you cannot trust anything that is published. And you don't, and you shouldn't, until you see that the results are supported by research published by other groups.

The existence of incorrect or untrustworthy articles
There are, of course, other factors that must be taken into account. There are some journals with poor reputations, who will publish almost anything. There are journals supported by special interest groups that cannot be trusted at all. There are scientists whose work is generally poor or sloppy, or who are covertly supported by special interest groups. Unfortunately, the identification of such scientists or journals is usually by word of mouth and often known only to specialists in the field. It is usually not something firm, well-established or important enough to justify a specific rebuttal article.

Mostly, working scientists just ignore such work. Even when a rebuttal is published, the original paper includes no reference to the rebuttal, which may be published in an entirely different journal, years later.

The unfortunate consequence of the existence of incorrect research reports
Unfortunately, the existence of articles or books that cannot be trusted is a boon to those who have their own agenda. "Believers" can always quote such work, and simply ignore any contrary evidence. Often, only specialists will know that the quoted articles are not to be believed, and exactly where the contrary evidence is published, if it is. This is why scientists usually use phrases like "'the overwhelming abundance of evidence" or "the consensus in the field," because you can always find a paper or two that contradict the vast majority.

Books are rarely thoroughly reviewed and any fact in them should be believed only to the extent that it is backed up by reference to peer-reviewed articles in reputable journals. A good author will always cite such sources liberally. Books that are self-published (by "Vanity Press" publishers) or whose publication is financed by special interest groups are never to be trusted. Such books are, of course, the very ones that believers use to support their particular point of view -- or that they publish themselves.

This blog
The above cautions apply to this blog, of course. It is not peer-reviewed. There are some real advantages to this, mostly being we don't have to do the scientific work ourselves and wait for the reviewers to mangle our pet ideas. (We have done enough of this in our life!) The blog is not a publishable paper, anyway, it is more a science newspaper combined with a plotting calculator. The views expressed are solely those of the science editor. Don't believe anything in here, at least don't bet on it, until you see it backed up by peer-reviewed articles.

DETAILS
Why would you think, in this modern era, that we would feel it necessary to explain how scientists know what they know? Because for about 20 years now there has been a very active campaign to con the general public into distrusting scientists and scientific findings in general. One of the most successful techniques has been turning generally accepted findings first into "theories," and later into questionable theories or "junk science." This campaign has been waged through individuals and front organizations funded by special interest groups including big tobacco, coal and oil companies, etc. For a comprehensive discussion see Hoggan and Littlemore, "Climate Cover-Up: The crusade to deny global warming."

COMMENTS
March 17, 2010. LouiseS. in Seattle said...

Very nice job explaining this process!!

How about a new section where you post biographies of scientists? Perhaps if they attained just a bit of the hero status we give to people (actors) who read other people's writings, then they would be believed more.

Keep up the good work.

March 23, 2010. David Mills replies...
That is a great idea, but I think everyone, scientists included, has to first recognize that people are being actively conned into distrusting scientists and science findings in general by people and organizations covertly supported by special interest groups. The overall results of such distrust of scientific findings is becoming very dangerous to society as a whole.

Maybe if we talk about the real conspiracies enough, some investigative journalists will be moved to expose the existence and tactics of such groups. And we can hope that their newspapers will print it on the front page when they do.