What Does “Peer Reviewed” Mean?

Articles about scientific matters (especially climate change) refer to studies with “peer reviewed” papers. Most readers assume that this means that the findings have been endorsed by qualified scientists and that they can be considered reliable.

The reality is that this is actually a low standard and is not an accurate guide to the reliability of the results.

A clear indication of the weakness of the system is that “spoof” papers were submitted, peer-reviewed and published. This is especially a problem in the “social sciences” where the appropriate use of PC language is enough to get a positive peer-review

I myself have published a number of papers (for a list, see here) , carried out peer reviews and chaired session at various conferences.

The system works roughly like this:

1. An organisation decides to host a conference (this may be a one-off or a regular event, often annual).
2. A conference organising committee is selected from notable persons in the organisation and the field of study, on a voluntary basis.
3. The technical sub-committee sends out a call for abstracts of papers to be presented.
4. The technical committee then grades the abstracts and selects those to be included to fill the available spots.
5. The authors of the selected papers present the papers at the conference and the written papers are included in the Proceedings of the conference. Each presentation of a paper is followed by a short question and answer period.
6. After the conference, papers for peer review are selected.
7. Each selected paper is sent to 2 or 3 volunteer reviewers who are known to be competent in the subject.
8. The reviewers send their comments and recommendations back to the organisation. There is often a questionnaire (for an example, see here).
9. For those papers selected for publication the reviews are sent back to the authors ( without indicating who the reviewers were) for correction/clarification.
10. After suitable editing, the paper is published in a “peer-reviewed” journal.

There are clearly several drawbacks to this system.

Firstly, it is not democratic. The committees are selected on the basis of reputation, personal relationships, availability and willingness to do unpaid work. It clearly discriminates against unconventional ideas or those that go against the beliefs (or vested interests) of the committee members. I have had a number of papers rejected from US based conferences, but have usually been accepted for Canadian conferences. On the other hand, papers submitted by the grad students of well-connected U.S. professors got accepted even though they did not present any significant new findings.

Secondly, the peers reviewing are on a volunteer basis. As the saying goes, ‘you get what you pay for”. One paper I wrote only had one reviewer and it was clear he/she did not understand the paper. For another paper, I had two reviewers. The first clearly understood what I had done and appreciated my novel solutions to certain problems. The second criticised the paper because I had not considered his specific areas of expertise (which in this case were totally irrelevant).

As a peer reviewer, the principal questions that need to be answered are:

1. Is the paper worth distributing to a wider audience? Is it relevant to the day’s problems?
2. Have the authors taken previous work into account? Are the references complete and relevant?
3. Is the description of the methodology adequate? Could someone else reproduce this work?
4. Are the limitations of the study clearly stated?
5. Are the conclusions and recommendations fully supported by this work? Are all relevant conclusions drawn?

Notice that none of this involves a detailed look at the data, the derivation of any equations or the calculations. This is where “peer-review” ends.

If the paper is truly significant, other researchers will look into the issues and build on the results (or even debunk them!) as necessary.

After that, the paper and work may be given general acceptance and its results fully incorporated into the body of scientific knowledge.