Peer review

What a peer reviewed journal means

A peer reviewed journal, also called a refereed journal, is one that routes research articles to outside experts for evaluation before accepting them. Three things follow from that definition and each of them is regularly misread.

• The review is of the manuscript, not the researcher. Reviewers assess the question, the method, the analysis and whether the conclusions exceed the data. They almost never repeat the work.
• The journal is peer reviewed, the individual item may not be. A peer reviewed journal also prints editorials, book reviews, letters, obituaries and news items, and those go through an editor rather than referees. The label attaches to the research articles.
• The reviewers are working from what they were sent. In most fields that means the text, the tables and the figures. Raw data, code and lab notebooks are usually not supplied, and a reviewer who cannot see them cannot check them.

The practice is old and it was never uniform. The Royal Society's Philosophical Transactions began in 1665 and the society established a Committee on Papers in 1752 to vet what it published, but systematic external refereeing spread across the sciences only in the twentieth century, and several long established journals adopted formal outside review remarkably late. Peer review is a convention that hardened, not a law that was passed.

How the peer review process works, step by step

The peer review process runs in 6 steps, and understanding them tells you what each stage can and cannot catch.

1. Submit. The authors upload the manuscript and a cover letter to the journal.
2. Screen. An editor checks scope, format and basic quality, and rejects a large share here without sending it out at all. This is desk rejection, and at selective journals it removes most submissions.
3. Assign. The editor invites reviewers, typically 2 or 3 per manuscript, chosen for subject expertise. Many decline, so this stage alone can take weeks.
4. Review. Each reviewer reads the manuscript and writes a report, recommending acceptance, minor revision, major revision or rejection, with specific criticisms attached.
5. Decide and revise. The editor weighs the reports, sends them to the authors, and usually asks for revisions with a point by point response. A paper may go around this loop two or three times.
6. Accept and publish. The accepted version is copyedited, typeset, given a digital object identifier and published, often with the received, revised and accepted dates printed on it.

Two features of that process matter more than the diagram suggests. Reviewers are volunteers doing this on top of their own work, unpaid, usually anonymous, and often reviewing several papers a month. And the whole sequence commonly takes between three months and well over a year, which is why preprint servers exist and why a preprint is a paper that has not yet been through any of the above.

Single blind, double blind and open review

Single blind, double blind and open review differ in who knows whose identity, and each trade off is real.

Psychology, economics and computer science lean toward double blind for conferences and many journals; much of the physical and life sciences remains single blind. No model has been shown to make review dramatically more accurate. They redistribute a known bias rather than removing it.

How to tell whether an article really is peer reviewed

To tell whether an article really is peer reviewed, check the journal rather than the article, and check it somewhere the publisher does not control. Run these 6 checks, in ascending order of effort.

1. Look for the dates. A research article from a refereed journal usually prints "received", "revised" and "accepted" dates. A gap of days between received and accepted is a warning; a gap of months is normal.
2. Read the journal's instructions for authors and its information for reviewers. A genuine journal describes its model, its criteria and its appeals process in detail. A fake one has a paragraph of adjectives.
3. Use a directory as your checker. Ulrich's Periodicals Directory records which serials are refereed and is the standard tool librarians use for exactly this question. The Directory of Open Access Journals lists open access titles that meet its published criteria.
4. Check the library database filter, which most academic databases offer as a peer reviewed or scholarly limiter. Treat it as a first pass, not proof: the filter marks the journal, so an editorial from a refereed title can still slip through.
5. Look at the editorial board. Named people, at named institutions, who really work in the field. Boards listing scholars who never agreed to serve are a documented tactic of predatory journals, which charge a publication fee and perform little or no review.
6. Search for the paper's record. A digital object identifier, an indexed record in a recognized abstracting service, and any published corrections or retraction notice attached to it.

A claim on a website that a paper is peer reviewed is not evidence that it was. The claim costs nothing to make. This is the same discipline described on Evaluating sources: go and find out what independent parties say about the container, rather than reading the container's account of itself.

What peer review does not prove

Peer review does not prove that a finding is correct, and being clear about that is what makes the process useful rather than magical. Peer review has 5 limits worth knowing, and none of them is a scandal.

• It checks for detectable problems. Reviewers catch an unsupported inference, a missing control, a wrong statistical test, an overreaching abstract. They do not catch a subtle measurement error in an instrument they cannot see.
• It rarely detects fabrication. Review assumes the data are what the authors say they are. Fabricated results are usually caught later, by readers who could not reproduce them or by image and statistics screening.
• It is thin. Two or three readers, unpaid, working in evenings, on a paper that may have taken a team three years. The asymmetry is structural and no reform has removed it.
• It is inconsistent. Send the same manuscript to two competent reviewers and you can get accept and reject. Agreement between reviewers is modest in every field where it has been measured.
• It does not test whether the result holds. That is what replication does, and the published literature is not a fair sample of everything that was tried. The reasons are set out on Publication bias.

The journal impact factor, which is a citation average for a whole title, is not a measure of any individual paper either. A high impact journal publishes papers that are later retracted, and a modest specialist journal publishes work that stands for forty years. Judging a paper by the letterhead is the same error as judging a person by the accent.

How to read a peer reviewed article once you have one

To read a peer reviewed article properly, treat it as a primary source about its own experiment and nothing more. Go to the methods before the abstract: what was measured, on how many subjects or samples, over what period, compared against what. Then go to the results tables and see whether the numbers there match the sentences in the discussion, because the discussion is where authors are allowed to speculate and where the qualification most often quietly drops. The difference between what a paper reports and what a later article says it reports is covered on Primary vs secondary sources.

Confirmation bias does its most efficient work at this exact moment. A paper agreeing with you gets read at the abstract and stops; a paper disagreeing gets read at the methods and gets criticized. You can measure this in yourself, which is where the test comes in.

So, before you cite a peer reviewed article, ask: would I be applying this same level of scrutiny to the methods if the result had come out the other way? If the honest answer is no, go back and read the methods again. The review the paper received is finished; yours is the one still available.

Where to go next

SOURCESEvaluating sourcesHow to check where a claim came fromSOURCESPrimary vs secondary sourcesThe difference, with examplesSOURCESMedia literacyHow to read a claim in the newsBIASESConfirmation biasDefinition and examples