How to review a paper

The more you publish, the more you will be asked to review papers. Below are some thoughts on how reviewing manuscripts fits into your academic career. The comments refer to the “traditional” publishing model of anonymous peer review. Recent developments in open-access / transparent peer review are considered at the end.

Try to avoid ending up on this list.

Requests to review papers will arrive once you have published in a particular academic area. You may be directly invited by an editor. Early in your career, your mentor may ask you to review a manuscript that they have been invited to review. In the latter case, you should:

• Ask your mentor to disclose your role in the review to the editor of the journal.
• Use the opportunity to discuss the paper with your mentor and revise the review together so that you have some feedback regarding your efforts.

How do you decide if you should agree to review a manuscript when invited? You may wish to review a paper so that you:

• stay current in your field
• maintain the accuracy of scientific claims in your field
• develop a favorable reputation with the Editors of a particular journal

You should decline to review a paper if:

• You do not feel you can provide a fair and unbiased review. For example, you may be conflicted because you are preparing to publish similar work, and your desire to not be “scooped” will color your review.
• You cannot provide a timely review. Journals remember your performance as a reviewer and it is possible to develop a reputation as a bad reviewer if you take on assignments and then complete them late.
• You are not interested in the study. You should only review the paper if you would have been interested in reading it if it had only been brought to your attention after publication.
• You are not capable of providing a thorough review. If you find that you are not capable of properly evaluating portions of the paper because of inadequate knowledge, you should write the editor and let them know that you have this limitation. The editor may wish you to review the paper regardless, with the explicit disclosure that you are unable to evaluate some aspect of the work.

While there is not a single correct way to structure a written review, there is commonly used style. There are also some common errors to avoid. The structure of a review is described here, and below is an example review that follows this format.

I advise a particular philosophical stance to take towards reviewing papers. Ultimately, the paper and its contents belongs to the author, not to you. Have clear in your mind and in your review critiques that concern matters of empirical fact–for which you should insist upon correction–and matters of opinion and interpretation for which you should advise, not command. You should ensure that the methods are fully and accurately described, and that concrete claims regarding the results are supported by the experiments in the study.

Before beginning to write your review, you should read through the paper twice. On the first pass, read through the entire document without stopping to take notes. This will help you avoid writing critiques of things that are then later fully addressed in the paper. On the second pass, identify issues you wish to address in your review, making note of the relevant page and (if possible) line number.

• Your summary - Begin with a summary of the methods and claims of the paper. The purpose of this summary is to communicate to the authors and the editor your understanding of the primary claims of the paper.
• Statement of the strengths of the paper - Even a weak paper has something to recommend. This might be the importance of the question that is being addressed, an aspect of the methodology, or the scholarship with which the authors engaged the literature.
• Overview of your critique - Here you pivot to your critique of the paper. If there is an organizing structure to your critiques, offer that structure here. You might write, for example, that you have methodological concerns regarding one experiment and more general critiques regarding the interpretation of the results from all experiments.
• Major issues, numbered - Now provide an organized list of your major critiques. Generally, issues that fall in this category impact your overall judgment of the quality of the paper. Moreover, these are issues that would prompt you to review a revised version of the manuscript to see how they were addressed.
• Minor issues, numbered - List here small issues that do not require your attention to address. This would include typos, word choice, and formatting errors
• References cited - If you refer to any papers in your review that are not cited in the original manuscript, include the full citation here.

Things to avoid in your review:

• Your opinion regarding if the paper should be published. You have the opportunity to provide this opinion through the journal web site. Placing this in the body of your review can complicate the job of the editor if they do not follow your advice.
• Insulting statements, including disparaging comments that assume the author is not a native speaker of English. Anonymous forums can produce obnoxious behavior. Hold yourself to a dual standard: 1) Write your review as if it is publicly available with your name attached, 2) Review a paper as you would hope to be reviewed. If you do find the writing to be poor or difficult to understand, you can communicate this in a gentle manner. For example “At points, I found the style of writing difficult to understand. Some sentences could benefit from proofreading or editing for grammar.”
• Gratuitous requests for citations of your work. You have been asked to review the manuscript because you have performed work in this area. It is sometimes the case that work of yours is directly relevant to the claims being made in the paper and this is a legitimate request. You should, however, feel sheepish about doing so.
• References to papers without providing a full citation. Don't be this person: “The authors should cite the work of the Smith lab”. This is lazy. If you are going to cite a paper in your review, be precise regarding the paper to be cited and the aspect of the paper you are drawing to the attention of the authors.

An encouraging change is taking place in peer review, particularly in “open access” journals. Increasingly, peer review is being conducted in an on-line forum, in which the comments of the reviewers and the authors are made public. These systems encourage discussion amongst the reviewers, and between the author and reviewers.

The same principles of good peer review described above apply in this new setting. Perhaps even more so, as the veil of anonymity is slowly being withdrawn.

<blockquote>Smith and colleagues test if acquired prosopagnosia is associated with deficits in reading single words and processing word style. These tests are motivated by neuroimaging (and to a lesser extent neuropsychological) work that suggests ventral temporal cortex specializations for face and word processing are at least adjacent, and may be intermixed. They studied 11 prosopagnosic patients, 6 with unilateral right lesions, and 5 with bilateral lesions. Experiment 1 tested for a word-length effect in reading time (a typical finding in alexia). A modest worsening of performance was found in some bilateral lesion subjects. In Experiment 2, subjects sorted index cards by either the meaning of a word, ignoring font / handwriting style, or sorted by style (with meaning held constant). No impairments in word meaning were found in the prosopagnosic groups. Almost all the prosopagnosic subjects had some impairment in sorting words by style. The authors suggest that their results support the necessity of right hemisphere structures for representation of word style, and more generally a shared (or at least closely proximate) neural resource for face and word perception.

The study builds upon prior, careful work of this group in measuring the perceptual abilities of patients with ventral cortical lesions. An important strength of this study is the relatively large number of prosopagnosic subjects studied, with an equal distribution of unilateral right and bilateral lesion sites. The impairment in sorting words by style is well established by the data, and seems to be a secure property of this patient group. In my comments below I consider some limits to the inferences that the study may draw, and some minor requests for additional experimental details.

The authors are quite aware of the limits of the claims that they are able to make regarding the effects of lesions at different locations and mental operations. The discussion is sensible and signposts these limits. To the points they already consider, I would add the following:

1. A general effect in the studied patient population of impaired perception of word style was found. Given the current model of extrastriate cortex organization, one would predict that right fusiform gyrus lesions would impact this aspect of word perception. However, an impairment was found in patients with right anterior temporal lesions for this putatively perceptual task (sorting by font). I suggest that the authors discuss this property of the data.
2. More generally, a limitation of the study is that, for the key measure the authors examine, essentially all the subjects showed some impairment. It is difficult to distinguish the possibility of a specific right fusiform effect from a general effect of having a ventral lesion. It would be reassuring to see data from subjects with cortical lesions who do not show this deficit.
3. Further, the specific association between the right fusiform gyrus and representation of word style would be greatly strengthened by examining a group of patients with homologous left hemisphere lesions and determining if they do or do not show these effects. Naturally, these patients would likely not have prosopagnosia, but such data would support the dissociation the authors are seeking.
4. Some impairment in object perception was found for the patients with bilateral lesions on neuropsychological tests (Table 2). The authors should consider the possibility that impairments in the bilateral group from Expt 1 might be related to a more general impairment in early stage processing (e.g., something closer to an apperceptive agnosia).

Smaller points:

1. Please include more details on the right / left hemianopia simulation data. How is eye position tracked? How does this simulation in controls compare to the performance of hemianopes who have more anterior pathway lesions (optic tract / V1)?
2. In Experiment 1, did mean response time include incorrect responses? Please indicate in the methods.

</blockquote>