Does the quality of the peer review comments your journal receives vary more than it should?
Well-thought-out reviewer comments aside, many editorial teams find that they too often receive comments that are:
- Meandering and difficult to interpret
- Sparse or lacking the level of detail needed to be constructive
- Hyper-critical in incidental areas while missing the bigger picture
Such unfocused or insufficient reviewer comments can create snags in peer review and cause frustration for all parties involved. Many editorial teams struggle with wanting to ensure that they’re providing authors with robust feedback but feeling like reviewer comment quality is outside of their control. While you can’t guarantee that your journal will receive top-notch reviewer comments all of the time, there are some steps your editorial team can take to improve reviewer comment quality.
Use a peer review feedback form
One of the best steps editorial teams can take to improve the quality of reviewer comments their journal receives is to require all reviewers to complete a standardized feedback form. When reviewers are left to fashion comments from a set of instructions, no matter how thorough and well-formatted they may be, the likelihood of some reviewers misinterpreting or skimming over expectations is high. However, using a required, standard feedback form journals can ensure that all reviewers address the same key manuscript areas while deterring reviewers from giving tangential feedback. Reviewer feedback forms work best when they are automated using peer review software, so reviewers literally can’t submit comments without answering all of the necessary questions.
Find the right balance of feedback form questions and make them specific
The results of your journal’s reviewer feedback form will only be as good as its design. Once your journal has a feedback form set up, track the quality of responses submitted and make adjustments to the questions as needed to improve feedback outcomes.
The best feedback forms have a balanced number of questions that are easy to interpret. Aim to provide enough questions to adequately guide reviewers but not so many that you begin to overwhelm them. Remember, reviewer fatigue is real! Make questions specific so reviewers understand the goal of each question—this includes writing questions clearly and formatting questions to reflect the level of feedback needed. For example, journals can use open-ended questions for substantive feedback and Likert Scale questions for high-level assessments and recommendations.
Some common feedback form flaws to avoid are:
- Requiring nonessential questions: As noted, it’s important to have a balanced number of feedback form questions. Avoid filling up your form with questions you don’t need.
- Combining questions: Make sure that each question in your feedback form is only about one thing. When questions touch on multiple areas it’s more likely for reviewers to submit unclear or partial responses.
- Not asking for a publication recommendation: While the decision to accept a manuscript, reject it, or ask for revisions is ultimately up to your editors, asking reviewers for a direct recommendation can help ensure you’re interpreting their comments correctly.
It’s a good idea to also include an open-ended response field for comments to the editor at the end of your feedback form. This will ensure that reviewers are able to comment on all aspects of the manuscript that they think are necessary, even if your form doesn’t address them all. Over time, this field can help you to keep improving your feedback form questions by revealing any important assessment areas you’ve missed.
Limit revise and resubmit rounds
In addition to taking steps to improve initial reviewer comments, journals should set clear parameters for revise and resubmit rounds to ensure subsequent comments remain helpful and on track. As explained by former managing editor of Aztlán Journal of Chicano Studies, Wendy Laura Belcher, reviewer feedback can become less focused over multiple rounds of revisions and review. Belcher said that a common problem that editors should look out for is reviewers picking apart new areas of a submission in each round of review. This can turn into a frustrating cycle for authors and reviewers who just want to fulfill their respective expectations. Journals can avoid these situations by setting clear parameters for which aspects of the manuscript reviewers should be commenting on as well as a limit to the number of rounds of revision a manuscript can go through.
Help reviewers improve
Finally, your journal can help reviewers grow and improve the reviewer comments you receive by giving reviewers insight into the quality of their feedback. Journals can either provide reviewers with direct feedback on their comments, noting what was most useful and whether any comments were unnecessary or confusing, or they can send reviewers copies of author decision letters to provide insight into how their comments were interpreted and relayed to the author. Additionally, journals working with more early-career reviewers may want to provide some training materials such as a guide to writing constructive reviewer comments.
