Have you ever received peer review feedback (on time!) only to find that it’s incomplete because the reviewer failed to address one of your journal’s primary manuscript assessment criteria? Or have you ever received a review report recommending manuscript revisions with lean or vague comments, giving authors little to go off of?
Handling incomplete or unclear reviewer feedback can be challenging, especially if it becomes a recurring problem for your journal. Having to go back to reviewers to ask for clarification takes up valuable time and is often frustrating for all parties involved. As an editor, you can’t control what reviewers write in their comments, but setting clear expectations and creating a structure for reviewer feedback can make a big difference in the quality of responses you receive. That’s where implementing a structured peer review process comes in.
Whether you’re managing an established journal or launching a new title, creating a structured reviewer feedback form can help make your peer review process more transparent, improve the experiences of the authors and referees you work with, and elevate the quality of the research you publish.
In this guide, we’ll cover the what, why, and how of structured peer review. To jump to specific sections, use the quick links below.
Table of Contents
- What is structured peer review?
- Implementing a structured peer review process
- Monitoring and refining the process
- Putting it all together
What is structured peer review?
Structured peer review is an approach to assessing academic manuscripts that uses predefined criteria. Unlike unstructured peer review, where editors ask reviewers to provide open-ended comments on the quality of a manuscript with little guidance as to what to focus on, in structured peer review, all reviewers are given the same set of evaluation questions, prompts, or scoring rubrics. The primary goal of structured peer review is to help ensure peer review reports are as comprehensive and consistent as possible.
Benefits of structured peer review
The potential benefits of implementing structured peer review for journals are many, including:
- More consistent review reports: Structured peer review ensures all reviewers assess manuscripts based on the same standards, which can also help ameliorate subjectivity in review reports.
- Improved reviewer efficiency: Structured feedback templates let reviewers know which aspects of manuscripts they should address, which can help save them time by eliminating guesswork around what to focus on.
- Better editorial decision-making: When reviewer feedback is structured, editors have more comparable data points across peer-review reports, making it easier to identify the strengths and weaknesses of a manuscript relative to journal criteria.
Recent studies on structured peer review have yielded promising results, including a 2024 report on a pilot of structured peer review at 23 Elsevier journals by Mario Malički, Associate Director of the Stanford Program of Research Rigor and Reproducibility, Co-Editor-in-Chief of Research Integrity and Peer Review, and EASE Peer Review Committee Chair, and Bahar Mehmani, Peer Review Innovation Lead at Elsevier, and part of the EASE Peer Review Committee. Their findings included that reviewers using structured feedback forms made the same initial recommendation on whether to accept, revise, or reject a paper 41% of the time, compared with 31% for unstructured peer review reports the journals received. Reviewers’ comments were also in agreement about specific parts of a manuscript up to 72% of the time.
Implementing a structured peer review process
Of course, there is no one-size-fits-all template for structured peer review. Your editorial team will have to carefully consider the critical evaluation requirements for the research you publish based on your journal’s discipline and the subject matter you cover, among other factors.
However, there are guiding principles for preparing structured peer-review feedback questions that all journals can look to, which we outline below.
What to cover in structured reviewer feedback forms
As advised by Mario Malički in an article for Nature discussing the findings of his and Mehmani’s research on structured peer review and subsequent recommendations, at the highest level, structured peer review questions should “focus on methodological, analytical and interpretative aspects of a paper.” Malički said, “other aspects of a study, such as novelty, potential impact, language, and formatting, should be handled by editors, journal staff, or even machines, reducing the workload for reviewers.”
Key criteria to consider covering in structured peer review questions include:
- Comprehensiveness of the literature review per the latest research in the area
- Methodological rigor and quality of statistical analysis (as applicable)
- Clarity of the interpretation of results and study conclusions
- Perceived replicability and/or reproducibility of the study
- Relevance to the journal’s scope
- Significance of the findings and level of contribution to the field
In his Nature article, Malički fleshed out what a structured reviewer feedback form might look like. He said, “for example, editors might ask peer reviewers to consider whether the methods are described in sufficient detail to allow another researcher to reproduce the work […] and whether the authors’ interpretation of the results is supported by the data and the study methods.”
Malički added that editors should also ask reviewers to consider whether they can sufficiently address each feedback form question and direct them to declare if they feel they lack the expertise to assess any aspects of the manuscript.
Letting reviewers know that you want and expect them to share if they have any hesitations about their level of knowledge in a particular area can help ensure you receive forthright assessments and know when additional input is needed.
Question formatting options and considerations
Once you’ve outlined the manuscript assessment areas you need reviewers to address, you can determine the best way to present your feedback questions. There are three primary formatting options to consider.
- Required overall assessment questions: usually an overall rating of the manuscript’s quality (e.g., all reviewers must score it on a rubric of 1-5, with five being the highest rating) and a required publication recommendation of Accept, Reject, or Revise & Resubmit
- Likert scale questions: reviewers are asked to rate their level of agreement or disagreement with a statement, usually on a scale of 5 or 7 points (e.g., strongly agree, agree, neutral, disagree, and strongly disagree)
- Open response questions: reviewers are given a box to write comments on specific questions
The first option, “required overall assessment questions,” encompasses general questions we recommend all journals ask reviewers. A manuscript quality rating will give your editors a quick snapshot of the reviewer’s overall confidence in the paper, and asking reviewers directly whether the manuscript should be accepted, rejected, or sent back for revisions will ensure you know precisely what they recommend.
The other types of questions — Likert scale and open-ended — each serve a distinct purpose. Likert scale questions are best for assessing high-level manuscript criteria that require a certain level of agreement, such as “To what extent do you think this manuscript offers a significant contribution to its field?” (to a great extent, somewhat, very little, not at all).
Open-ended questions enable you to ask reviewers to think critically about specific aspects of the manuscript and to provide constructive feedback for the author(s). For example, you could ask reviewers to comment if any elements of the research methodology appear flawed.
Quick note for Scholastica users: Journals that use the Scholastica Peer Review System can easily set up a reviewer feedback form with all of the above question types by following the steps in this help document. You’ll learn how to customize your reviewer feedback form in Scholastica, including adding, removing, and changing the order of questions and customizing your publication decision recommendation options.
Tips for developing structured reviewer feedback questions
When designing a reviewer feedback form, it can be helpful for your editors to follow an inverted pyramid question structure. By that, we mean starting with fundamental assessment questions and then working towards “nice to have” questions.
That way, you can begin with the criteria that need to be met for the manuscript to move forward and then ask reviewers to share more details about their recommendations and comment on other aspects of the submission, like writing quality or style, if you want to give the option (remember, you don’t have to require responses to everything).
As discussed, peer review criteria will vary by journal. Examples of possible questions applicable to a broad range of manuscripts, per the highest level assessment areas to focus on covered earlier, include:
- Have the authors clearly articulated the primary purpose/objective(s) of the manuscript?
- Is the literature review comprehensive, up-to-date, and appropriate for the manuscript topic?
- Are the interpretation of results and study conclusions supported by the data and/or study design?
Keep in mind that reviewers are busy people! If you require them to answer too many open-ended questions, you may end up with a lot of short responses that are less helpful than more thorough responses to fewer questions would have been. So, we recommend that editors aim to provide a framework for addressing essential aspects of manuscript quality in as few questions as possible.
For examples of structured peer review feedback form questions at other journals, check out this bank of questions used by Elsevier journals and these structured peer review questions from mSphere, an American Society for Microbiology journal.
Publish your review questions to make the process transparent
Once you’ve finalized your structured peer review feedback form questions, consider publishing them on your journal website, as recommended by Malički.
Publicly posting the questions will help make your journal’s peer review process more transparent for authors, reviewers, and readers. It will also enable authors to check their manuscripts against your journal’s evaluation criteria and reviewers to see exactly what you’ll expect of them before taking on an assignment. That way, authors will have the opportunity to refine their manuscripts before submitting them, and reviewers will be able to make more informed decisions about whether they have the bandwidth and level of expertise needed to contribute meaningful feedback.
Going a step further, Malički also suggested that journals consider publishing review reports alongside accepted articles. He noted, “this would allow readers to judge for themselves how a paper was assessed, and would enable researchers to study peer-review practices.”
Monitoring and refining the process
Finally, make a plan to assess the outcomes of your structured peer review process and make adjustments as needed. Think of your reviewer feedback form as a living document open to new iterations — that’s the only way to optimize it!
Consider asking for feedback
One of the best ways to assess the quality of your reviewer form and overall peer review process is to collect feedback from reviewers, authors, and editors during your initial structured peer review pilot and related data points to identify areas for improvement. From there, consider sending periodic peer review experience check-in surveys and regularly reviewing journal peer review performance metrics.
Key areas to monitor include:
- Reviewer experience: Are reviewers finding your feedback template easy to use?
- Author satisfaction: Are authors satisfied with the quality and detail of the feedback they are receiving?
- Decision times: Is your structured peer review process contributing to speedier editorial decisions?
For more comprehensive guidelines on assessing peer review quality, check out this set of recommendations from EASE, which Malički led the development of. You can learn more about that process in our past interview with him.
As you gather peer review data and input on your peer review process and feedback form (meta, we know!), use it to refine your manuscript assessment criteria, questions, or reviewer training offerings as necessary.
Putting it all together
Ultimately, your peer review process will be as structured as you make it, which will directly affect the clarity and consistency of the review reports you receive.
Asking reviewers to comment on a manuscript’s quality without giving them specific areas to address is like dropping them off in the woods and telling them to navigate to a destination without clear directions. Some will find it via more direct routes than others, and some may not get there.
Whereas providing a structured peer review feedback form is akin to giving reviewers a series of trail markers on their forest journey. While some reviewers may still go slightly off the trail with their comments, they will ultimately have to take the same general path as everyone else to address your required manuscript criteria and reach your “review destination.”
We hope you’ve found this guide to structured peer review helpful! If you have any questions, feel free to add them to the comments section below. Happy editing!
