Survey participants’ comments on the importance and/or wording of the 20 items to assess peer-review report quality
| Themes | Definition | Codes | Examples |
| Dependencies | Theme including codes on how the importance of an item depends on different factors (e.g., type of study, paper quality, type of journal, etc.) | Dependency on the type of study (n=34) | Depends on type of study. For systematic reviews of course fundamental. For other studies this will be more and more important for easier comparisons between studies and for quality improvement. It makes our work easier if the authors'compliance also improve. |
| Dependency on the paper quality (n=20) | This depends on the quality of the manuscript. Sometimes the quality is so low that a reviewer can highlight one or two major methodological flaws, which are sufficient to reject. | ||
| Dependency on the type of journal (n=19) | This depends on the journal's criteria. | ||
| Dependency on the author’s claim and impact of the study (n=7) | This depends on the claims made. | ||
| Importance | Theme including codes on the importance (or not) of an item. | Importance of the item (n=43) | This is absolutely key to the interpretation of the study. Unfortunately most reviewers, in my field, do not fully understand current (and correct) methods. |
| Importance of replication and conformation study (n=18) | Not always important to be original study as some are trying to duplicate findings from previous studies. | ||
| Importance of perceptions, opinions and experience (n=14) | Some comments will inevitably be opinion, regarding emphasis, values, writing style. | ||
| Importance of a high-quality review rather than on time review (n=13) | Better to have a late high quality report than a moderate quality report on time. | ||
| Improvements | Theme including codes on how an item is useful for both authors and editors in the peer-review process. | Useful for authors and editors (n=21) | It's important to make it easy for the editor and authors to understand the review, and for authors to respond. |
| Improving the manuscript (n=9) | Important when it will help improve the quality of the communication. Not necessary when it flows well. | ||
| Avoiding exaggeration and misinterpretation (n=8) | This is an area where the reviewer may have a valuable role in tempering an author's enthusiasm, hubris or bias. | ||
| Item | Theme including codes on the characteristics of an item. | Related to other item (n=43) | Yes, but it is confusing to separate this from the general strength and weaknesses. The question should be if the reviewer thinks that the message can (potentially) answer the research question. |
| Subjective item (n=22) | Too subjective! What is relevant to one person of field could be totally not-relevant to another. | ||
| Requirement (n=9) | It's an ethical requirement, and helps improve everyone's experience. | ||
| Reviewer | Theme including codes on the expertise and characteristics of a peer reviewer. | Reviewer’s expertise (n=148) | Some reviewers know about methods and some about content. It would be ideal to always have both, but that is often not the case. |
| Impossibility to be totally objective (n=35) | 100% objectivity doesn't exist. | ||
| Reviewer as an extra unpaid job (n=10) | For the most part, reviews are done on a voluntary basis. | ||
| Responsibility | Theme including codes on the editor and/or author’s responsibility to assess an item. | Editor’s responsibility (n=48) | In my experience this is usually picked up by the Editors and Associate Editors rather than the reviewers. |
| Joint responsibility (n=24) | I think this is the role of the editors as well as the reviewers. | ||
| Author’s responsibility (n=6) | Authors should already be doing this. | ||
| Structure and content | Theme including codes on the structure and content of a peer-review report. | Straight to the critical points (n=14) | Sometimes a succinct review is still helpful, if it cuts straight to the critical points. For example, if it is clear that a manuscript has major flaws, then a review that points out those flaws clearly and dispassionately would be very helpful. It would not necessarily need to delve into the finer details. |
| Unnecessary to provide evidence to each comment (n=10) | I don't think reviewers need to cite something for every point that they make. | ||
| Declaration of COI (n=8) | Peer reviewers should disclose COI. | ||
| Standard structure of a review (n=7) | I would suggest providing a template to reviewers. | ||
| Not necessary for all reviews (n=6) | Reviews come in all lengths and vary in detail. It is helpful to have some reviewers provide detailed information but not necessary that all do so. | ||
| Wording | Theme including codes on how to improve the wording of an item. | Wording of the item (n=110) | Rather than ‘The reviewer's comments are evidence-based’ I would suggest that the category should be: ‘The reviewer distinguishes between comments that are supported by evidence (and provides suitable citations) and those based on opinion or experience’. |