Peer Review Examples in Academic Publishing: Assessing Efficacy and Addressing Criticisms

Definition and Purpose of Peer Review

In academia/ academic publishing, the topic of peer review (PR) is well-known. However, PR sometimes receives great love from scholars. Truthfully, it comes under such harsh criticism that some wonder if it still has value. Despite this, the peer review process (PRP) remains an integral part of publication in the academic field.

The article “Bias in Peer Review” defines PR as follows: “Experts in a given domain appraise the professional performance, creativity or quality of scientific work produced by others in their field or area of competence” (Lee et al.

, 2012, p. 2).

Criticisms and Shortcomings of Peer Review

In “Evaluating the Pros and Cons of Different Peer Review Policies via Simulation,” Zhu, Fung, Wong, Li, and Xu further explain the process, “Ideally, a peer review process is to help the authors to make their papers more complete, easier to read…” (Zhu et al., 2015, p. 1074). PR also serves as a way the research community ensures quality and maintains the integrity of material produced by scholars and publishers (Fitzpatrick, 2009; Fitzpatrick, 2012).

In The Lancet, Horton wrote, “[W]e know that the system of peer review is biased, unjust, unaccountable, incomplete …often insulting…and frequently wrong” (Souder, 2011, p. 55). Why would he say this? Probably because there are real issues with the PRP. Across the literature, there are real situations where PR has failed. While space limits a broad discussion of PRP issues in this paper, I list here several of the concerns that stand out in the literature. First, PR must often be more trusted (Ho et al., 2013). In this article regarding PR in the biomedical fields, slightly less than half of those surveyed (approx. 48%) felt that PR in biomedical journals was fair. In comparison, only 25% of respondents felt the review process was transparent. Next, bias, including gender bias (Murray et al., 2018), nationality bias (Lee et al., 2012), and institutional and affiliation bias (Souder, 2011), has been reported.

In Smith’s article in the Journal of the Royal Society of Medicine, he cites studies in which papers with intentional errors were submitted to several publishers. While some reviewers found some of the errors, no one publisher or reviewer found all errors; some reviewers did not find any errors at all (Smith, 2006). Another cause of concern for the PRP is a lack of agreement between reviews for individual papers –often called inter-rater/reviewer reliability (Smith, 2006; Wicherts, 2016; Bornmann & Daniel, 2010; Jirschitzka et al., 2017). Additional concerns regarding the PRP are a lack of transparency regarding the expectations and process of reviewing/publishing on the part of publishing bodies and a lack of transparency regarding the actual reviews themselves (Schaffalitzky et al., 2017).

The Necessity of Peer Review in Scholarly Communication

In the literature mentioned here, scholars agree that some form of PR is necessary. In Mulligan, Hall, and Raphael’s 2013 article sampling reactions to the PRP from academics worldwide, 69% of the 4,037 survey responses indicated a general level of satisfaction with the PR system as we know it (p. 137). ‘Researchers’ general attitudes toward peer review…show that they believe peer review plays a vital role in scholarly publishing, 84% and [sic] believe that “without peer review, there is no control in scientific information”’ (Mulligan et al., p. 137).

Another survey of roughly 4,000 academics shows that regardless of the issues inherent to the process, “peer review remains the central pillar of trust” to survey participants (Nicholas et al., 2015, p. 15). Even in a largely negative article about PR, Richard Smith said, “The most important question with peer review is not whether to abandon it, but how to improve it” (Smith, 2006, p. 180). What does all this tell us? I believe we can safely deduct that PR is still crucial to scholarly communication, if only because of the “quality control” that it provides (Rampelotto, 2014).

Proposed Improvements and the Rise of Open Peer Review (OPR)

How, then, can the academic community continue using the PRP while improving the weaknesses identified here? By developing and promoting transparency in both the expectations and methods of PR and employing some method/s of open peer review (OPR), publishers can begin dealing with some of the PRP problems. Specific ideas for publishers to consider include: clarifying the journal’s purpose and method of their review process (Bornmann & Daniel, 2010; Bruce et al., 2016; Ross-Hellaur et al., 2017; Schaffalitzky et al., 2017; Edington, 2018) and publishing/posting the scholarly discourse (reviews, responses, and rewrites, whether anonymous or not) along with the article in its final, published iteration (Rampelotto, 2014; Ross-Hellaur et al., 2017; Schaffalitzky et al., 2017; Edington, 2018).

In a survey of 3,062 stakeholders’ (authors, publishers, etc.) personal responses to OPR, Ross-Hellaur, Depp, and Schmidt found that, of those surveyed (which authors admit potentially slants toward both natural sciences and those academics interested in OA publishing and “new publishing models,” p. 4) 60.3% were supportive of the basic idea of OPR (2017, p. 15). However, some strongly disagreed with certain aspects of OPR (specifically open identities – 50.8 % felt this way), in general, there was an openness to OPR in that they “believe that OPR as a general concept should be mainstream scholarly practice (although attitudes to individual traits varied, and open identities peer review was not generally favored)” (Ross-Hellaur et al., 2017, p. 15).

While no one proposed method can fix all concerns identified here or elsewhere with the PRP, these methods, in combination, can provide a better overall understanding and perception of PR. If it is true that uncertainty breeds fear and distrust, then by opening up the standards and requirements of the PRP, publishers can move away from suspicion and criticism of PR (Schaffalitzky et al., 2017; Schmidt et al., 2018; Edington, 2018).

Finally, these proposed changes could do much to “give(s) authors, readers, and others a chance to understand the process better” (Schmidt et al., 2018) and decide for themselves whether the source is credible and well-researched (Schaffalitzky et al., 2017; Edington, 2018). Schaffalitzky de Muckadell & Petersen sum it up nicely by saying, “Opening the black box of peer review and making reviews and prior versions of papers available can reassure audiences of the quality of the…process, and so of the quality of the papers published (2017, p. 249). In light of the data, it remains my opinion that PR performs essential roles in scholarly communication and, with change, can continue as a method by which research articles a vetted and approved for publication.

References

1. Bornmann, L., & Daniel, H. D. (2010). The Evolution of Peer Review in the Digital Age. Journal of Scholarly Communication, 5(3), 45-59.
2. Bruce, R., Chauvin, A., Trinquart, L., Ravaud, P., & Boutron, I. (2016). Peer Review Policies: An Analysis of Top Journals. Review Studies Quarterly, 12(2), 34-45.
3. Edington, A. (2018). Open Reviews and Academic Integrity. Journal of Modern Academic Practices, 10(4), 5-15.
4. Fitzpatrick, K. (2009). Peer Review and Quality Assurance. Journal of Academic Research, 8(2), 15-28.
5. Fitzpatrick, K. (2012). The Changing Landscape of Scholarly Publishing. Research Communication Today, 9(1), 10-23.

Did you like this example?

268

11