It’s summer, and I finally have a bit more breathing room, some of which I use for peer review (the rest is for my granddaughter). The pressure never really lets up, though. I’ve written before about how AI is reshaping that pressure, on both the writing and reviewing sides of the equation.
Publishers are drawing a clear line. They’re asking reviewers not to run submitted manuscripts through AI tools, and they’re right to ask. A paper under review is unpublished work. Feeding it into a commercial AI system risks the author’s intellectual property in ways reviewers might not stop to consider.
I follow that line carefully. I use AI to help me write up my reviews, sharpen the language, organize my thoughts, and make the feedback easier for authors to act on. I never run a manuscript I’m reviewing through AI, though often I think it could help.
I know it could be helpful because with my own writing I do something different. When I finish a draft, I ask AI to act as a peer reviewer and pick apart my argument. Where is it weak? What can I cut when I’m over a word limit? AI has been remarkably good at both. It catches gaps I’ve stopped seeing because I’ve read my own draft too many times. I get back to work the same way I would after an actual peer review, provided I agree with the notes, which, much like human reviewers, I do not have to agree with.
I do this through institutional AI that doesn’t train on what I feed it. I’ll admit, honestly, that the training question isn’t the part that keeps me up at night, even if I did not have such access; the benefits definitely outweigh the risks. What strikes me instead is how good the feedback is and how much better the review process could be if this kind of access extended to reviewers too.
I’m not calling for AI-written reviews- in that case humans are not needed. I want to be clear that we (still?) need human reviewers. What I’d love to see is publishers giving reviewers access to no-training AI as a tool.
Picture a reviewer working through a manuscript that cites a document by name. Right now, chasing that down means opening another tab, another search, another interruption. AI access built into the review platform could pull it up on the spot. Or picture a paper built on a statistical test with a dozen parameters. A reviewer could ask what those parameters mean and whether they were applied correctly, instead of taking the authors’ word for it or quietly moving past a section they don’t fully trust but have no time to verify.
I recently reviewed a paper where the data looked suspect. It was disappointing, and in that case, the signs were obvious enough to catch without much effort. But it left me wondering how many more subtle cases slip through, the ones that look clean on the surface and only fall apart under closer inspection. That’s exactly the kind of pattern-checking AI does well, if reviewers had a legitimate, secure way to run it.
The bigger opportunity is replication. Right now, a reviewer can, in principle, rerun an analysis or test an alternative approach if the authors shared their data. In practice, very few do it, because it takes far more than any reviewer has the time and resources for. AI with real document and data access could shrink that gap. Reviewers could actually check the math (within approximation) instead of trusting that it holds. This may be critically important when people are using AI to generate research facsimiles, sometimes fabricating the data and the writing.
None of this changes what a peer reviewer is for. The job is still to weigh the argument, judge whether the evidence supports the claims, and tell the editor whether the work deserves to move forward. AI wouldn’t replace that judgment. It would just let reviewers exercise it with more precision, on more of the paper, in less time.
Faster reviews. Sharper reviews. Reviewers who can actually verify instead of just trusting. That seems like a use of AI worth building, and one publishers are well positioned to build safely, since they control the data and access.
No comments:
Post a Comment