What I Actually Want From AI in Peer Review

 It’s summer, and I finally have a bit more breathing room, some of which I use for peer review (the rest is for my granddaughter). The pressure never really lets up, though. I’ve written before about how AI is reshaping that pressure, on both the writing and reviewing sides of the equation.

Publishers are drawing a clear line. They’re asking reviewers not to run submitted manuscripts through AI tools, and they’re right to ask. A paper under review is unpublished work. Feeding it into a commercial AI system risks the author’s intellectual property in ways reviewers might not stop to consider.

I follow that line carefully. I use AI to help me write up my reviews, sharpen the language, organize my thoughts, and make the feedback easier for authors to act on. I never run a manuscript I’m reviewing through AI, though often I think it could help.

I know it could be helpful because with my own writing I do something different. When I finish a draft, I ask AI to act as a peer reviewer and pick apart my argument. Where is it weak? What can I cut when I’m over a word limit? AI has been remarkably good at both. It catches gaps I’ve stopped seeing because I’ve read my own draft too many times. I get back to work the same way I would after an actual peer review, provided I agree with the notes, which, much like human reviewers, I do not have to agree with.

I do this through institutional AI that doesn’t train on what I feed it. I’ll admit, honestly, that the training question isn’t the part that keeps me up at night, even if I did not have such access; the benefits definitely outweigh the risks. What strikes me instead is how good the feedback is and how much better the review process could be if this kind of access extended to reviewers too.

I’m not calling for AI-written reviews- in that case humans are not needed. I want to be clear that we (still?) need human reviewers. What I’d love to see is publishers giving reviewers access to no-training AI as a tool.

Peer Review made with ChatGPT 5.5

Picture a reviewer working through a manuscript that cites a document by name. Right now, chasing that down means opening another tab, another search, another interruption. AI access built into the review platform could pull it up on the spot. Or picture a paper built on a statistical test with a dozen parameters. A reviewer could ask what those parameters mean and whether they were applied correctly, instead of taking the authors’ word for it or quietly moving past a section they don’t fully trust but have no time to verify.

I recently reviewed a paper where the data looked suspect. It was disappointing, and in that case, the signs were obvious enough to catch without much effort. But it left me wondering how many more subtle cases slip through, the ones that look clean on the surface and only fall apart under closer inspection. That’s exactly the kind of pattern-checking AI does well, if reviewers had a legitimate, secure way to run it.

The bigger opportunity is replication. Right now, a reviewer can, in principle, rerun an analysis or test an alternative approach if the authors shared their data. In practice, very few do it, because it takes far more than any reviewer has the time and resources for. AI with real document and data access could shrink that gap. Reviewers could actually check the math (within approximation) instead of trusting that it holds. This may be critically important when people are using AI to generate research facsimiles, sometimes fabricating the data and the writing.

None of this changes what a peer reviewer is for. The job is still to weigh the argument, judge whether the evidence supports the claims, and tell the editor whether the work deserves to move forward. AI wouldn’t replace that judgment. It would just let reviewers exercise it with more precision, on more of the paper, in less time.

Faster reviews. Sharper reviews. Reviewers who can actually verify instead of just trusting. That seems like a use of AI worth building, and one publishers are well positioned to build safely, since they control the data and access.

What am I using AI for now as a Teacher Educator and Professional

 Since Generative AI came out, I have been using it extensively. As an exercise, I am logging all the direct Generative AI I use, knowing that there is much AI in the background of which I am less aware.

Generic letters: Looking at my log, I have used generative AI to create four official letters that required carefully worded messages that were sensitive yet firm. In each case, I used Chat GPT to create an initial wording, then edited the text to bring back my writing style and some of my personality when appropriate, and finally, I ran it through Grammarly to make sure that I had no embarrassing grammar and spelling errors. The use of generative AI for composing official letters creates great efficiencies for me and reduces the response times. Interestingly, one person asked me for a letter of support that they generated with the help of GenAI as well as a starting point.

In teaching: I have used ChatGPT to create a description of the social networks between students in a classroom for an activity on creating groups in an elementary classroom. Once again, I needed to refine the prompt a few times and finally edit the document, but the result was quite good, and I created an assignment that I will keep using in the future.

I tried to see what Gen AI would produce for an in-class presentation about reading instruction. The result was VERY generic, and I ended up discarding the suggested slides, retaining the I Dall-E to create unique artwork for the slides I designed for teaching writing. While Generative AI use was limited in creating content, I continue enjoying the use of the Designer feature in PowerPoint as a way to quickly spiff up my slide decks. Since we came back from Spring break, I created a set of questions for a welcome-back exercise that went very well.

Finally, I engaged my students in using GenAI to create groupings in their classroom (mock data) to see what the benefits and challenges are. The discussion that ensued included comments ranging from amazingly fast and accurate to a student questioning whether it is worth the time after a lot of editing.

Review of academic paper: Once I read the paper I was reviewing and had the main points that I wanted to stress to the authors so they could improve their research paper, I used Cen AI to expand and explain my bulleted points. The amount of editing this exercise created for me was a very limited return on investment, and I doubt I will use it in this way again.

Podcasting: I used GenAI to create episode summaries of the Not That Kind of Doctor podcast using the transcripts as the raw material. One episode summary was well done while ina. second GenAI completely missed the point. Both needed editing but were still a major time-saving application.

Across multiple uses, I usually prompt GenAI there times before I get everything that I want (or give up). More detailed prompts yield much more accurate results and less follow-up. Grammarly let me know that it made over 6000 suggested edits. Gen AI has changed how I work; it has made some things much easier and saves me time every day. However, I am still concerned with accuracy and specificity that can be achieved only through my deep seated professional knowledge.