What I Actually Want From AI in Peer Review

 It’s summer, and I finally have a bit more breathing room, some of which I use for peer review (the rest is for my granddaughter). The pressure never really lets up, though. I’ve written before about how AI is reshaping that pressure, on both the writing and reviewing sides of the equation.

Publishers are drawing a clear line. They’re asking reviewers not to run submitted manuscripts through AI tools, and they’re right to ask. A paper under review is unpublished work. Feeding it into a commercial AI system risks the author’s intellectual property in ways reviewers might not stop to consider.

I follow that line carefully. I use AI to help me write up my reviews, sharpen the language, organize my thoughts, and make the feedback easier for authors to act on. I never run a manuscript I’m reviewing through AI, though often I think it could help.

I know it could be helpful because with my own writing I do something different. When I finish a draft, I ask AI to act as a peer reviewer and pick apart my argument. Where is it weak? What can I cut when I’m over a word limit? AI has been remarkably good at both. It catches gaps I’ve stopped seeing because I’ve read my own draft too many times. I get back to work the same way I would after an actual peer review, provided I agree with the notes, which, much like human reviewers, I do not have to agree with.

I do this through institutional AI that doesn’t train on what I feed it. I’ll admit, honestly, that the training question isn’t the part that keeps me up at night, even if I did not have such access; the benefits definitely outweigh the risks. What strikes me instead is how good the feedback is and how much better the review process could be if this kind of access extended to reviewers too.

I’m not calling for AI-written reviews- in that case humans are not needed. I want to be clear that we (still?) need human reviewers. What I’d love to see is publishers giving reviewers access to no-training AI as a tool.

Peer Review made with ChatGPT 5.5

Picture a reviewer working through a manuscript that cites a document by name. Right now, chasing that down means opening another tab, another search, another interruption. AI access built into the review platform could pull it up on the spot. Or picture a paper built on a statistical test with a dozen parameters. A reviewer could ask what those parameters mean and whether they were applied correctly, instead of taking the authors’ word for it or quietly moving past a section they don’t fully trust but have no time to verify.

I recently reviewed a paper where the data looked suspect. It was disappointing, and in that case, the signs were obvious enough to catch without much effort. But it left me wondering how many more subtle cases slip through, the ones that look clean on the surface and only fall apart under closer inspection. That’s exactly the kind of pattern-checking AI does well, if reviewers had a legitimate, secure way to run it.

The bigger opportunity is replication. Right now, a reviewer can, in principle, rerun an analysis or test an alternative approach if the authors shared their data. In practice, very few do it, because it takes far more than any reviewer has the time and resources for. AI with real document and data access could shrink that gap. Reviewers could actually check the math (within approximation) instead of trusting that it holds. This may be critically important when people are using AI to generate research facsimiles, sometimes fabricating the data and the writing.

None of this changes what a peer reviewer is for. The job is still to weigh the argument, judge whether the evidence supports the claims, and tell the editor whether the work deserves to move forward. AI wouldn’t replace that judgment. It would just let reviewers exercise it with more precision, on more of the paper, in less time.

Faster reviews. Sharper reviews. Reviewers who can actually verify instead of just trusting. That seems like a use of AI worth building, and one publishers are well positioned to build safely, since they control the data and access.

The Tinkering Ways AI is Transforming Research

 I recently sat on a grant panel. As on many panels, I was asked not to discuss the details of the decisions, a request I always respect. The proposals spanned a wide variety of fields, so there was plenty to learn and chew on. What struck me most was the robust use of AI.

Well, you’d say, we live in the age of AI. Everybody’s got ChatGPT (or pick your favorite model), and people mumble something about how it can be used because generative AI is sexy and new. The cynic would point out that we’re saturated with AI, both as an economic and a cultural phenomenon, and that if you want funding, you have to at least nod toward some magical use of AI that will completely transform your field. The cynic would add that in most cases, the AI is described vaguely and hand-wavily. And they wouldn’t be entirely wrong.

But in many of these applications, I saw something else entirely. Most of them actually included well-thought-out machine learning, not generative AI. They used small models, built from scratch for a specific purpose, as opposed to large language models. In a way, these proposals were “unsexy”: they didn’t reach for the new models at all. They offered innovative solutions using the old ones.

Created with ChatGPT 5.5

The insight took a few days to land. AI was everywhere, but not in the cynical way I’d expected. Here’s how I read it: the cultural significance of generative AI, and the sheer accessibility of AI, has gotten researchers in every domain thinking, “Hey, maybe this AI thing can help me solve this really hard problem.” (Sorry, I can’t reveal much from the panel, so no details.) It’s fairly clear to me that generative AI has started a conversation inside research that will produce exciting new solutions to some of our most intractable problems.

The next challenge, oddly, will be having enough compute, coders, and AI experts to bring these projects to fruition. I saw this at one of our research gatherings earlier this year: everyone is looking for an AI research partner, while our AI scientists are flooded with collaboration requests. I think we could triple the number of people doing that work and still have demand to spare.

So where does that leave us? I believe research universities should invest in compute and AI know-how as a shared service that can support many research efforts at once. Some have already gone this direction. I hope more, including UNL, will follow.

Incoherent Thoughts from the Jagged Edge

[originally posted on my substack] 

I started this week sitting across from a long-time colleague, someone whose work I respect and whose judgment I trust. Let’s call him Eric. Eric is co-authoring something with a younger scholar, and he has been finding hallucinated references in the manuscript. He has, I think, tacitly accepted that AI was used in the writing. The references are the tell. He has not said so explicitly, but the evidence is sitting right there in the reference list.

What struck me about the conversation was not the hallucinations. Those are a real problem, and anyone using AI-assisted writing needs to know that. What struck me was that Eric is not using AI himself, which means his mental model of the technology is frozen at whatever he last heard about it. He does not know yet that frontier models (the current generation) are substantially less prone to fabricating citations than earlier versions. His experience of the technology is secondhand and dated, and that gap between perception and current reality is itself a kind of problem. The tool is a moving target, and the critique needs to move with it.

Later in the week, I had a completely different kind of conversation. Dave Fowler, a math education professor who retired a few years ago, shared with me the exchanges he has been having with ChatGPT about the nature of theories. Dave is curious, playful even about what this thing is and what it can do.

He shared some of the thinking he was doing with and about AI. We smirked together at the turns of phrase the model deployed in those conversations, including things like “You’re drawing a very precise and interesting analogy, Dave.” which my brain heard in the voice of HAL 9000. There is something both charming and slightly unnerving about that sentence. It flatters, it redirects, it sounds like a thoughtful interlocutor. Is it? Is it not? Dave was not sure, and neither was I, and that wondering felt like the right place to be.

I am still turning that conversation over in my head. What Dave is modeling is something I think we are not talking about enough in education: what it looks like to approach AI with curiosity rather than with a verdict already in hand. He retired, he has no institutional pressure either way, and for $20 a month he is just... exploring. There is something intellectually honest about that.

Uncertainty Saloon (Created with ChatGPT)

Meanwhile, the news cycle this week offered its own jagged edges.

A winner of the 2026 Commonwealth Short Story Prize has been accused of submitting an AI-generated story, and the situation quickly became a full-blown literary scandal. The accusations were amplified when Pangram Labs ran every Commonwealth Prize winner back to 2012 through their detection tool and found strong signals of AI-generated text in multiple entries, including the 2025 overall winner. The literary world, predictably, is having a moment.

And then Barnes & Noble CEO James Daunt appeared on the Today show and said he had no problem stocking AI-written books, as long as they were clearly labeled and not misrepresenting themselves. Cue the boycott calls. Cue the “all generative AI is ripping off someone else” counter-arguments on social media. Daunt has since clarified, repeatedly, that Barnes & Noble does not knowingly sell AI-generated books and takes active measures to exclude them. The clarification landed with about as much impact as you would expect, which is to say, not much. The outrage had already found its shape.

These two stories (the prize and the bookstore) are related. Both are really about thresholds. At what point does AI-assisted become AI-generated? Who gets to decide? What is the meaningful distinction between a human writer who uses AI as a tool and one who uses it as a ghostwriter? I do not have clean answers. What I am sure of is that any calls for no AI will just feed into shadow AI use.

On a more personal note: I have been noticing that when I write text on my own, without AI assistance, Grammarly flags it as AI-generated. (I still have a Grammarly subscription, though the controversy around the company has me reconsidering that). The flag itself made me stop and think. Am I absorbing AI patterns from all this use? Or is this a simpler and more uncomfortable explanation: that AI was trained on enormous quantities of mediocre writing (including mine), and is now reflecting the patterns of mediocre writing back at us, and Grammarly is simply recognizing them? I am a mediocre writer, so am I no better than AI?

I find this genuinely interesting and only slightly humbling.

Social media, that well-known venue for nuanced and measured discourse, has been full this week of a particular kind of certainty. The message, in various forms, is this: In five years you will look back and realize that AI in schools was a terrible mistake.

I do not know how to evaluate that claim, because I do not think anyone can know it. We are, as a society on the verge of something, but the outcomes are unclear. The people making this prediction with the most confidence tend to be the ones least encumbered by curiosity or evidence. As Ted Lasso misquoted “Be curious, not judgmental.” I think of Dave Fowler, sitting with his ChatGPT transcripts, genuinely wondering, and then I think of the social media certainty crowd, the UnDaves. The UnDave has an opinion. The UnDave does not let curiosity or facts complicate the opinion. The UnDave is very confident about what five years from now will look like.

I am not an UnDave. I hope I never become one. Here is where I am, for this week at least.

I love the exploration. I love what AI does for the scope and scale of what I can get done. And more than either of those things, I love that GenAI keeps opening doors to new questions, questions I would not have thought to ask, problems I would not have noticed, conversations like the one with Dave that I will be thinking about for weeks.

On schools specifically: I have said this before, and I will say it again. Caution is warranted. We do not have enough evidence yet about long-term impacts on learning, on writing development, and on productive struggle that builds capacity. Those concerns are legitimate and deserve to be taken seriously.

But on the other side of that, we must teach students about AI, what it is, how it works, and how it is changing the world they are growing up in. Not doing so is irresponsible. These students are going to live in a world shaped by this technology, whether we prepare them for it or not. Much like sex ed, if we don’t teach about it in schools, they will learn it elsewhere, with potential negative consequences.

I understand the desire to put the djinni back in the bottle. But we learned from the nuclear age that you cannot undo technologies. We must work as a society to reckon with the age of AI, and part of that is education.

Eric’s colleague used AI and did not know how to use it well. Dave used AI and understood enough to find the conversation genuinely interesting. The difference is not access. The difference is curiosity and intellectual engagement.

That is what education is (or at least should be).