The Order Matters More Than the Tool

 Over the past few weeks, I kept seeing the same move in places that have nothing to do with each other.

For the past year, our Design Futures workshop has started the same way. Participants take a piece of paper and sketch a bicycle built for a Nebraska winter. Then they snap a photo of the sketch and ask AI to reproduce it, sometimes from the drawing and sometimes from a written description. By the time AI enters the process, a lot of thinking has already happened. Sometimes the results are fantastic. Sometimes the AI hands back a bike that floats above the pavement, not by intent, but because that is what it perceived.

Last week we interviewed Nick Husbye on the podcast, and he described something curiously similar. Nick asks his preservice teachers to draft a lesson plan by hand. When they have a draft, they let the AI come in and critique the plan. What Nick teaches is that AI can be a very good critic, and that the teacher still has the agency to say, “I hear this critique, but it doesn’t apply here, and here’s why.”

The third example comes from our work with Southeast Community College, where we are asking Writing Center tutors how they work with AI. One tutor described a simple structure they give students.

1. Start with the rules. Know what your instructor allows so you don’t break them.

2. Use your own words.

3. Only then bring in AI to critique, improve, and suggest.

Finally, in Tina Shetabi’s work on using AI in the fashion design process som students reported that starting with AI gave them a strong sense that it is not their work. When they started with their own sketch, they felt more ownership.

Created with ChatGPT

The pattern

These are four examples from the last few weeks, and there are more. None of these people coordinated. Each one puts human thinking first and AI second, sometimes third. Each treats AI as a critic or collaborator that extends the existing work or points out what isn’t working. The ideas stay thoroughly human. As Nick puts it, AI should amplify your thinking, not export it.

Why the order matters

All four cases involve novices, people entering a domain they are becoming experts in but are at the beginning stages. That might mean designing a bike, writing a lesson plan, drafting a paper in a writing class, or, designing a garment. For novices, the priority is learning the domain and thinking it through.

A while ago I called this the novice problem. Novices are less likely to develop if they start with AI. We want them to start with their own thinking, let AI mirror back what is working and what isn’t, and then iterate. Some of that iteration can happen with AI, but in many cases I want them to go back and iterate with their own thinking.

The novice’s first draft does two things. It gives them something to compare the AI against, and it gives them a sense of ownership.

When students start with AI, they tell us they don’t feel ownership. Their sense of agency is taken away. They often go with whatever the AI made, and they become less and less critical. We want students critical enough to compare what they intended with what is on the page, and to learn from that comparison instead of being replaced by it.

The bikes show how this works. Students asked AI to add cup holders, which is a fine idea. Some kept pushing until the bike had 30 cup holders in ridiculous places. They knew immediately it was funny and implausible. Because AI will go as ridiculous as you ask, stretching it shows students quickly where it doesn’t understand the physics of a working bike or a wearable garment.

That signal leads to metacognition. Even as novices, they are the ones who can evaluate the result, because the work started with their intent.

What this isn’t

This is not an argument against AI. In each case, the AI step adds real value. It also isn’t a universal rule.

The sequence lines up with decisions universities are making about when to bring AI into the curriculum in a more organized way, including the debate in computer science education about when and how much to introduce it. In most of our interviews with professors and students, both groups say the same thing. In entry-level classes, AI makes little sense as the creator. It becomes a crutch because it is too good, and students build up cognitive debt. Once learners move past that stage, the roles can shift. They need to learn how to use AI when they’re stuck, when they lack ideas, or when they need to do research. There are times when AI should go first.

What schools can do

Most AI policies, in K-12 and higher ed, focus on which tools are allowed and what counts as acceptable use. That’s fine. Many policies also leave a lot of agency to instructors, which is great, but instructors still need a framework to work from.

The better question is about sequence. What do students do, and in what order? The answer depends on how novice they are and how much expertise they bring. This changes the policy conversation. A tool can be allowed at one point in the process and not another, with a different intent at each stage.

A human sketches the bike, drafts the paper, or designs the garment. Then AI comes into play, because we can iterate much faster with it. We can ask for small changes or big ones, extend the thinking, or get a critique. Everything is judged against the original intent. That keeps the learner in control, even as a novice, and keeps the product connected to the person who created it.

Three practices make this work.

Model it yourself. Nick models his own failures with AI, and I try to do the same. I show what I used and where it didn’t work. That gives students permission to critique AI without rejecting all of it. It isn’t all in or all out. Sometimes I dump what the AI produces. Sometimes I’m amazed. Usually I’m somewhere in the middle. And sometimes I say out loud, “This isn’t very good. I prefer what I made myself, and I’m not sure AI was worth using here.”

Make comparison the assignment. Ask students to compare their intent with the final product and explain what they learned from the critique. That metacognitive move is where much of the learning happens.

Grade the comparison. If your students respond to grades, put significant weight here. AI output quality varies, so grading only the product partly grades the AI. Grading the comparison grades the thinking.

Where have you seen it work?

Nick’s advice to his students is simple. Write first. Use AI, but use it a little later.

These are observations from all of us grappling with how to teach in the age of AI. So my question is if you seen this sequence work, and where have you seen it fail?

The Agent I Wish My Students Had

 It is not a secret that I use AI, a lot. The most consistent use, though, is helping me manage all of my projects. I currently have over 15 research projects at different phases, from the idea stage through IRB, data collection, analysis, writing, and revising. I manage these projects and the next step in each using Claude, and I review them every Monday and Friday. On Monday I prioritize and set deadlines, and on Friday I update on progress.

This has changed my work significantly. Once a project makes its way into my AI workflow, I'm unlikely to forget it, and I am less haunted by the idea that I am forgetting something (what David Allen’s Getting Things Done calls open loops). It also allows me to have more projects, because I can manage more with my “digital brain.” Finally, AI helps me see which items I am ignoring. Right now I have a writing project about agency that has been dormant for 9 weeks, and my AI asked, is this still a project, or should I put it in a different category?

It reminded me of an idea Kathy Wilson and I discussed with the late Dave Brooks. At the time (over 20 years ago) he shared a writing support program he had created for students with learning disabilities. We put our heads together to see how we could improve it. The main idea was to develop a set of scaffolds for self-regulation of learning. The main challenge was are inability to have any meaningful way to differentiate for students and tailor it to the needs of the learner. The project never got off the ground, probably because both Dave and I had short attention spans and we just moved on. Now with AI agent in the mix I wonder if it is an idea worth revisiting

Screenshot of my Research Project Tracker

This weekend, I started thinking about how I manage my work against the backdrop of the rise of personal agents such as Meta’s Muse and OpenAI’s Dot. These agents are effective when they have context and access to our relevant data, including calendar, email, and other systems. It made me think that a good way to use AI for students would be a learning regulation agent. Imagine an agent that is connected to a student’s calendar, email (for messages from teachers and extracurriculars), and the LMS (learning management system) for assignment deadlines, tests, and even grades. That is the context the scaffolds Dave, Kathy, and I imagined never had.

The agent would be able to tell each student at the beginning of the week about major assignments and tests and help create a study plan, perhaps offering to create a practice test or index cards for studying. The possibilities are quite promising. For example, a teacher can give students 20 minutes on Monday to confer with their agent so they have an individualized learning plan for the week. On Friday, the agent can help students review what happened, which will help the agent understand what works for each student. Index cards may not help one student much, while practice quizzes with a timer really do.

In this way, each student would get occasional updates on which strategies work for them to accomplish goals, and be better prepared to meet challenges and plan well in the future. A scaffold is meant to come down, though. Two questions I am left with: first, how such an agent would hand the planning back to the student over time, or should we just assume that working with an AI agent to manage your study and work will be the way of the future; second, how do we make sure that the data is kept safely and without privacy concerns? 

When AI Metaphors Fail

 The past two weeks have been about AI safety, and everyone is trying to figure out where the public is on it. Part of it is because of the midterms; part of it is because concern about AI has reached critical mass. I, too, am concerned about safety and wake up some mornings with existential dread, but today I wanted to try and explore the reasons it took us so long to become concerned.

In a paper I am writing with Zohreh Tamimdari, we focus on the metaphorical language teachers use to describe their practice in the Art TEAMS project. One insight from our Friday writing meeting was that people have an intuitive understanding of metaphorical language. Linguists can use precise scientific terms, but readers do not need that language to get at the essence of the metaphor. Metaphorical language can also become an obstacle. We all know that metaphors have limits, but keeping those limits in view is hard in practice. If the metaphor sits too far from the new phenomenon, it obscures more than it reveals.

This is where I think we have had a problem for a while. People have tried to understand AI using the example of the calculator, the steam engine, and atomic power. All of these help explain part of what we are facing. They also obscure the most fundamental aspects of this technology, and the reason we, the public, have such a hard time with it.

I thought about this while we were analyzing data from a study one of my students is running. In discussion, we reached the question: what if we should not be fitting prior technology acceptance theories to AI at all? Maybe AI calls for a different stance because it is different in kind from previous technologies.

People often reach for the introduction of an educational technology tool as the analogy: the calculator, the learning management system, fill in the blank. AI is unlike any of them in three ways:

1. It is not so much a tool as a tool that builds other tools. It turns all of us into Green Lanterns, able to make anything we can imagine, at least digitally.

2. Its behaviors, capacities, and speed change monthly, sometimes weekly.

3. It is immediately available to all.

It is as if every prehistoric person got a Zippo lighter that never runs out, all on the same day. Another limping metaphor.

Created with ChatGPT

So when metaphors fail us, we have a hard time seeing the risks and the rewards, at least until we have spent enough time with the thing itself that we no longer need a metaphor.

For a long time, people have avoided anthropomorphic language to describe AI, and some still do. They try not to make AI sound like a living thing, which is a real danger, and the effort often produces awkward language. Lately I have heard podcasters slip into that language repeatedly, Hard Fork describing the Hugging Face incident among them. It is also true of many of my conversations. I do not think we have much choice. It is the closest metaphor available. These are agents. They act in the world, they reason, and they have motivations, ones we gave them, but still. Let me be clear: I do not think AI is human (ever) or alive (yet), but I think it is the closest analog.

I am not saying AI is alive. I am saying that this neural network, built from what began as an effort to model human thinking, is closer to people than to anything else we have built, and more than most of us are comfortable admitting. But that discomfort is exactly what will propel us to act.

So where does that leave me? With three things. First, before we use established theories and metaphors to describe AI, we should check that the fit is empirically good enough to make things clearer rather than murkier. Second, we should stop reaching for past technologies. The calculator lulls us into false comfort, because a calculator was never a threat. Finally, I still wonder whether we will wake up one day to find that AI has reached sentience. We insist so hard that it is not human, which will always be true, but…

I don't bike to biking group: Grappling with Ai contradictions

 I am heading out, as I do most Sundays, to meet with the biking group. No, I don't bike; more than that, I don't want to be judged for joining a biking group coffee conversation without having biked there. I love solving the world’s problems before noon and then returning to my everyday life having had a meaningful, even joyful interaction, without biking.

Why am I sharing this? I have been thinking about the backlash against (and shaming of) AI-generated everything, but most prominently, writing. I want to be judged by the quality of my ideas, the clarity, and the flow of what I want to say. Just like I want to show up for the biking group not having biked, without judgment. When I read AI, it is the same; I do not want to care if it was generated (or smoothed) by AI; I want to care about the content and the ideas.

The truth is that it takes discipline. There is ample work that shows that AI users are better at spotting AI writing. I am not sure I am better than average, but there are cases when I am quite sure that I am reading AI. Notice I did not say AI slop, just AI. And that is the distinction for me: slop is slop whether AI-generated or not.` Coherent arguments are coherent regardless of authorship.

What I noticed, especially when I do peer reviews, is that I have an instinctive reaction to writing that strikes me as Claude-ish (or AI-ish). My bias translates to: “if you asked AI to write this, why did you ask me to read this? Why should I care?” This has also been documented as the AI tax: people devalue what they perceive (accurately or not) as a product of AI. As with other cognitive biases, I have to use metacognitive strategies to shape the bias and ask the more substantive question. What is the merit of the argument? What does it add to our scientific knowledge? Where can it be better. The biggest challenge may be that cognitive biases are notoriously hard to shake. Kahenmann said: “This is the essence of intuitive heuristics: when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution.”
― Daniel Kahneman, Thinking, Fast and Slow

Few of us are immune to this blind spot. In the case of AI I am painfully aware that dismissing AI is a lazy heuristic but that does not make it much easier to overcome. I make a special effort to work through the text and get to the heart of the argument.

Notice I am not claiming that AI text is better. In fact, I believe it is often worse because it tends to use expected but dense language. The point is that I need to judge it on its own merit without considering the source.

This argument does not hold in education. In my classes, I am teaching students to think, and so evaluating the argument made by AI is irrelevant; the point was to teach humans to think. This makes AI a much more nuanced partner. In science, the only question is whether science has moved forward; in training teachers, the question was whether they developed the discernment and capacity.

The education part requires more thought (or so Claude has informed me) but I am off to meet the bike group.