Who Checks the Machine When We Forget How?

A few weeks ago I read a New York Times article on an MIT report which warns that AI is causing a “cognitive surrender” and how universities are being affected by it. A little over a week later, the same phrase came up in a question at OpenAI’s AGI Economy conference. Cognitive surrender – it sounds almost apocalyptic. But what, exactly, are we surrendering?

I wanted to understand what separates using AI to help us think from allowing it to do the thinking for us. And whether that distinction could change over time: could something we initially delegate by choice become something we struggle to do ourselves? These questions sent me down a rabbit hole, from studies of AI and learning, to automated cockpits and the ways literacy changed our brains.

Let’s start with cognitive surrender. Steven Shaw and Gideon Nave, in their paper “Thinking—Fast, Slow, and Artificial,” introduce this term. The authors define cognitive surrender as the tendency, both behavioral and motivational, to hand over judgment, effort, and accountability to an external system (here AI), especially when its output is presented fluently, confidently, or seamlessly. They do separate this from the concept of cognitive offloading, which is utilizing an external source, physical actions, tools or technology to reduce the mental effort required to complete a task. Like writing down grocery lists, or using a calculator or using GPS for navigation.

To understand this better we need to understand the dual-process theory popularized by Daniel Kahneman. The theory divides human thinking into two distinct systems; System 1 is fast, unconscious, and automatic – relying on intuition and instincts to make quick everyday choices. System 2 is slow, conscious, and effortful – stepping in to handle complex logic, mathematics, and critical analysis. Shaw and Nave propose a third, artificial system – the Tri-System Theory, where in System 3 the reasoning is performed outside by AI, which can supplement or replace some of what we would otherwise do ourselves.

Now let’s revisit cognitive surrender and offloading. While offloading, one strategically retains control over reasoning and the output, complementing System 2. And cognitive surrender is when one skips System 2 and entirely relies on System 3 (AI) for reasoning and judgement. This is a really interesting theory and I can see why it is worrisome. With AI, an answer can arrive with an explanation, a justification, even a response to our objections. We can either use that explanation to work through a problem or just adopt it as our own without understanding much of it. But how do we identify that surrender?

The authors test this idea by randomly assigning participants to answer reasoning questions either independently or with optional access to AI. In the AI condition, the advice is randomly set to provide either the correct answer or a convincing explanation of the question’s intuitive mistake. When participants consulted the AI, they followed its advice 92.7 percent of the time when it was correct and 79.8 percent when it was wrong. Without AI, participants answered 45.8 percent correctly. This rose to 71 percent when AI advice was correct and fell to 31.5 percent when it was wrong. Average confidence nevertheless rose from 65.3 to 77 percent with AI. The authors interpret this as cognitive surrender. However, I would take these results with a grain of salt. While it clearly shows that AI can influence answers and confidence, it does not establish whether someone accepted its answer without thinking, tried unsuccessfully to check it, or found it confirmed an intuition they already held.

Did they go with their intuition, ask AI to verify it, and feel reassured? Did they reason carefully but fail to identify the mistake? Did they decide that the model was more likely to be right than they were? Or did they simply copy the answer? Moreover, the incorrect advice reinforces the intuitive trap already contained in the question. Agreement could reflect confirmation of an existing mistake.

This does not make the experiment uninformative. Changing the advice changes responses. But I would read it as evidence about reliance and confidence, while keeping the stronger claim about surrender open. The distinction matters because delegating a task can be quite reasonable. I do not recalculate everything my calculator does. Whether delegation becomes a problem depends partly on what I still understand and what I need to be able to check.

Consider a simple example. I ask an LLM to calculate the average attendance rate and provide individual attendance records for two groups. One group has 9 people with an average attendance rate of 90 percent. Another has 14 people with an average of 50 percent. Everyone has the same number of opportunities to attend. The model writes Stata code to perform the calculation.

* Calculation 1: give each GROUP equal weight
preserve
collapse (mean) attendance_rate, by(group)
summarize attendance_rate     // mean of group rates:      70.00%
restore

* Calculation 2: give each PERSON equal weight
summarize attendance_rate     // mean of individual rates: 65.65%

Now here’s the tricky part. Take the mean of the two group averages and the answer is 70 percent. Take the mean across all 23 individuals and the answer is about 65.65 percent. Both calculations are legitimate. They answer different questions: what is the average group’s attendance rate, and what is the average person’s attendance rate? If I want the second answer and receive code producing the first, the code can run perfectly and still answer the wrong question.

To catch that, I need to understand what is being averaged and read enough of the Stata code to see whether it performs the intended calculation. With a large dataset, I cannot simply calculate the answer in my head. A plausible number tells me very little. If I want the average across individuals, I need to notice whether the code first reduces the data to group averages and then gives each group equal weight. I can look up a command I have forgotten, but I still need enough statistical and coding knowledge to know what to check.

This is where “just check the AI” becomes a more demanding instruction than it sounds. We begin by delegating work we understand well enough to check. But if that delegation repeatedly removes the practice through which we maintain our understanding, the ability to check may itself weaken. If we retain less of the relevant knowledge, we may check worse, and offloading can slowly slide into surrender. What begins as a choice to save effort may eventually become a dependence on the answer. The concern, then, extends beyond whether someone accepts an answer without thinking. It is whether repeated delegation makes them less able to evaluate the next one.

A panel study by Strömberg, Lei, and Wu follows a little over 26,000 students in China for 30 months and examines education outcomes by exploiting the staggered AI adoption among students. The authors report an uncomfortable combination: better homework performance alongside worse exam performance. The learning losses are concentrated among students whose unusually short homework completion times suggest they are handing much of the work over to AI. These effects build over approximately six months before stabilizing. The authors suggest that students become better at using AI to complete homework while gaps in their learning accumulate as more course material is covered this way.

That deserves attention. Homework is supposed to be part of the process through which students learn. If the submitted work improves while independent performance deteriorates relative to the comparison group, then the homework score may be telling us less about what the student has learned.

The design of the assistance also matters. In Hamsa Bastani and colleagues’ mathematics experiment, students using a general AI interface did better during assisted practice but scored 17 percent lower than the control group when the assistance was removed. But an AI tutor designed with learning safeguards largely avoided that penalty. In a different experiment, Kestin and colleagues found that a carefully designed AI tutor improved immediate learning outcomes compared with an active-learning physics class at Harvard. These are different experiments with different comparisons, but together they suggest that what the tool asks the student to do matters. Receiving a finished solution and working through a problem with assistance are different learning experiences.

That brings me back to an MIT study mentioned in the NYT article which started this rabbit hole. In “Your Brain on ChatGPT,” researchers assigned 54 participants to write essays using an LLM, a search engine, or neither. Alongside analysing the essays, they recorded brain activity using EEG and interviewed participants afterward. The LLM group showed weaker connectivity in the researchers’ EEG analysis and had more difficulty accurately quoting their own essays just minutes after writing them. They also reported less ownership of the text.

The recall result interests me because it brings us back to the distance between producing something and learning from producing it. A finished essay can contain an argument that its author has barely worked through. But it’s also important to note that the EEG results describe differences in activity during the task; they do not demonstrate structural deterioration of the brain. Only 18 participants returned for the final session switching between assisted and unassisted writing, so the longer-term implications remain open. Still, it is reasonable to ask what happens when this way of completing work becomes a daily habit.

There is an older argument that helps make sense of this concern. In her 1983 paper, “Ironies of Automation,” Lisanne Bainbridge described a problem with leaving humans to supervise automated systems. Automation takes over routine work, while the operator remains responsible for recognising failures and intervening when something goes wrong. Those interventions can be unusually difficult. Yet the work being automated may also have provided the practice through which the operator developed and maintained the knowledge needed to intervene.

This is the difficulty with the reassurance that we will simply become supervisors of AI. Checking an answer is itself skilled work. In the attendance example, I need enough understanding to notice that the code has changed the unit being averaged. If repeated delegation weakens that understanding, my next check becomes less reliable. I may then rely more heavily on the model, including asking it to check its own answer. Offloading can slowly slide into surrender, even when I began with a reasonable decision to save time. Following Bainbridge, the concern is that delegation can gradually undermine the expertise on which effective delegation depends.

Bainbridge also pointed to a problem across generations. Experienced operators could draw on skills acquired before automation; their successors could not be assumed to possess the same skills. With AI, we should therefore distinguish between an expert delegating work they already understand and a student delegating the exercises through which they would have developed that understanding. One risks losing practice. The other risks missing the opportunity to acquire the skill in the first place.

We can see why this matters in a small study of cockpit automation by Stephen Casner and colleagues. Sixteen experienced airline pilots flew a Boeing 747 simulator under varying levels of automation. Their basic aircraft-control and instrument-scanning skills were mostly intact. More substantial difficulties appeared in cognitive tasks such as tracking the aircraft’s position, deciding which navigational steps came next, and recognising instrument failures. Poorer performance on these tasks was associated with more task-unrelated thoughts during automated flight. The sample was small, and the experiment did not track the pilots’ skills from before their exposure to automation, so it cannot establish that automation caused the difficulties. But it makes the question of active supervision concrete: being present while a system operates is not enough to tell us whether the relevant skills are being exercised.

A more recent study of endoscopists in Poland raises a related concern. Across four centres, the researchers compared colonoscopies performed without AI in the three months before and after an AI detection system was introduced. The proportion in which at least one adenoma was detected fell from 28.4 to 22.4 percent, a decline of six percentage points. This was an observational before-and-after comparison, so it does not establish that the clinicians had lost their underlying expertise; changes in attention or behaviour could also matter. What makes it relevant here is that performance without assistance changed after the tool became part of ordinary practice.

These studies leave me with a question that goes beyond whether AI gives us correct answers. What happens to the skills we bring to the next task? Better performance while using a tool can coexist with weaker performance when it is absent. But there is another possibility: the work we stop doing could make room for a different kind of learning.

This was what drew me to literacy in the first place. In “Martin Luther Rewired Your Brain,” Joseph Henrich connects the spread of literacy to Protestantism that made reading a widespread expectation. Religious commitments, schooling and everyday practice helped turn a cultural invention into an acquired capacity. The interesting chain runs from changes in society to changes in what people repeatedly do, and from there to changes in cognition and the brain.

Dehaene and colleagues’ 2010 study gives this story a concrete neural basis. They compared brain responses in 63 adults who had learned to read in childhood, learned as adults, or remained nonliterate. Literacy was associated with stronger responses to writing in the left occipitotemporal cortex, an area involved in visual recognition. It also changed how visual and spoken language were processed. At the site increasingly responsive to writing, the researchers found a small competition with responses to faces. Many differences were present even among those who learned to read as adults. These were comparisons of functional responses, rather than a record of the same brains being transformed, but they show how a culturally acquired skill can be associated with a different organisation of existing systems.

How can the brain accommodate an invention as recent as writing? Dehaene and Cohen’s 2007 account of neuronal recycling proposes that cultural skills find a “neuronal niche”: existing circuits whose functions are close enough, and sufficiently adaptable, to support the new task. Reading recruits parts of a visual system that did not evolve specifically to recognise written words. The earlier organisation constrains what can be learned and is not simply erased. The brain adapts through the reuse of existing machinery.

So what happens when we start relying on AI more and more? Perhaps some of our work shifts toward evaluating reasoning produced elsewhere. We may spend less time recalling syntax and more time deciding whether a method answers the question. We may practise comparing an explanation with independent evidence, designing a test that could reveal an error, and judging when our own knowledge is insufficient to assess an answer. These are familiar human abilities, but working with a system that can produce convincing explanations at such speed could place new demands on them.

Repeatedly doing that work could develop a different profile of skills. Whether it also produces lasting changes in neural networks involved in reasoning and monitoring our own judgments is an empirical question. Literacy gives us a reason to investigate that possibility; it does not identify a cortical region that AI-use will recycle. For now, the more useful distinction is between forms of use that exercise judgment and forms that allow us to avoid it.

And this brings us back to Bainbridge. The ability to evaluate an outside answer still draws on knowledge of the subject. If I never learn why the two attendance averages differ, being told to “think critically” about the code will not get me very far. Developing skill in checking AI and maintaining the understanding that makes those checks possible have to go together. AI could help with that: I could attempt a solution first, ask it to challenge my assumptions, and test whether I can explain the result afterward. What matters is whether the interaction leaves me better equipped for the next problem.

That will depend partly on the environments in which we use these tools. A classroom that rewards only a completed assignment gives students different reasons to engage from one that also asks them to explain and apply what they have learned. A workplace can spend the time saved by AI on developing expertise, or simply demand more output. As with literacy, institutions help determine which activities become everyday habits.

I began with a term that sounded apocalyptic. I now find the quieter possibility more concerning: we could continue producing better work while becoming less able to judge it. Offloading can slowly slide into surrender if it erodes the knowledge that makes supervision meaningful. But that outcome depends on what replaces the work we delegate. The question is whether our use of AI gives us enough practice to keep learning, reasoning, and recognising when an answer has gone wrong.

I keep returning to the attendance example. The output can look perfectly reasonable. The code can run. Someone still needs to know which question it answered. How do we make sure people continue to learn enough to notice?