The person checking the AI still needs to learn the work

A common plan for AI at work gives the system the first attempt and gives a person the final check. The arrangement sounds cautious. It also assumes that the person already has the knowledge to spot a weak answer, and that they will keep that knowledge while the AI does more of the work.

That assumption becomes fragile when the person is still learning. A junior developer learns partly by writing code that fails, finding the cause and trying again. A new caseworker learns which details alter a decision by handling cases and receiving feedback. If AI removes much of that practice, the organisation may get a usable answer today. The number of people able to assess a difficult answer later may also change.

A 2026 randomised study by Judy Hanwen Shen and Alex Tamkin offers a compact version of this problem. Fifty-two experienced Python programmers were asked to use an unfamiliar asynchronous programming library called Trio. Half had access to a GPT-4o coding assistant. Half used the task instructions and web search without the assistant. Everyone then took the same test without AI.

The AI users finished the practical exercise a little faster on average, though the difference wasn't statistically significant. Their later test scores were 17 per cent lower. The finding comes from a small, short preprint, so it cannot tell us that AI causes long-term loss of skill at work. It does show why finished work and learned capability need separate measures.

The task was done before the skill had formed

Participants had 35 minutes to complete two coding tasks. The AI assistant could see their current code and could supply a complete, correct solution when asked. The later 27-point test covered conceptual understanding, code reading and debugging. These were the skills the researchers considered relevant to supervising AI-written code.

The mean test score in the AI group was 4.15 points lower than in the group without AI. The standardised difference was substantial, and the result remained statistically significant after the researchers adjusted for performance on a warm-up task. Their exploratory analysis found the largest gap in debugging questions. That part of the analysis was not preregistered and should be read as a lead for further study.

The completion result was mixed. Every participant with AI finished both practical tasks. Four of the 26 people without AI failed to finish the second one within the time limit. Average completion time didn't differ reliably between the groups. The study therefore supports a measured loss in immediate understanding and leaves the average speed effect unresolved.

Ordinary work can hide this distinction. A manager sees a closed ticket or running code and may have little reason to test what the worker could do without the system. The missing capability becomes visible when the AI is unavailable, or when its answer is plausible and wrong.

Calling the human a reviewer doesn't supply the knowledge required for review. A person can approve an answer because it looks familiar, because the system usually works or because finding the underlying error would take longer than doing the task again. None of those actions establishes that the person checked the part most likely to fail.

The coding study captured this problem soon after one learning episode. Its participants already knew Python. The test concerned knowledge of a new library, while basic programming was familiar. They were paid crowdworkers, mostly freelance or professional developers, and the whole exercise lasted around an hour. People learning a tool for their actual job may have stronger reasons to understand it. They may also face stronger pressure to finish quickly. The experiment doesn't tell us which force would dominate over several months.

People used the same assistant in different ways

The screen recordings provide the most useful behavioural detail. One recording from the AI group was unavailable. In the remaining recordings, people gave the assistant very different jobs.

Four participants delegated the coding almost completely. They were the fastest observed group, finishing in 19.5 minutes on average compared with 23 minutes for the condition without AI. Their later scores were low. Another four began with a few questions and moved towards full delegation. A further four repeatedly asked the assistant to debug or verify their work and also scored poorly.

Other participants kept more of the thinking. Seven asked only conceptual questions, wrote the code themselves and resolved their own errors. This group was the second fastest observed pattern and scored well. A few people asked for generated code and then questioned how it worked. Others requested code and an explanation together.

The researchers built these groups after watching the recordings. Each group is tiny, and participants chose how to use the AI. The patterns cannot show that one style of prompting caused the test result. A person who already takes a more deliberate approach may both ask better questions and learn more. The recordings still show something that an access-versus-no-access comparison hides: the same assistant supported different divisions of work.

The AI's behaviour made full delegation available. When asked, it supplied finished code. It could also explain a concept. The person decided which offer to take under a time instruction that rewarded fast completion.

The two groups encountered different numbers of errors. The median participant with AI encountered one during the exercise. The median participant without AI encountered three. The researchers suggest that finding and resolving those errors helped the second group learn. They didn't randomise error exposure, so the study has not established its causal role. It remains a plausible explanation. Debugging skill is hard to develop when someone else prevents most faults and repairs the ones that remain.

An organisation can easily reinforce the delegation pattern. If it measures closed tasks and short handling time, a new worker has a reason to request the finished answer. Asking for an explanation adds time now. The later benefit belongs partly to the employer and may never appear in the worker's performance figures. Training advice that tells people to stay curious has to compete with the way their work is assessed.

The AI's response rules alter the practice

A larger study in schools tested a related design choice. In a peer-reviewed field experiment published in PNAS, Hamsa Bastani and colleagues gave nearly 1,000 secondary-school pupils in Turkey one of two GPT-4 tutors during mathematics practice. A third group had no AI tutor.

The first tool resembled a standard chat interface. Pupils could ask for an answer and receive one. The second received teacher-written solutions and was instructed to guide the pupil with hints while withholding the full solution. During supported practice, both AI groups performed better than the group without AI. Grades rose by 48 per cent with the open chatbot and by 127 per cent with the guided tutor.

The result changed when the help disappeared. On a later exam without AI, pupils who had used the open chatbot scored 17 per cent below the group that never had access. The guided tutor largely removed that penalty, though it didn't create a reliable advantage on the unaided exam.

The interaction records help explain the difference. Pupils often asked the open chatbot for the answer and copied it. The guided tutor kept them working through the problem. The study therefore changed the behaviour available to the AI, not just the quality of its explanations. A prompt inside the tutor limited what the system would hand over.

School mathematics is not professional work. Pupils had scheduled practice and a clear later assessment. An employee usually has a live result to deliver, mixed task types and no protected test of independent capability. The study still establishes a useful design fact. Two systems built on the same underlying model produced different learning behaviour because they answered the person differently.

Requesting an explanation after receiving a finished answer may help. The small coding study suggests that it can. The evidence doesn't support treating explanation text as a complete solution. A fluent explanation can be skimmed, and the worker may never have to use the idea. The guided maths tutor changed the action required of the pupil. It preserved some of the work through which learning happened.

AI can also carry expert practice into the work

Evidence from a live workplace gives a different result. Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied a staggered rollout of a generative AI assistant to 5,172 customer-support agents. Their peer-reviewed Quarterly Journal of Economics paper found a 15 per cent average rise in issues resolved per hour. Newer and lower-skilled agents gained the most in speed and quality.

The assistant had been developed using examples of successful conversations. It suggested responses while the agent remained responsible for the customer exchange. New agents with two months of experience and AI performed at least as well on several measures as untreated agents with more than six months of experience. Lower-skilled agents also began using language that resembled the communication of higher-skilled colleagues.

The researchers examined periods when the software stopped supplying suggestions. Workers retained some productivity gains relative to their own pre-AI performance, especially when they had received more recommendations and followed them more often. This evidence supports an account of durable learning and makes complete reliance on live suggestions less likely as an explanation.

The study took place in one company, in a fairly stable area of customer support. It cannot settle how AI affects learning in other jobs. The system also behaved differently from an assistant that completes a whole task. It placed examples of effective practice inside a continuing customer conversation. The worker read a suggestion, decided whether to use it and saw how the customer responded.

The highest-skilled agents gained little. Some quality measures fell slightly for them, even as their use of suggestions grew. This finding raises another skill question. When the system learns from strong workers and those workers begin following it, the organisation needs a route for new expert practice to enter. Repeating yesterday's best answer can improve a novice's work while narrowing what experienced staff contribute to tomorrow's method.

Loss of unaided performance can appear in consequential work

A 2025 multicentre observational study of colonoscopy gives the concern a higher-stakes setting. Four Polish centres introduced AI support for detecting polyps. Researchers compared 1,443 colonoscopies performed without AI during the three months before introduction and the three months after clinicians had begun using it on other procedures.

The adenoma detection rate in unaided procedures fell from 28.4 per cent before exposure to 22.4 per cent afterwards. The absolute difference was six percentage points and was statistically significant. The authors described the result as a possible change in endoscopist behaviour after repeated use of AI.

This was a retrospective before-and-after comparison. The clinicians were not randomly assigned to months of exposure, and conditions could have changed over time. Patient mix, scheduling or other features of practice may have contributed despite the researchers' adjustments. The study identifies a signal; it doesn't prove that AI use deskilled the clinicians.

Its outcome deserves attention because it tested performance when the AI was absent. Many workplace evaluations stop while the system is present. That leaves an incomplete account for any job in which a person may need to take over, detect an unusual case or work safely during an outage.

Design for the capability the person must retain

A team first needs to decide which human capability remains necessary. ‘Checking’ is too broad. A software engineer may need to trace data flow and explain why a concurrency pattern is unsafe. A benefits caseworker may need to recognise missing evidence and know when a case requires discretion. Each capability calls for different practice and a different test.

The division of work can then reflect that decision. AI can give a full answer on established, low-consequence tasks. On work that develops a required skill, the system can ask for a first attempt, offer a bounded hint or explain an error without replacing the worker's reasoning. A person approaching a novel case may need an independent view before seeing the generated answer. These are design proposals. Teams should test them against task performance and learning in their own setting.

Protected unaided work also has a place. A sample of cases can be completed without AI, followed by feedback. Teams can run short outage exercises or give workers new examples that the training set didn't cover. The aim is to observe usable capability. A quiz may suit factual knowledge; a simulated case or code review may suit judgement and diagnosis.

Managers need to include the cost. Guided work may take longer at first. If an organisation wants junior staff to learn, it has to allow time for the behaviour that produces learning and avoid penalising it through narrow output targets. Senior review has a cost too. The comparison should include the work required to build the next group of competent reviewers.

Measures should follow the result beyond task completion. Record total time, accepted quality, later rework and performance on a representative task without AI. Observe which suggestions people accept and where they ask the system to take over. A rise in throughput alongside falling unaided performance gives the team a reason to change the arrangement.

Ruth Schmidt's recent work with Philip Cash and Weston Baxter helps place this choice. Their review of behavioural design processes argues that teams should make intermediate outcomes such as knowledge building and expertise development explicit. They also call for clearer criteria about when a process should continue, repeat or change.

Applied to AI-supported work, skill is one of the results produced during the process. It affects future performance even when today's output is already accepted. A pilot can therefore set a revision condition in advance: if workers complete more cases while their unaided diagnosis falls below an agreed standard, the team revises the allocation or the AI's response rules.

Longer field studies are still needed. Researchers need to follow workers as task difficulty changes and models improve. They also need credible comparisons between AI help, human coaching and independent practice. The current studies do not tell us how much unaided work is enough, or which skills can safely fade because the work no longer requires them.

Current evidence supports testing retained capability wherever human oversight depends on it. It doesn't yet show how much practice each job requires, or how that threshold changes as models become more capable.

Human-AI Performance

By Lauren A Kelly

© 2026 Alterkind Ltd. All rights reserved.
Human-AI Performance™ is a proprietary methodology developed by BehaviourStudio using our Behaviour Thinking® framework. All content, tools, systems, and resources presented on this site are the exclusive intellectual property of Alterkind Ltd.

You’re welcome to use, share, and adapt these materials for personal learning and non-commercial team use.

For any commercial use, redistribution, or integration into client work, services, or paid products, please contact lauren@laurenakelly.com to discuss licensing terms.

Icons by Creative Mahira, The Noun Project.

Thanks to Nicholas Edell, Valentina Tan and multiple VPs implementing AI for your feedback during development.

LICENSE
Based on work by Lauren A Kelly.
For commercial licensing contact: lauren@laurenakelly.com