The person checking the AI still needs to learn the work

A new developer gets a piece of work they have never done before. They ask the AI for help. The code runs, the ticket closes and their manager sees a good result. Everyone has a reasonable basis for thinking the arrangement worked.

The developer may have learnt something along the way. They may also have learnt how to get a finished answer from the assistant. Those two forms of progress look very similar in a weekly report.

This becomes awkward when the organisation’s safety plan relies on the same person checking future AI output. A checker needs enough knowledge to recognise the answer that looks right and fails under one unusual condition. That knowledge usually comes from doing some of the work, getting it wrong and finding out why.

The issue reaches well beyond coding. A new benefits caseworker learns which missing detail changes a decision by handling cases and receiving feedback. A junior researcher learns to notice a weak source by following claims back through the evidence. When AI takes away much of that practice, today’s work may still be useful. The organisation also changes the route by which people become experienced.

A 2026 randomised study by Judy Hanwen Shen and Alex Tamkin gives us a small, unusually clear example. Fifty-two experienced Python programmers were asked to work with Trio, an asynchronous programming library they did not know. Half could use a GPT-4o coding assistant. Half had the task instructions and web search. Everyone then took the same test without AI.

The programmers with AI completed the practical exercise a little faster on average, although the difference was not statistically significant. Their later test scores were 17 per cent lower. This was a short preprint with 52 people, so it cannot tell us what happens to workplace skill over a year. It can show how a finished task and the knowledge needed to inspect that task can move separately.

The work was finished before the skill had formed

Participants had 35 minutes to complete two coding tasks. The assistant could see the current code and, when asked, provide a full correct solution. Afterwards, everyone took a 27-point test covering conceptual understanding, code reading and debugging. The researchers chose these areas because they matter when someone has to supervise AI-written code.

The AI group scored 4.15 points lower on average. The difference remained statistically significant after the researchers adjusted for performance on a warm-up task. An exploratory analysis found the largest gap in debugging. That part was not preregistered, so it gives us a useful lead for further study. It is not a settled finding.

Completion gives a slightly different picture. Every participant with AI finished both tasks. Four of the 26 people without it failed to finish the second task before time ran out. Average completion time did not differ reliably between the groups. AI access appears to have helped everyone get to an answer inside the session. The average speed gain remains uncertain, and the later understanding was lower.

An ordinary workplace rarely runs the second test. The manager sees working code, a completed claim or a draft sent to the client. They may only discover the missing knowledge when the AI is unavailable, or when it produces a confident answer with a fault buried in the middle.

Calling someone a reviewer does not give them the knowledge required for review. A person can approve work because it looks familiar, because the system has usually been reliable or because checking the reasoning would take longer than starting again. The approval still appears in the process log as human oversight.

The study measured learning immediately after one short episode. These participants already knew Python and were learning a new library. Most were freelance or professional developers recruited as paid crowdworkers, and the whole exercise lasted about an hour. Someone learning in their own job may place greater value on understanding the material. Time pressure can still pull them towards finishing and moving on. This experiment does not tell us which pressure wins over time.

People gave the same assistant very different jobs

The screen recordings show what happened between access and outcome. One AI recording was unavailable. In the rest, people used the assistant in strikingly different ways.

Four participants delegated almost all the coding. They were the fastest observed group, finishing in 19.5 minutes on average compared with 23 minutes in the condition without AI. Their later scores were low. Another four began with a few questions and moved towards full delegation. Four more repeatedly asked the assistant to debug or verify their code and also scored poorly.

Seven participants kept hold of more of the reasoning. They asked conceptual questions, wrote their own code and resolved their own errors. They were the second fastest observed group and scored well. A few people asked for code and then questioned how it worked. Others requested code and an explanation together.

These patterns are interesting and fragile. The researchers created the groups after watching the recordings. Each group contains only a handful of people, and participants chose how they used the AI. A deliberate learner may be more likely to ask conceptual questions and more likely to perform well afterwards. The recording cannot show that the question style caused the learning.

It does reveal something that the label ‘AI user’ hides. One person used the assistant as a teacher. Another used it as a coder. A third used it as a checker. They had access to the same tool and practised different parts of the work.

The assistant made full delegation easy. It gave finished code when asked and explanations when asked. The participant chose between those offers inside a timed task where getting to the end quickly had obvious value.

People with AI also met fewer errors. The median AI user encountered one during the exercise. The median participant without AI encountered three. The researchers suggest that finding and resolving those extra errors helped people learn. Error exposure was not randomised, so the study cannot prove that explanation. It remains plausible. Debugging is difficult to learn when someone else prevents most faults and fixes the one that slips through.

Workplace incentives can push people towards the full answer. A new starter measured on closed tickets and handling time has a reason to ask the AI to finish. Requesting an explanation takes longer. Trying the problem first takes longer again. The future value of learning belongs partly to the employer, while the immediate cost sits in the worker’s figures for this week.

Advice to ‘stay curious’ has to compete with those conditions. People usually learn the behaviour that the job makes workable.

The AI’s response changes what the person practises

A much larger school experiment tested a related design choice. In a peer-reviewed study published in PNAS, Hamsa Bastani and colleagues gave nearly 1,000 secondary-school pupils in Turkey one of two GPT-4 tutors during maths practice. A third group worked without an AI tutor.

The first tool behaved like an open chatbot. Pupils could ask for an answer and receive one. The second had access to teacher-written solutions and was instructed to guide the pupil with hints while withholding the complete solution.

Both AI groups did better during the supported practice. Grades rose by 48 per cent with the open chatbot and by 127 per cent with the guided tutor. The later exam took place without AI. Pupils who had used the open chatbot scored 17 per cent below the group that never had access. The guided tutor largely removed that penalty, although it did not produce a reliable advantage on the unaided exam.

The interaction records make the result easier to understand. Pupils often asked the open chatbot for the answer and copied it. The guided tutor kept more of the problem with the pupil. The researchers changed the behaviour available to the AI as well as the quality of its explanation. A prompt inside the tutor stopped the system handing over the whole solution.

School maths has clear answers, scheduled practice and a known exam. Professional work is messier. An employee has a result to deliver and may never be asked to prove what they can do without the tool. Even so, the experiment establishes something useful. Systems built on the same underlying model can create different learning behaviour through the way they respond.

An explanation added after a finished answer may help. The coding recordings give us a little evidence in that direction. Explanation text on its own offers no guarantee. It is easy to skim a fluent account and feel that it makes sense. The guided tutor required the pupil to keep acting on the problem. That preserved some of the practice through which the learning happened.

AI can also carry expertise into the work

Evidence from a live customer-support operation shows how AI can support learning. Erik Brynjolfsson, Danielle Li and Lindsey Raymond followed a staggered rollout of a generative AI assistant to 5,172 agents. Their peer-reviewed Quarterly Journal of Economics paper found a 15 per cent average increase in issues resolved per hour. Newer and lower-skilled workers gained most in speed and quality.

The assistant had been developed using examples of successful customer conversations. It suggested responses while the person remained responsible for the exchange. New workers with two months of experience and AI performed at least as well on several measures as untreated workers with more than six months in the job. Lower-skilled agents also began using language that resembled the communication of higher-skilled colleagues.

Software outages gave the researchers another view. When suggestions stopped appearing, workers retained some of their productivity gains compared with their own performance before AI. The effect was stronger among people who had received more suggestions and followed them more often. This supports the idea that some useful practice had transferred to the worker.

The study took place in one company and a fairly stable area of customer support. Parts of the learning evidence are suggestive and carry some uncertainty. The assistant also occupied a different position in the task. It offered examples inside a conversation that the worker continued to manage. The person read a suggestion, decided whether to use it and saw how the customer responded.

Highly skilled agents gained little, and some of their quality measures fell slightly even as they used more suggestions. This raises a different question about expertise. A system trained on strong past performance can help a novice catch up. The organisation still needs a way for experienced people to develop new practice and feed it back into the work. Repeating yesterday’s good answer will only take the service so far.

A clinical study gives us a reason to keep watching

A 2025 multicentre observational study of colonoscopy looked at what clinicians did when AI support was absent. Four centres in Poland had introduced AI to help detect polyps. Researchers compared 1,443 unaided colonoscopies carried out during the three months before introduction with unaided procedures during the three months after clinicians had begun using AI in other appointments.

The adenoma detection rate in procedures without AI fell from 28.4 per cent to 22.4 per cent. The six percentage-point difference was statistically significant. The authors described it as a possible change in endoscopist behaviour following repeated AI use.

This was a retrospective before-and-after study. Clinicians were not randomly assigned to periods of exposure, and plenty of other things could have changed over six months. Patient mix, scheduling and local practice may have contributed despite the researchers’ adjustments. The result is a signal, not proof that AI deskilled the clinicians.

It is still valuable because the researchers looked at performance when the tool was unavailable. Many evaluations finish while the AI is present. That misses the moment when a person has to take over, deal with an unusual case or work through an outage.

Decide which capability the person must keep

‘Checking the AI’ is too vague to design around. A software engineer may need to trace data flow and explain why a concurrency pattern becomes unsafe. A caseworker may need to spot missing evidence and know when a rule allows discretion. These are observable capabilities. They can be practised and tested.

The division of work can follow from that decision. AI may provide a full answer on established, low-consequence tasks. On work that develops a necessary skill, it could ask for a first attempt, give a bounded hint or explain an error while leaving the reasoning with the worker. A person dealing with an unusual case may need to record their own view before seeing what the AI suggests. Each choice needs local testing because the studies do not give us one design that suits every job.

Protected unaided work may sound old-fashioned once a tool is available all day. It can still tell an organisation whether its oversight plan is real. A sample of cases could be completed without AI and followed by feedback. Teams could run short outage exercises or give people examples that differ from the material the system usually handles. The assessment should resemble the capability the job needs. A factual quiz will not tell us much about diagnosis or judgement.

The time cost has to be counted honestly. Guided work can be slower at first. Senior feedback also takes time. If an organisation wants junior staff to become competent reviewers, it has to allow the behaviour through which that competence grows. A target built only around today’s throughput will reward the worker who hands over the most thinking.

Measures can follow the result beyond task completion. Record total time, accepted quality and later rework. Add a representative task without AI. Observe which suggestions people accept and when they ask the assistant to take over. Higher throughput alongside falling unaided performance gives the team a clear reason to revisit the arrangement.

Philip Cash, Ruth Schmidt and Weston Baxter make a related point in their 2026 review of behavioural design processes. They argue that teams should name intermediate outcomes such as knowledge and expertise, then decide what would make them continue, repeat or change a process. Skill is one of the things produced while the work gets done. It affects later performance even when today’s output has already been accepted.

A pilot could set a simple revision condition. If people complete more work while their unaided diagnosis falls below an agreed standard, the team changes the allocation, the AI’s response rules or the amount of protected practice. This does not require preserving every old skill forever. It requires being explicit about the capability the human part of the service still depends on.

Longer workplace studies are needed. We do not yet know how much unaided practice is enough, which skills recover quickly after a period of reliance or how the answer changes as models improve. AI assistance, human coaching and independent work also need credible comparison over time.

The current evidence supports testing retained capability wherever human oversight depends on it. A finished task can tell us that the work got done. It cannot tell us, on its own, who is becoming able to handle the difficult one next month.

Human-AI Performance

By Lauren A Kelly

© 2026 Alterkind Ltd. All rights reserved.
Human-AI Performance™ is a proprietary methodology developed by Lauren A Kelly using my Behaviour Thinking® framework. All content, tools, systems, and resources presented on this site are the exclusive intellectual property of Alterkind Ltd.

You’re welcome to use, share, and adapt these materials for personal learning and non-commercial team use.

For any commercial use, redistribution, or integration into client work, services, or paid products, please contact lauren@laurenakelly.com to discuss licensing terms.

Icons by Creative Mahira, The Noun Project.

Thanks to Nicholas Edell, Valentina Tan and multiple VPs implementing AI for your feedback during development.

LICENSE
Based on work by Lauren A Kelly.
For commercial licensing contact: lauren@laurenakelly.com