Are AI Tutors Really Teaching Students - or Just Teaching the Test?

Why Better Test Scores Don’t Always Mean Better Learning.

PART 2 (of 2)

In the early 2000s, a concept called Brain Training began to take hold around the world. Companies such as Lumosity and BrainHQ argued that spending just a few minutes each day playing memory or logic games would make cognition generally stronger - that better game performance would translate into better performance at school, at work, and throughout everyday life.

By 2010, the brain training industry had exploded into a multi-billion-dollar market. Millions of people subscribed to brain training apps, schools and employers purchased licenses, and investors poured money into what many believed was the future of cognitive enhancement.

The twist is that today, the brain-training industry has largely faded from public consciousness. A handful of companies still exist, but the movement itself has all but disappeared.

Why?

TRANSFER: FROM NEAR TO FAR

In our last post, we explored the concept of transfer: the ability to carry learning from one context to another.

For simplicity, in that discussion I treated transfer as a binary process – you could either do it or you couldn’t. In reality, transfer is better understood as a spectrum: on one end lies near transfer, while at the other lies far transfer.

Near transfer occurs when learning is applied to a new situation that closely resembles the one in which it was acquired. The underlying structure, environment, and goals are the largely same - only minor details change. For example:

• Learning to drive in a Toyota Corolla and then moving those skills to a Honda Civic

• Learning Beethoven on a piano and then moving those skills to a keyboard

• Learning chess on a wooden board and then moving those to digital chess

Far transfer occurs when learning is successfully applied to a situation that differs substantially from the original learning context. While the underlying principles remain consistent, the structure, environment, and goals may all have changed. This requires the learner to recognize deeper patterns rather than superficial similarities. For example:

• Learning to drive a car and then adapting those skills to operate an agricultural tractor

• Learning to play the piano and then adapting those skills to play a two manual pipe organ

• Learning to play chess and then adapting those skills to play Shogi (Japanese chess)

How might this look in the classroom?

If a student learns Newton's laws and successfully solves twenty textbook problems involving moving objects, she has demonstrated near transfer. Weeks later, if she correctly explains to her parents why all the passengers lurched forward when their bus suddenly braked, she has demonstrated far transfer: applying familiar concepts in a novel, untrained context.

Between these two extremes lies a continuum of increasingly difficult transfer.

As any educator can attest, transfer - even relatively near transfer - is one of the greatest challenges in learning. It is remarkably common for students to perform well in one setting yet struggle to apply that same knowledge in another.

This is precisely why (as discussed in our previous post) effective curricula deliberately vary the way students encounter new ideas. Rather than practicing one type of problem repeatedly in one context, students explore the same underlying concept through different modes (writing, speaking, acting), different media (textbooks, whiteboards, flashcards), different environments (the classroom, the laboratory, field trips), and different instructional approaches (explicit instruction, hands-on demonstrations, group discussions).

The surface deliberately changes, but the underlying principle does not. The goal is to help students recognize larger conceptual patterns so they can carry and apply that knowledge beyond the context within which it was originally learned.

Which brings us back to Brain Training.

A BILLION-DOLLAR COLLAPSE

Despite it’s meteoric rise in the public sphere, many academic researchers were skeptical of brain training from the very beginning. The central claim (that repeatedly exercising memory would make cognition generally stronger, faster, and more resilient) ran counter to decades of research on learning transfer.

After years of individual studies, a 2016 meta-analysis pooled the results from 87 working-memory training studies to explore one question: how far does the learning actually travel?

Importantly, rather than treating transfer as a single outcome, the authors distinguished between increasingly distant forms of transfer:

• Near Transfer: Performance on working memory tasks that closely resembled those used during training.

• Intermediate Transfer: Performance on verbal and visuospatial working memory tasks that differed from the training exercises in form, but measured the same underlying cognitive construct.

• Far transfer: Performance on broader cognitive abilities that differed substantially from the trained tasks.

The pattern that emerged was clear:

The closer an assessment resembled the training exercises, the larger the apparent benefit. But the further it moved from the training environment, the smaller those benefits became until they effectively disappeared.

Several years later, the final nail in the coffin arrived. A second-order meta-analysis (a study that pools the findings of multiple meta-analyses to provide the broadest picture possible) reported the same pattern across different populations:

Again, brain training produced modest improvements on untrained tasks that closely resembled the training exercises, but those gains failed to generalize beyond closely related tasks.

The billion-dollar promise of brain training was never that people would become better at brain-training games; it was that they would become better thinkers all-around. Needless to say, that promise never materialized.

Which brings us back to digital tutors.

THIS TIME WILL BE DIFFERENT

The overall effect size of intelligent tutoring systems currently sits at +0.52 – comfortably above the threshold often used to indicate meaningful educational impact (around +0.40).

In our previous article, however, we asked how much of this benefit reflects improvements within the tutoring ecosystem versus beyond it. The evidence suggested that a non-trivial portion of +0.52 value likely comes from assessments that closely resemble the tutoring environment.

Here, we can be a bit more specific and ask : how much of this benefit reflects near transfer versus far transfer?

In 2012, 2015, and 2018, PISA (Programme for International Student Assessment) asked students how frequently they used digital technology at school for practice and drilling. Across all years and subjects, greater use of these digital tutoring activities was associated with lower performance, suggesting that skills acquired within the tutoring environment may not readily transfer to even moderately distant assessments.

In 2019 and 2023, TIMSS (Trends in International Mathematics and Science Study) similarly asked how frequently students use digital technology at school for taking digital quizzes and playing learning games - tasks similar to those found in many digital tutoring programs. Collapsed across both cycles and subjects, the overall pattern mirrors that of PISA: greater reliance on these activities was associated with lower performance on assessments removed from the tutoring environment.

Perhaps most tellingly, in 2017, 2019, 2022, and 2024, the NAEP (National Assessment of Educational Progress) asked both 4^th- and 8^th-grade students taking the reading exam how frequently they used digital technology to play reading-related tutoring games. Students who reported using these games as their primary mode of reading instruction achieved the lowest scores. Those who used them only to supplement traditional instruction performed somewhat better. Those who reported never using them at all achieved the highest scores.

Importantly, this pattern was replicated across all four testing cycles (though the figure below collapses the results across years for ease of presentation).

Taken together, these findings are consistent with the transfer pattern discussed above: while performance may be impressive on assessments closely aligned with the digital tutoring format, those advantages are far less evident on broader standardized measures removed from the tutoring environment.

While this may seem like an obscure methodological detail, it changes how we should interpret virtually every claim about digital tutoring.

WHY DOES THIS MATTER?

By-and-large, the purpose of assessment is not to determine whether students can perform tasks that closely resemble those on which they were trained: it is to determine whether they can carry what they have learned into new situations. Near-transfer assessments therefore tend to provide the most optimistic estimate of learning, while increasingly distant assessments offer a more demanding (and often more informative) measure of whether knowledge has become durable, flexible, and portable.

That may be the single most important point in this series, so it bears repeating:

The closer an assessment resembles the learning environment, the more likely it is to overestimate how well learning will generalize beyond that environment.

THE NEAR TRANSFER ILLUSION

Rather than treating the transfer literature as a warning, some developers have used it as a blueprint for maximizing the apparent effectiveness of digital tutors. Specifically, there are two ways near transfer can be exploited to make the gains from digital tutors appear larger than they truly are.

1. Use assessments that closely mirror the structure and cognitive demands of the tutor.

Consider a typical adaptive digital tutor. Using a computer, students work through a continuous stream of questions. Each response is evaluated immediately, the system estimates the student’s current level of mastery, and the next question is adjusted to match.

Now consider an assessment such as MAP Growth. Using the same computer they trained on, students again work through a continuous stream of questions. Each response is evaluated immediately, the system estimates the student’s current level of achievement, and the next question is adjusted to match.

Although one is designed to teach and the other to assess, both rely on the same adaptive cycle delivered within essentially the same digital interface.

If students spend hundreds of hours learning through an adaptive question-by-question tutor, then an adaptive question-by-question assessment is about as close to the learning environment as a standardized test can get.

2. Change the assessment to more closely mirror the tutoring environment.

Through 2023, The SAT – America’s primary college entrance exam – had looked essentially the same for decades. Every student received the same paper test booklet containing the same questions. Using a pencil and a bubble sheet (and, for many years, a handwritten essay), students completed the exam under strict time limit.

Structurally and cognitively, this experience differed substantially from adaptive digital tutors. This means that, although the SAT has never surveyed students about their use of digital tutors, we would expect it to exhibit the same transfer limitations observed on assessments such as PISA, TIMSS, and NAEP.

In 2024, however, the SAT underwent a fundamental redesign. The assessment moved online and adopted a multistage adaptive format: students complete an initial module, after which the difficulty of the second module is adjusted according to their performance. Furthermore, many students now complete the exam on the same school-issued device they use throughout the year for digital learning. In other words, the redesign made the SAT substantially more similar to modern adaptive tutoring systems.

As the SAT becomes more similar to the tutoring environment, the transfer distance between the two shrinks - and if near transfer is driving much of the reported success of digital tutors, we would expect students trained on those systems to perform relatively better on the digital SAT than they would have on its paper predecessor. Importantly, such improvements need not reflect deeper or more durable learning: they may simply reflect the fact that the assessment itself has become more similar to the environment in which students practiced.

Think about that for a moment:

If students who use digital tutors begin achieving higher scores on future standardized tests, those gains may not reflect deeper learning or more effective tutoring. They may simply reflect the fact that the assessments themselves have become increasingly similar to the tutoring environment.

Sometimes the easiest way to improve measured learning isn’t to change the teaching - it’s to change the test.

SO NOW THEN…

Over the course of these two articles, we’ve traced the same pattern across more than a century of educational technology. From Pressey’s teaching machine to brain training to AI tutors, the promise has remained the same: master the program, and that mastery will generalize to the wider world.

But the evidence tells a different story. Students often become highly proficient within the training environment, yet struggle to carry that knowledge into novel situations.

The major point here isn’t that students can’t learn from digital tutors – they clearly can. It’s that the learning they produce is often more contextualized, less durable, and less transferable than knowledge developed through more traditional pedagogies. Equally important, where and how we measure learning matters enormously: the closer an assessment resembles the learning environment, the more likely it is to overestimate what students have actually learned.

Put simply, the effectiveness of any digital tutor cannot be separated from the transfer demands of the assessment used to evaluate it.

Ultimately, the true test of learning is not what students can do with the tutor - but what they can still do without it.

Note: For simplicity, this article treats transfer as a continuum with two anchor points: near and far. Some educational researchers distinguish an intermediate form of transfer between these extremes. Several examples labeled “far transfer” here would more precisely fall into that intermediate category.