At the turn of the century, sprinting became an increasingly popular sport.

Concerned that his athletes might be unprepared for competition, one progressive coach made an unusual decision: instead of focusing solely on running, he devoted a significant portion of training to teaching his runners how to use a stopwatch. They practiced starting it cleanly, stopping it precisely, and reading split times with confidence.

His reasoning: races are timed, so athletes should be fluent with the timer.

By the end of the season, his runners were excellent timekeepers – but none qualified for competition.

The Point:

Some tools exist solely to measure performance. They do not create it.

To be sure, athletes must be familiar enough with a stopwatch that it doesn’t confuse or distract them. But beyond that basic threshold, time spent training the tool is time not spent training the skill.

State academic tests – whether digital or paper-based – are no different. These are instruments of measurement: they capture performance, but they do not create it.

Of course, there is a difference between tools that are part of a skill and tools that merely display it. A pianist must train on a piano, because the instrument is the skill. But a stopwatch is not running – it merely records it.

Digital tests fall into the second category. When assessing reading comprehension, writing, or scientific reasoning, the screen is not the skill – it is simply the interface through which the skill is expressed.

THE MODE EFFECT

In a recent study, two equivalent groups of fifth-grade students were taught the same material over the same period. The only difference was how their learning was assessed: one group took paper-based tests, the other digital exams.

By the end of the semester, students tested on paper improved by roughly 14%, while students tested on computers declined approximately 5%.

This is a textbook example of the Mode Effect: the well-documented tendency for students to perform worse on digital assessments than on equivalent paper-based tests. As the authors noted:

• “…some students who demonstrated strong understanding during lessons and performed well on the pre-test experienced sharp score declines in subsequent tablet-based assessments.”

The students didn’t fail to learn - the testing medium interfered with how they could demonstrate their learning.

Faced with this problem, testing systems have tried to adapt. Some invest heavily in refining digital questions to eliminate formatting effects. Others change the construct itself: reducing long-form reading to skimming, or replacing writing with editing tasks.

The irony here is difficult to ignore: enormous time, effort, and expense are being devoted to mitigating a problem introduced by the testing medium itself – a problem that would be immediately resolved by returning to paper.

WHAT SHOULD WE DO?

Some tests are not designed to measure individual learning. The NAEP, for example, is used to track broad trends across schools and states. In these cases, as long as the mode effect is consistent, the testing medium is less important.

But if the goal is to understand what students actually know and can do, then introducing a medium that systematically distorts performance is, at best, counterproductive.

In my opinion, any test that directly impacts a child’s future options (e.g. the SATs, MAP, ACTs) should return to analog formatting - this ensures the most accurate measurement of a child’s development. However, because digital testing offers administrative efficiency, scalability, and rapid analysis, the return to analog formats may not happen immediately.

So the question becomes: if digital testing is required (at least in the short term), how much digital training do students need?

MORE TECH ≠ BETTER DIGITAL TESTING

There is no question that students need basic familiarity with the testing environment. A student who cannot type, navigate a screen, or select responses will be at a disadvantage.

But beyond basic familiarity, the evidence is remarkably consistent:

• A 2018 review found that student access to and familiarity with digital tools did not improve digital test performance. In some cases, the relationship was actually negative.

• A large-scale study (>25,000 participants) found that taking one digital practice test improved performance – likely by reducing initial friction. But additional practice tests provided no further benefits, and excessive exposure was associated with declining scores.

• A 2016 study found that computer-based training improved performance on digital writing tasks (where typing is directly relevant), but harmed performance in math (where typing is largely irrelevant).

• Studies of digital language learning consistently find that computer familiarity explains little-to-none of the variance in performance.

• A 2025 study showed that increased digital familiarity improves student comfort and preference for online testing. However, in controlled comparisons, students do not perform better in their preferred testing medium. In other words, preference is uncorrelated with performance, and mode effects persist.

Taken together, these findings suggest digital testing follows a threshold model.

Students need enough familiarity with the tool to avoid interference. But beyond that point, additional exposure provides no benefit – and may even be harmful.

SO NOW THEN…

Just because digital testing is the new norm, it does not follow that schooling needs to be redesigned around this format.

We should teach each subject and skill in whatever way best supports learning – whether that’s discussions, worksheets, problem-solving, or direct instruction. Then, shortly before the digital exam, we provide brief, targeted exposure to the testing platform.

That’s it.

If I knew my students had a digital science exam on December 30^th, I wouldn’t put them on screen all year. I would teach the material using the most effective and proven pedagogies. Then, a few days before the test, I would run short (10-20min) sessions to familiarize them with the digital format. One to three exposures is enough.

When a measurement tool changes, it’s tempting to believe everything else must change with it. It doesn’t. We should continue to teach in the ways that best support learning – and adapt only briefly to the final tool.

The screen is not the skill.

This marked the beginning of the MOOC: Massive Open Online Courses. Since then, more than 35 platforms have launched, offering over 20,000 courses to more than 220 million learners worldwide.

For over a decade, this explosion of digital learning has been framed as a breakthrough for equity. The logic is obvious: if education can be delivered cheaply, globally, and at scale, then students anywhere should have access to the same opportunities as those in the most privileged systems.

It’s a compelling idea.

It’s also one of the most persistent (and damaging) misconceptions in modern education.

THE MOOC PROBLEM

Enrollment numbers are impressive, but on their own they tell us very little about learning. What matters is not how man y students sign up, but how many succeed.

Of the 160,000 students who enrolled in Stanford’s original MOOC, only 23,000 completed it; a graduation rate of about 14%. Today, most MOOCs see completion rates below 10%.

More revealing, however, is who actually finishes. Students who complete online learning programs tend to share three characteristics:

1. They overwhelmingly come from affluent regions (roughly 80%)

2. They’re already highly educated (around 80% hold a bachelor’s degree, and nearly 50% hold a master’s degree).

3. They possess strong self-regulated learning skills.

Importantly, self-regulated learning skills - the ability to set meaningful goals, manage time, monitor progress, and sustain effort - are not innate. They are developed over time through structured practice.

And where does that practice typically occur?

School.

These skills are built in environments with clear expectations, guided instruction, and sustained human feedback. In other words, the very systems digital learning is often meant to replace.

This creates a fundamental problem for the equity argument: if digital learning requires strong self-regulation, and if self-regulation depends on prior educational experience, then digital learning will tend to primarily benefit those who are already well educated.

Rather than closing learning gaps, digital platforms are far more likely to widen them.

THE COVID TEST

The pandemic provided a global test of this idea.

When schools shut down, instruction moved online almost overnight. If digital learning truly equalized opportunity, this should have been EdTech’s moment to shine. Instead, its limitations became impossible to ignore.

UNESCO estimates that roughly half the world’s students were unable to access remote education at all. Even in the United States - one of the most digitally connected countries in the world - an estimated 16 million students lacked reliable access to devices or internet.

The result was greater inequality.

But even among students who did have access, a deeper pattern emerged. Survey data from 2019 (pre-pandemic) and 2021 (during the pandemic) show that students from lower-income households consistently spent more time on screens than their higher-income peers.

If technology were an equalizing force, increased exposure should have narrowed gaps. It didn’t – instead, gaps grew substantially wider. This aligns with recent longitudinal research, which concludes, “[A] causal inference can be drawn [that] the use of EdTech contributes to the widening achievement gap between high and low achievers.”

Why?

Because digital environments train patterns of fragmented attention and habitual multitasking. The more time students spend switching between messages, videos, and feeds, the more that becomes their default mode of engagement.

When those same devices appear in school, the habits follow. Much like a Pavlovian response, repeated digital exposure conditions students to more distraction, shallower processing, and weaker learning—especially those students who depend most on structure and human guidance.

THE DATA PROBLEM

This pattern also shows up across large-scale datasets.

When we examine long-term U.S. NAEP trends across all states normalized to 2012 (the final trend-data year before more than half of states crossed their digital inflection point), the divergence between lower- and higher-performing students becomes unmistakable.

Lower-performing students have declined far more sharply than their high-performing peers.

And the pattern is not unique to the United States. Across both TIMSS and PISA, the gap between higher- and lower-performing students has widened by roughly 20 points and 10 points, respectively, since 2012.

In fact, within the nine countries with data from both the 2019 and 2023 TIMSS and ICILS cycles, the gap between the top and bottom performers widened by over 11 points in just four years – even as the proportion of students reporting daily computer use for learning increased sharply, both in and out of school.

Although this evidence alone cannot prove causation, it does show that the achievement gap has continued to expand alongside the widespread adoption of digital tools. If technology were the great educational balancer, then we would expect to see at least some counteracting effect.

We don’t.

THE ACCESS ARGUMENT FALLS APART

A common response to these patterns is that technology may still benefit learning, but only for those higher-performing students who have greater access. But with roughly three-quarters of students reporting daily technology use for learning at home, access is no longer scarce enough to explain a global pattern of widening gaps.

More importantly, the data contradict the prediction this argument makes. If technology primarily benefited top students, we would expect to see stable performance at the bottom (where no tech-use occurs) and rising performance at the top (where technology is supposedly effective). Instead, across NAEP, PISA, and TIMSS, we see the opposite: performance at the top is flat or declining slightly, while the lowest-performing students are falling rapidly.

Put simply, the widening gap is not being driven by the top pulling ahead – it’s being driven by the bottom falling behind.

SO NOW THEN…

If we define ‘equity’ narrowly – as providing some access to learning where none previously existed – then yes, digital technology may be helpful.

But that’s not the promise being made in modern school systems. There, the claim is not better access – it’s better learning.

In practice, however, educational technology functions less like an equalizer and more like a cognitive amplifier: students who already possess knowledge, discipline, and academic support may gain modest benefits (or at least suffer less harm), while those lacking these foundations fall further behind.

Whatever else digital tools may be doing in education, closing equity gaps is not one of them.

If anything, EdTech appears to be widening them.

i-Ready began in 2011 as an adaptive diagnostic test: a tool designed to identify gaps in student reading and mathematics skills. Today, however, it doesn’t merely diagnose problems — it attempts to fix them, assigning students to ‘personalized’ lessons based on their diagnostic results.

What began as a measurement tool has become a full-service teaching system.

According to company reports, i-Ready is used by over 13 million K-8 students in the United States — representing roughly one in three elementary and middle schoolers.

With such widespread adoption, you’d expect a deep body of research demonstrating its effectiveness.

But you’d be wrong.

Where’s The Data?

Over the past two weeks, I’ve searched extensively for credible evidence of i-Ready’s impact on student learning. Here’s what I found:

• ZERO randomized controlled trials

• ZERO top-tier (Q1 or Q2) peer-reviewed journal articles

• TWO lower-tier (Q3) journal articles

  • This article suggests i-Ready diagnostics are less predictive than traditional end-of-year state assessments

  • This article measures improvements only within i-Ready itself — not beyond the tool

• TWO unlisted journal articles

  • This article finds no difference between users and non-users

  • This article does not include any form of a control group

• A LARGE VOLUME of gray literature

  • Dissertations, vendor reports, conference posters, and white papers.

That’s it.

Fifteen years. Thirteen million students. Not a single high-quality, independent study showing i-Ready improves learning.

What About the Johns Hopkins studies?

i-Ready advocates often point to two studies from the Johns Hopkins School of Education: one examining reading, the other mathematics.

A few important caveats

• Both studies were conducted in partnership with Curriculum Associates, the developer of i-Ready.

• Both were released as white papers — meaning they were not peer-reviewed.

The name ‘Johns Hopkins’ carries weight, however, so it’s worth examining what they actually found.

On reading, the researchers concluded: “Results of this study did not show any statistically significant associations between i-Ready Personalized Instruction usage and SBA ELA achievement.”

In plain terms, they found no measurable impact.

On math, the researchers reported a positive impact of i-Ready. However, a closer look at the data tells a different story:

Across grades, students using i-Ready improved by 7.8%, while students not using it improved by 7.0% — a difference of less than one percentage point.

In a sample of 7,646 students, at roughly $70 per student, that amounts to over $535,000 spent on i-Ready to achieve a 0.8% difference in mathematics performance, with no measurable change in reading.

In medicine, we distinguish between statistical significance and clinical significance. The former tells us that a result is unlikely to be due to chance; the latter asks whether that result is actually meaningful for patients. In other words, a finding may be detectable in the data, but still be irrelevant in practice.

By that standard, even if i-Ready math produces statistically significant differences, the effect is clinically meaningless.

The Bigger Problem

Not only is there zero high-quality evidence supporting the widespread adoption of i-Ready, but the platform itself appears to misunderstand why certain teaching practices improve learning in the first place.

To see this, we need to distinguish between two concepts:

• Summative Assessment: End-of-unit or end-of-term evaluations that result in a final mark or grade (e.g., an AP exam).

• Formative Assessment: Ongoing, low-stakes checks for understanding designed to inform the learning process (e.g., a weekly quiz).

For decades, research has shown that formative assessment is one of the most powerful drivers of student learning. The entire field of adaptive tutoring — including i-Ready — is built on this idea: regularly assess students, provide feedback, and tailor instruction accordingly.

But there’s a critical question that is rarely considered: why does formative assessment work?

The common assumption is that its power lies in the immediate feedback given to students — that by seeing their mistakes, students can adjust and improve.

However, decades of research from Welsh educationalist Dylan Wiliam suggest a different mechanism. Formative assessment works primarily because of the information it provides to teachers — information that allows instruction to be adjusted in response to student understanding.

In fact, formative assessment is only ‘formative’ when the information it generates changes what the teacher does in response.

When teachers design assessments, review responses, and provide feedback, they gain insight into what students understand, where they struggle, and what they need next. This allows them to adapt instruction in real time — adjusting explanations, pacing, and examples to guide students through new concepts. Importantly, human teachers can adapt across emotional, cognitive, and behavioral dimensions in ways no digital program can.

And this is where the problem emerges.

When students use i-Ready, the system determines which questions are asked, what feedback is given, and how instruction is sequenced. The only thing teachers see are summary performance metrics. They don’t see the questions, the responses, or the misconceptions as they emerge. And critically, teachers cannot use summary performance metrics to refine their instruction.

Put simply, the key mechanism that makes formative assessment effective — its ability to inform and adapt teaching — is largely removed. What remains is abundant activity, but little learning.

So Now Then…

If one of the most widely used tools in American education operates without an evidential base, this signals a broader problem with how we evaluate EdTech.

Somewhere along the way, we stopped asking, “Does this actually help students learn?” — or perhaps we never seriously considered this question in the first place

We’re not asking for miracles. We’re merely asking for evidence.

NOTE: If you know of any peer-reviewed i-Ready research from academic outlets I have missed, please send along and I will update the article accordingly.

Then more states adopted seatbelt laws. And each time they did, fatalities fell.

This is called staggered policy adoption, and it is one of the strongest natural tools social scientists have for identifying causal effects. When different jurisdictions implement the same policy at different times, and outcomes shift in alignment with those adoption dates rather than with a single calendar year, researchers gain strong evidence that the policy itself is driving the change.

Why does this matter?

Subscribe now

Staggered EdTech Adoption

During my recent Senate testimony, I stated that when NAEP performance is aligned with state-level digital adoption, scores plateau and then decline.

I should note that in that testimony, I misspoke slightly - referring to ‘one-to-one’ deployment when the analysis, in fact, examines broader, statewide digital adoption. The distinction matters, and I want to be precise.

Since that testimony, I’ve received repeated requests to publish the underlying data. What follows is that analysis.

In the United States, education policy is largely controlled at the state level. As a result, digital infrastructure was not embedded into classrooms everywhere at once. Some states operationalized statewide digital testing and instructional systems earlier; others followed later.

When each state’s digital inflection year is aligned within a common event-time framework and mapped against NAEP performance, a striking empirical signal emerges.

For readers outside the United States, the National Assessment of Educational Progress (NAEP) is the largest nationally representative assessment of student learning. Often called ‘The Nation’s Report Card’, it tests 4^th and 8^th grade reading and mathematics every two years. Importantly, NAEP remains anchored to its original 1992 scoring scale, allowing genuine longitudinal comparison.

(NOTE: Florida and Texas represent the early and late ends of adoption. Because each lacks meaningful comparator states at those extremes, their outermost data points were excluded from slope estimation to avoid distortion.)

Across state after state, scores in both 4^th and 8^th grade rose steadily for many years prior to large-scale digital adoption. After adoption, however, the trajectory shifts - often sharply - toward decline.

To put some values to this:

• YEAR 4 MATH: -1.45 points per year

• YEAR 4 READING: -1.07 points per year

• YEAR 8 MATH: -1.81 points per year

• YEAR 8 READING: -1.16 points per year

The magnitude increases from grade 4 to grade 8, consistent with intensifying digital exposure in later schooling.

But Correlation Isn’t Causation…Right?

‘Correlation is not causation.’

Although strictly speaking true, this is one of the most misused phrases in public debate.

This warning only makes sense when examining a single, isolated dataset.

For example, swimming correlates with ice cream consumption. That sounds meaningful until you realize both are driven by a third variable: heat. In summer months, when temperatures rise, people tend to swim and eat ice cream. The correlation is real - but there is no genuine causal link between them.

Now imagine something different.

Imagine that the same correlation appears repeatedly across different populations, different cultures, different countries, different age groups, and independent datasets - and that it demonstrates a consistent dose-response pattern: as exposure increases, outcomes worsen proportionally.

This is how we move beyond loose association into the realm of genuine pattern with justified causal inference.

Returning to swimming and ice cream - the correlation disappears beyond summer conditions. It does not hold for indoor winter swimming. It does not hold for athletes training year-round. It does not hold in countries where dairy consumption is minimal. It does not hold where swimming is mandatory (such as military training or among sea nomad communities).

With classroom technology, the opposite occurs.

Across international assessments (PISA, TIMMS, PIRLS), and across national testing (NAEP), the same pattern appears again and again.

Routine digital exposure in instructional contexts is associated with weaker academic outcomes in a dose-response relationship. This pattern appears across states, countries, grade levels, subjects, and years.

And notably, there is no robust, internationally consistent signal suggesting that routine classroom digital exposure systematically improves learning outcomes. To be fair, there are isolated positive findings - but they do not replicate across contexts with the strength, scale, or consistency of the negative association.

Is this causation in the strictest sense? No. But it does allow for far stronger inference than the slogan “correlation is not causation” would suggest.

That phrase was meant to prevent naive conclusions - not to justify dismissing replicated, cross-context patterns spanning decades. To dismiss this data may feel like scientific rigor - but in truth, it’s little more than selective skepticism.

So Now Then…

As in the seatbelt example, the repeated alignment between policy timing and outcome-change across jurisdictions provides strong evidence that digital technology is contributing to the observed declines in academic performance.

Three additional clarifications.

First, this is not a COVID story. Performance declines begin prior to the pandemic and align with widespread digital adoption - not lockdown timing. To remove any ambiguity, the graphs above exclude data from the 2022 NAEP cycle - the year most disrupted by COVID effects.

Second, as noted above, NAEP scores are not periodically re-normed. Unlike many assessments that reset scoring scales over time, NAEP remains locked to its original 1992 scale. As a result, these declines reflect genuine changes in measured performance rather than statistical adjustments.

Third, to strengthen causative inference drawn from scaled and repeated correlational patterns, two additional elements are necessary: converging academic research and plausible biological mechanisms that explain the findings more convincingly than competing alternatives.

As explored in The Digital Delusion, both lines of evidence exist - and they align converge on the same patterns observed in the large-scale data.

A Note About Digital Inflection Points

Unfortunately, there is no central database that lists when digital technology shifted from peripheral to central within education across each state. As such, I felt it best to include my determinations.

NOTE: Digital inflection years were defined as the first operational period in which statewide accountability systems or legislative mandates required routine computer-based administration or mandatory online instructional participation, thereby institutionalizing digital infrastructure across districts. Where a clear statewide trigger existed (e.g., mandated online coursework or formal transitions to computer-based statewide assessments), the implementation year of that policy was used. In states without a single discrete triggering event, the inflection year was determined by the operational transition to routine computer-based statewide assessment across the majority of districts, as evidenced by coordinated device deployment, broadband readiness, and testing infrastructure requirements. Sensitivity analyses shifting inflection years by ±1–2 years did not materially alter event-time estimates, indicating that results are robust to reasonable timing variation.

At one point, Zuckerberg was questioned about Meta’s use of beauty filters: digital effects that make users, including children, appear younger, fitter, and more conventionally attractive in photos and videos.

The prosecution referenced Meta’s own internal review, Project MYST. According to reports, 18 out of 18 wellbeing experts who evaluated the psychological impact of these filters raised serious concerns about potential harm to young users’ mental health.

Despite those warnings, the filters remained available.

Zuckerberg’s defense rested on a familiar line of reasoning: there was no peer-reviewed, causal evidence demonstrating this specific product directly harmed children. Absent validated proof of causation, harm could not be established.

“There is no evidence of harm.”

This is the same argument now being deployed by EdTech lobbyists at statehouses across the country as lawmakers attempt to regulate classroom technology.

No Evidence of Causative Harm

This year, more than a dozen bills aimed at regulating EdTech have been introduced across at least nine states. Utah’s SAFE and BALANCE acts led the way, followed closely by Vermont’s effort to formalize parental opt-out rights and Tennessee’s proposal to remove digital devices from primary classrooms.

These efforts are informed by decades of research showing that, on average, classroom technologies do not outperform – and often underperform - well-implemented analog instruction.

Despite strong bipartisan support in most states, pro-tech lobbyists are pushing back with a familiar refrain: “There is no evidence of harm for emerging EdTech products.”

Strictly speaking, that statement is often true.

Educational technology evolves so rapidly that by the time researchers evaluate one platform, it has already been patched, rebranded, or replaced. Product-specific causal evidence is perpetually just out of reach.

But this is not a scientific defense. It is a misleading procedural maneuver.

When Causation Becomes Dangerous

Demanding product-specific, long-term, high-risk causative trials in children sets an unrealistic and ethically impossible standard.

Returning to beauty filters, no ethics board would approve a study deliberately exposing children to a tool that 18 experts consider risky simply to ‘prove’ harm. That is why no randomized control trial has tested whether these filters damage young users’ mental health – the likely harms of such a study outweigh any possible benefits.

Luckily, we don’t live in a vacuum.

A substantial body of correlational research links image manipulation and filter use to body dissatisfaction, self-objectification, weight concerns, and reduced wellbeing. The experts reviewing Meta’s policies were not guessing – they were applying decades of psychological research to a new technological wrapper.

Software changes. Human biology does not.

The same logic governs learning.

Returning to Utah

Utah’s digital inflection year occurred in 2014, corresponding with the statewide launch of SAGE - a fully computerized adaptive assessment system. Before this, digital tools were largely peripheral in Utah classrooms. After this, they became structurally embedded.

Before widespread digital adoption, Utah NAEP scores rose consistently from 1992 through 2013. Pooled by subject and indexed to 2013:

• Math scores increased +0.76 points per year

• Reading scores increased +0.14 points per year.

After 2014, the slopes reversed:

• Math scores declined -0.39 points per year

• Reading scores declined -0.88 points per year.

This represents a structural swing of -1.15 points per year in math, and -1.02 points per year in reading. Importantly, excluding 2022 - the year most impacted by COVID-related closures – leaves these swings essentially unchanged: -1.05 points per year in math and -1.07 points per year in reading. In other words, this pattern is not a lockdown artifact - it’s a structural break beginning in 2015.

These are correlational patterns, but so were the early signals about smoking, lead exposure, and beauty filters.

When consistent patterns appear across nearly all 50 states’ NAEP data and across dozens of countries’ PISA, TIMSS, and PIRLS results - and when those patterns align with established cognitive mechanisms - we are no longer looking at coincidence. We are looking at converging evidence.

And what we cannot ethically do (just as with beauty filters) is deliberately expose children to systems we have strong reason to believe may undermine learning simply to satisfy an unrealistic evidentiary demand.

Demanding perfect causation before action doesn’t protect children; it protects developers.

A Generous Interpretation

Even if we assume the decline argument is overstated – that Utah’s NAEP data has merely ‘plateaued’ since 2014 - the harm does not disappear.

Between 2015 and 2025, Utah invested roughly $500 million in K-12 educational technology. If half a billion dollars produces stagnation, that’s not neutral.

Every dollar committed to devices and platforms is a dollar not spent on interventions we know improve learning: teacher development, structured literacy programs, small-group instruction, targeted support for struggling students.

Even under the most generous interpretation of the data, the opportunity cost alone is staggering.

So Now Then…

Demanding definitive causative proof of harm before acting to protect children sets an unrealistic and dangerous standard. If we wait for perfect causation, we will always act too late.

Our society does not demand product-specific randomized trials before regulating food additives, vehicle safety standards, or consumer protections. We act when converging evidence suggests that risk outweighs benefit.

Education should be no different.

When billions of dollars and millions of children are involved, the burden of proof should rest on demonstrating clear, durable, replicable benefit – not on proving harm after the fact.

Caution is not fear, and restraint is not regression. They are marks of a society that prioritizes children over products.

Nearly fifty years have passed since that original article, and in that time hardware, software, and digital platforms have transformed beyond recognition. We have seen the emergence of the internet, touchscreens, the stylus, cloud computing, and now artificial intelligence.

And yet, half a century later, the combined effect size of digital technologies on learning remains… 0.29.

Despite extraordinary advances in speed, power, and sophistication, the educational impact of these tools has barely moved. This consistency suggests the limitation of digital technology in the classroom is not primarily one of training, implementation, or context. Rather, it lies in the deeper tension between how digital systems function and how human biology learns most effectively.

Still, whenever I speak about EdTech – whether it’s with parents, teachers, or administrators - there is always one question I receive:

“So, when is technology good for learning?”

I’m not going to lie: this confuses me to no end.

The Underlying Assumption

Embedded within every question is an assumption about how the world works: which definitions matter, which possibilities are worth considering, and which answers will be treated as relevant or irrelevant.

If someone asks a doctor, “Under what circumstances should I be taking Tylenol to relieve a headache?” the question assumes that headaches are a solvable problem, that Tylenol is a viable solution, and that not all circumstances are considered equal when it comes to ingesting medication.

By contrast, it’s difficult to imagine a patient asking, “Under what circumstances should I smoke cigarettes to improve my eyesight?” because nobody meaningfully assumes that smoking could (or should) positively influence vision.

This is not to say cigarettes could never be engineered to improve eyesight. Perhaps someone with the gumption could spend the next decade developing a retinal-enhancing nicotine stick. But why pursue such a path when we already possess far better tools, treatments, and procedures for supporting vision?

When people ask, “When does edtech work?” the question carries two embedded assumptions: that learning is an achievable goal, and that digital tools are plausible mechanisms for achieving it.

The question itself – especially when asked after reviewing negative evidence – begs the answer. It presumes that digital technologies are a natural educational solution just awaiting proper implementation. The starting assumption is one of efficacy and the burden of proof shifts to the skeptic.

But what if the more appropriate question is not, “When does it work?” but rather:

“Why do we assume these tools should work at all?”

In domains where clear, effective strategies already exist (long-form reading, handwriting, explicit instruction, guided practice, deliberate feedback), we do not normally replace proven methods with experimental ones and ask society to wait while the evidence catches up. The burden of proof remains with the claimant; only when a proposed alternative clearly demonstrates superiority should the pendulum of practice begin to shift.

Perhaps in business, where the bottom line is profit, the concept of ‘disruption’ makes sense: replacing what works with something that possibly works better might be tolerable when the primary downside is financial loss. But when the bottom line is health (as in medicine), safety (as in engineering), or learning (as in education), disruption becomes a far more dangerous idea. Here, the guiding principle should not be ‘embrace the new until proven harmful’, but “first, do no harm.”

Three Exceptions

With this standard in mind, there are only a few narrow circumstances where educational technology consistently meets that burden.

• 1) Something is Better Than Nothing

  When learning would otherwise be completely precluded in the absence of technology, then its use is not only appropriate – it’s essential.

  • Do you have a student with a motor disability who cannot write without technological assistance? Bring in EdTech.

  • Are you an F1 driver who cannot shut down the streets of Monaco to practice race laps? Bring in EdTech

  • Has a global pandemic forced schools to shut their doors? Bring in EdTech.

  When there is no viable alternative, digital tools provide access that would otherwise not exist. But if you have the luxury of choosing between two options, always select the option better suited to your desired outcome. And if your desired outcome is deep learning, then technology will almost never be the superior option.

• 2) Intelligent Tutoring

  There is evidence that using digital technology can be beneficial when used to drill well-defined skills. However, two important caveats must be considered.

  First, adaptive tutoring systems work best in highly constrained domains, typically those with clear right-or-wrong answers such as mathematics or grammar. These programs show far weaker effects in subjects that demand open interpretation or complex reasoning, such as literature or civics. As MIT Professor Justin Reich notes, “…adaptive tutors have some applications in learning to read, but very limited applications in reading to learn.”

  Second, improvements often remain context-bound: students may demonstrate gains within the tutoring environment itself, but struggle to apply that learning outside the program. As Sydney Pressey, inventor of one of the earliest teaching machines, lamented back in 1926, “If the [learning material] is rephrased or presented in a different context, almost nothing seems retained.”

  In short, adaptive tutoring can be effective when the goal is fluency within a tightly defined skill, but its benefits narrow considerably when deeper understanding or transfer is required.

• 3) Specific Learning Disorder Intervention

  There is also evidence that digital technologies can prove useful in the remediation of specific learning disorders where consistent and repetitive training is required (e.g. dyslexia and dyscalculia). As psychologist Bruce Pennington notes, “Children with [specific learning disorders] do not appear to require a qualitatively different instructional approach from typical learners; they may just need more of it, broken into smaller steps, with more opportunities for review…”

  Astute readers will recognize that this emphasis on repetitive, structured practice is exactly what adaptive tutoring programs (mentioned above) provide. As a result, we observe benefits for learners with specific learning disorders for the same reasons intelligent tutoring can be effective – along with the same two important caveats.

So Now Then…

EdTech was introduced to education as a gateway to exploration and creative expression. It was promoted as the means by which classrooms would move beyond the supposedly outdated traditions of memorization and repetitive practice into a future of personalized, self-directed discovery.

Accordingly, it’s more than a little ironic that digital tools show their strongest benefits when used for the very practices they were meant to replace: rote memorization and skill drilling.

This certainly says something about the power of marketing hype in the EdTech sector. But it also says something important about learning itself. Learning is often effortful, difficult, and sometimes tedious. It requires repetition and sustained practice to succeed. Rather than viewing these characteristics as flaws to be engineered away, we may be better served leaning into them and designing instruction that works with (rather than against) the biology of how learning actually occurs.

For decades, education has asked “When does EdTech work?” – a question that tacitly assumes the answer must be affirmative. But when the question changes to “Why should this improve learning at all?”, the conversation changes with it – and so does the burden of proof.

We demand proof from our medicines, our cars, even our household appliances. Why should we demand anything less from the tools placed in front of our children?

• “Our kids are less cognitively capable than we were at their age.”

Perhaps unsurprisingly, that claim traveled quickly.

At first, it circulated in a relatively measured way (Gen Z is Less Cognitively Capable Than Their Parents). More recently, however, it has taken on a far more sensational tone (Gen Z is Officially Less Intelligent Than Millennials).

Such is the media cycle.

But beneath the headlines lies a serious question: is something measurable actually changing in human cognition?

If you’ve read The Digital Delusion, you already know the answer is a clear yes. Still, to help ground the broader discussion, I want to focus on one specific strand of evidence the media has latched onto: IQ Scores.

IQ and The Flynn Effect

Throughout the 20^th century, IQ scores showed a remarkably consistent pattern: they rose by approximately three points per decade – that’s roughly 6 points per generation.

This phenomenon, known as The Flynn Effect, has been documented across a wide range of countries and contexts, spanning Western and Eastern nations, as well as both developed and developing economies.

Until recently.

Beginning around the turn of the 21^st century, reports started to emerge suggesting that full scale IQ scores were no longer rising – they were beginning to decline. Across a handful of well-designed studies, the drop has been estimated at as much as six to eight points per decade.

This reversal, known as The Negative Flynn Effect, has been reported primarily in high-income western nations, though its timing and magnitude vary by cohort and country.

One of the most recent and detailed analyses exploring this phenomenon was published in June of 2025 in a paper called Digging In. Let’s take a closer look.

Digging In: The Reverse Flynn Effect

Intelligence tests typically pool together many different kinds of tasks – ranging from visual to verbal; speeded to deliberate; concrete to abstract. While these tasks are often reported as a single IQ score, they draw on fundamentally different kinds of cognition.

One useful way to divide tasks is by considering whether they’re externally or internally scaffolded.

• EXTERNALLY SCAFFOLDED

  In these visually mediated tasks, the structure, rules, and relationships are explicitly represented in the environment. A substantial portion of cognition is offloaded onto the task itself, placing minimal demands on sustained memory, prior knowledge, or internal transformation. Success is driven primarily by visual perception, pattern detection, and rapid rule application.

• INTERNALLY SCAFFOLDED

  In these language-mediated or symbolic tasks, the structure, rules, and relationships must be constructed, mediated, and manipulated internally. This places high demands on working memory, accumulated knowledge, attentional control, and executive functioning. Success is driven by language, memory, and internally generated abstraction, often requiring sustained mental effort over time.

Put simply: externally scaffolded questions show you the structure and ask you to recognize it, while internally scaffolded questions hide the structure and ask you to create it.

As an example of an externally scaffolded task, which image completes the pattern below?

All the relevant information is visible on the screen: the task itself holds the structure.

Now consider an example of an internally scaffolded task:

Here, you must generate the relationship yourself, drawing on language, memory, and abstract reasoning. Thinking happens internally.

Although intelligence tests do not formally classify subtests as “externally” or “internally” scaffolded, the distinction reflects a well-established cognitive divide between stimulus-supported perceptual tasks and internally generated reasoning tasks that depend on memory and executive control.

Returning to the article at hand, these researchers explored IQ change across the last two decades and found a general flattening. The Flynn effect appears to have disappeared for modern teenagers – but with an important underlying pattern.

Adolescent performance on Externally Scaffolded tasks showed moderate growth:

By contrast, adolescent performance on Internally Scaffolded tasks showed significant declines:

At face value, this pattern suggests that Gen Z can still perform well on highly visual, externally guided tasks - perhaps unsurprising given the increasingly visual, heavily guided nature of the digital tools they use daily.

Concerningly, adolescents appear to be struggling with knowledge-based, internally sustained forms of cognition – those that depend on language, memory, and manipulation without visual support. This includes significant declines in core memory functions like working memory and sequential manipulation (abilities psychologists long believed to be biologically grounded and relatively resistant to environmental change).

That said, this is too simple an interpretation. Other studies - using different tests, cohorts, and countries – have documented declines in both internally and externally scaffolded abilities, including measures of abstract visual reasoning. When viewed across the full body of evidence, the neat distinction between what is lost and what is preserved begins to blur.

The more sobering conclusion is this: we are not merely seeing a reshaping of cognitive strengths, but signs of a broad-based cognitive decline – one that first becomes visible in internally sustained thinking, but that does not necessarily stop there.

So Now Then…

This leaves us with one final question: is IQ synonymous with Intelligence?

The short answer: no.

If we take the Flynn Effect as read, Millennials average roughly 30 IQ points higher than their great-great-grandparents. If IQ truly measured intelligence itself, that would imply either that over half my generation qualifies as genius – or, conversely, that much of ‘The Great Generation’ was cognitively impaired.

Neither conclusion makes sense.

In truth, IQ is better understood as a measure of school-ability: a particular form of thinking that structured education systems have favored and cultivated for centuries. Crucially, this kind of thinking can be trained.

This is why the relationship between years of schooling and IQ is nearly perfect. In fact, research suggests each additional year of formal education is associated with an increase of roughly 1-5 IQ points – and this relationship appears to be causal, not merely correlational.

For that reason, we cannot concretely say that Gen Z is less intelligent than Millennials. What this data does suggest is something narrower but arguably more troubling: despite spending more time in school than any generation before, Gen-Z is losing school-ability.

In other words, something has shifted within schools themselves. The values, habits, and cognitive skills that education once reliably supported are no longer being cultivated in the same way.

This shift can’t be blamed on teachers, who remain as dedicated and motivated as ever. Nor can it reasonably be pinned on curriculum, which is arguably more complex and demanding than at any point in history.

The change, I would argue, lies largely in the tools we now use to mediate learning.

Screens tend to support cognitive outsourcing rather than deep thinking. The more students rely on easy, supportive digital tools, the less friction they encounter and the less mental effort they must exert. But friction is not a flaw of learning: it is learning.

When school becomes too smooth, too guided, too effortless, thinking itself begins to erode. Accordingly, we should not be surprised when an entire generation educated under these conditions begins to show cognitive declines unlike any we have previously documented.

So I’ll end where I started:

• “Our kids are less cognitively capable than we were at their age.”

What happens next depends on whether we continue to embrace tools that do the thinking for students, or whether schools demand students return to doing the thinking for themselves.

Rather than relying on any single study, meta-analyses aggregate dozens – sometimes hundreds – of independent experiments to estimate an overall impact.

The output of a meta-analysis is typically represented as an effect size: a standardized number capturing both the direction and magnitude of an intervention’s impact. Positive values suggest benefit; negative values suggest harm.

Over the past few years, the global effect size reported in EdTech meta-analyses has exploded - from roughly 0.30 in 2018 to well over 1.00 in 2024.

That should make us stop cold.

It’s extraordinarily unlikely (bordering on impossible) for the true impact of any large-scale educational intervention to triple in such a short period of time. When effects inflate this rapidly, it almost always signals a methodological problem rather than a genuine breakthrough.

So, what’s really happening?

NOT ALL RESEARCH IS CREATED EQUAL

Across academia, research outlets vary dramatically in rigor, transparency, and review standards. Broadly speaking, the EdTech literature can be grouped into four tiers:

• Tier 1
  These experiments are published in Q1 and Q2 journals. These outlets are highly selective, apply demanding peer review, and require strong methodological and statistical transparency. As a result, findings tend to be more robust and defensible.

• Tier 2
  These experiments are published in Q3 and Q4 journals. These outlets often publish competent work, but typically apply looser review standards and shorter timelines. This increases variability in methodological quality and raises questions about long-term reliability.

• Tier 3
  These experiments are published in unlisted or non-indexed journals. These outlets often operate with minimal editorial oversight, inconsistent peer review, and weak reporting standards. Findings from these sources should be interpreted with substantial caution.

• Tier 4
  Dissertations, conference proceedings, technical reports, book chapters, and other forms of “gray literature.” These outputs often represent early-stage or exploratory work and undergo limited formal vetting. Their findings are provisional until independently replicated.

WHY THIS MATTERS

I randomly selected:

• One EdTech meta-analysis published in 2019.

• One EdTech meta-analysis published in 2024.

I then traced every included study to determine which tier the underlying evidence came from.

The result?

Surprise surprise, the post-2020 meta-analysis was overwhelmingly dominated by Tier 3 research - studies from unlisted or non-indexed journals.

And when you break the data down by tier, a clear pattern emerges:

Perhaps unsurprisingly, effect sizes from Tier 2 and Tier 3 sources are dramatically larger than those from Tier 1 journals.

When review standards weaken, then transparency declines, methodological scrutiny softens, and inflated effects become far easier to publish (and far easier to aggregate in meta-analyses).

THE TAKEAWAY

Claims that “EdTech is finally getting better” are not supported by improved outcomes or stronger evidence. What has changed is the quality of research being pooled.

We are largely where we have always been. The optimism comes not from better technology, but from noisier data.

In an upcoming post (and video for subscribers), I’ll walk through a simple, practical way to identify which tier a given study comes from - no advanced statistics required!

Until then, additional data is available in my new book The Digital Delusion - and here’s a short video exploring the same problem in the context of ChatGPT research:

Subscribe now