Fascinating data dive. The $535K for 0.8% difference is the smoking gun here.
What really strikes me is how this mirrors the broader EdTech problem: we're treating formative assessment as if it's about the feedback loop to students, when decades of research (Wiliam, Black, Hattie) show the mechanism that matters is the information it provides to teachers. i-Ready takes the diagnostic data away from educators and leaves them with summary dashboards that can't inform actual instructional decisions.
The fact that even the "positive" Johns Hopkins findings show no meaningful impact on reading and a clinically irrelevant difference on math tells you everything about the field's willingness to accept "statistically significant" as a substitute for "actually helpful."
What the author of this post failed to include, which doesn’t surprise me since he doesn’t appear to understand evaluation research, is that in addition to statistically significant differences on mathematics scores, the effect sizes were of values that are considered of medium effects. We don’t interpret effects by a percentage difference. Further, comparison students scored significantly higher than treatment students on pretest, so the treatment students had a somewhat greater uphill battle that can’t be captured the way this author reported. Lots more to the story
I agree with you that if you are going to reanalyze the data away from how education research is typically analyzed, this needs to be both noted and explained, but I see no evidence of that in this substack.
However, you are right that there is a lot more to the story, including this being a badly designed and poorly timed study. A quasi-experimental design where the experimental and control groups are not very similar is bad enough but understandable at times (though less understandable when the funding agency is a massive corporation), but then collecting the data during the 2020-2021 school year in Massachusetts was even worse. The students would have all been virtual for nearly the entire school year.
And since they were using readily available school data, why did they not compare the students' baseline MCAS scores from the prior year to the MCAS scores the following spring after the intervention, instead of comparing pre/post test on two different measures - iReady for the fall and then MCAS for the spring? Because the MCAS wasn't administered in Spring 2020 due to the pandemic is likely one reason. But the whole paper is weirdly silent on the whole COVID thing throwing a big wrench into their study and all of the resulting limitations to the study and any ability at all to generalize the results because of that...
Additionally, there were actually low rates of student usage in most of the grades they highlight as having significant results, so it's not clear to me how they can take credit for improvements in say, 6th grade math scores when only 29% of the students actually used the program as intended...
@John Allen Wooden - fantastic deep dive on your end too. The five Johns Hopkins commissioned studies point is huge - CRRE has been churning out vendor-funded papers for ed-tech publishers for years. And your point about the "Research Library" being an SEO-optimized facade for Curriculum Associates' in-house science-washing is spot on.
The pattern here is clear: we're seeing decades of what looks like "peer-reviewed" or "expert-validated" EdTech that's really just vendor-funded marketing dressed up as research. It's not just i-Ready - I'm seeing this across the field.
Question for you: in your series, have you found any cases where districts broke with i-Ready after realizing the data gap? Would love to see examples of successful exits from the i-Ready trap.
Tech-based "personalized learning," sans any supporting evidence, has contributed to the near-death of actual teaching--but still has incantatory power. It points up Daniel Kahneman's lament that even the most groundless ideas will acquire "the illusion of validity" if they are endlessly and enthusiastically repeated.
Texas uses I ready- and the state politicians get kick backs
See the state decline in outcomes should be enough. But TX children are commodities and data is gold. Ready for you to come to TX- we are alone out here in the tumbleweed fighting for 6 years to save 10% of the nations children from mind altering decay.
As a teacher, I pointed out your exact conclusion about formative assessment needing to inform and drive teacher decisions…and that the inability of teachers to analyze for misconceptions or weaknesses in an area made it impossible to react properly to the data! I left teaching ELA because of the sham…. BUY the modules from the same company that provides the test… but no info from iReady for what a non-module school could do to analyze data… It’s just a box-checking money maker at this point… money schools need for other things!
Thank you for this interesting and informative piece. It would be interesting to examine whether or how I-Ready changes instructional practices by teachers -- more reliance on computer-based lessons, less partner work, etc. No doubt difficult to obtain such data, given that it seems no one has bothered to examine the direct effects of I-Ready!
1. Re: Johns Hopkins Commissioned Marketing Studies: There have actually been at least five of these for i-Ready, not just two. The Hopkins CRRE group is prolific in producing such commissioned studies for many ed-tech publishers. You're right that the Hopkins name carries weight, and the degree to which they trade on it to produce "studies" in "partnership" with their paying clients represents a serious scandal in the making.
2. The role that i-Ready's own painstakingly search-optimized "Research Library" (https://www.curriculumassociates.com/research-and-efficacy/research-library) plays in this data efficacy ecosystem (and hoodwinking school administrators) cannot be overstated. These officiously packaged documents rank at the top of most Google searches for information about the platform's efficacy. Most are products of Curriculum Associates' own in-house skunk works dedicated to science-washing i-Ready on a state-by-state level, with their own credentialed employees churning out officious "research" and "studies" which all declare i-Ready the cat's meow. Shocker!
Key understanding here: formative assessment is only helpful when teachers can use it to inFORM instruction. These tools are notorious for flooding teachers with unusable data. I think some administrators make the mistake of assuming that the data/insights will be easy to visualize--and that their teachers know how to use them. They don't. I don't even think most principals have a system for using this data.
Yes. This is such a great point. Data is only useful when it is digestible and actionable. Even trained researchers can get overwhelmed by too much data.
I think these tools also send a signal that their data should take precedence over a teachers own intuition. An experienced and well trained teacher might not be able to fully explain all of what they know about a student, but they will have a million nuanced data points and comparisons that help them guide a student effectively. We don't even know what we don't know, how can we program that full understanding of a good teacher into a digital tool?
Wow, that is such a great point. I’ve never thought of it that way. It’s like… How can we expect something that has been programmed by humans to be as smart as humans, when humans are constantly growing their understanding of students and their needs?
This is exactly what I mean when I say teachers deserve better. 13 million students. Zero meaningful evidence. But teachers are still being handed these tools and told to use them — while their professional judgment gets completely sidelined. This is why we built TDI: so educators have a space to cut through the noise and get PD that actually works. Glad someone is asking the hard questions. Keep going.
Thank you for breaking down the data (or lack there of), Jared! I have sat next to a number of students using the program and witnessed why a response was incorrect, impacted by so many factors that the program does not pick up (because it is not a human) much to the frustration of the student. So many students get frustrated and end up guessing in order to get through the assignment, so essentially data provided by the program does not accurately reflect their true abilities. There are countless other edtech programs being used that have all the same issues.
@jared What do we do about state requirements? Just got home from a school district tech task force meeting and they said Oregon is now going to require K-2 to take assessments on a device starting this fall. Previously they used an assessment called Dibels that was an in-person assessment and apparently the state does not have any device-free assessments on their "approved list" starting this fall. WHAT?! Are you aware of this? We need to contact our state legislator and the governor. Ridiculous. I was ready to help get the iPads out of K-2 and now the word is we have to prepare the kids to take assessments on devices. I can't believe it. That's backward from where we need to go...Would love your thoughts!!
Yes, Oregon just passed new legislation that all k-8 students in public schools will be tested 3 times per year with one of the approved online tests: MAP, i-ready, Star, or SBAC interim. Schools have no choice in the matter.
This is so frustrating--schools will now feel forced to cater to a curriculum that pushes online learning just so the kids can pass a test 3x per year. UGH.
DIBELS is an excellent benchmark screener for reading. As a reading specialist, I administered it 3 times a year school-wide K-5, and it provided such important insight for our MTSS process, with very little time taken from the students. What a shame that it is not on the approved list.
Just for clarification purposes, I believe you are referring to some kind of benchmark test taken 3 times a year. It’s frequently called “I-ready” or “I-ready diagnostic”. I believe the article is referencing both that AND the daily lessons students are expected to do on their devices, spending an hour and a half WEEKLY on math and reading lessons. This “iready” program is also called “my path”. In my state we’re required to give all students “universal reading screeners” three times a year do identify characteristics of dyslexia. I-ready diagnostic qualifies for that, but other programs do as well.
This is the pattern. Every tool that could inform teacher judgment gets converted into a replacement for it. The mechanism you're describing — formative assessment works because it changes what the teacher does, and i-Ready removes the teacher from that loop — is the same thing that happened to the LMS, the rubric, and the standardized test. The system doesn't adopt tools to improve learning. It adopts tools to remove judgment from the process, because judgment is expensive, unscalable, and impossible to audit. $535,000 for a 0.8% difference in math with zero impact on reading is not a failure of the tool. It's the tool performing exactly as the system needs it to — generating data, demonstrating compliance, and leaving learning to fend for itself.
This thread has been a masterclass in EdTech skepticism. John Allen Wooden's documentation of the CRRE/JHU relationship is particularly damning - five commissioned studies, all vendor-funded, all bearing Johns Hopkins' imprimatur. That's not just weak evidence, that's the academic equivalent of a company writing its own press release.
Molly, Erica, Manju - thanks for reading. The pattern here is so familiar: we've normalized "evidence" that's really just marketing dressed up in peer review clothing. What's your experience been? Are you seeing districts break free from these tools once they've done the audit? Or is the momentum so strong that escape is harder than it should be?
The bigger question: if we applied clinical significance standards (vs just statistical significance) to EdTech adoption, what would survive? The $535K for 0.8% math gains doesn't pass. But neither does any other adaptive tool I've reviewed that lacks independent, third-party RCTs.
I’m all about student growth and strategies to help teachers help students grow. I-ready provides an extensive report that breakdowns student strengths and areas of growth which are aligned with our state standards. This is something that can be given to parents and teachers can use this information to tailor lessons and/or create small groups to a. See if the student is truly struggling b. Provide targeted support for students because it identifies strengths and areas of growth. It’s a tool that can be used to support enrichment and remediation. This is how we used iReady as a formative assessment tool. Teachers are able to see the questions that students were asked (unless this has changed)
I’m only speaking from my experience with using iReady as a tool.
I appreciate this article because it does highlight the over reliance on programs and the drive for profit with Ed tech. I cannot speak to national statistics or data, but I can tell you from experience that iReady is a tool that can improve student performance if the data is used with intention along with systems in place that ensure consistency and accountability. IReady is a useful tool for teachers, students, parents, and school leaders. When schools use IReady just to put kids on a program, they are not utilizing all that IReady has to offer to support student growth and inform instruction. I’m not an iReady rep,
I’m just speaking from experience when I used it as a teacher, instructional coach, and district coordinator.
We use iready and the data reports never match what we see in the classroom. Also, the test is used as a bludgeon for districts to purchase their curriculum. It's a giant circular fire hose of money. It also takes SO MUCH TIME away from actual instruction. I could go on for hours how awful it is.
I think your tone needs to kick back a bit. She is entitled to her opinion, whether it disagrees with yours or not. Implying that she is bought and paid for just because she had a better experience is insulting.
Will this be the next Sold A Story - EdTech Edition? Shiny brochures and marking campaigns seemed to have worked wonders.
Final score:
Big tech 1
Students 0
I’m still furious about Sold A Story. I don’t think I’ll EVER get over it!
Fascinating data dive. The $535K for 0.8% difference is the smoking gun here.
What really strikes me is how this mirrors the broader EdTech problem: we're treating formative assessment as if it's about the feedback loop to students, when decades of research (Wiliam, Black, Hattie) show the mechanism that matters is the information it provides to teachers. i-Ready takes the diagnostic data away from educators and leaves them with summary dashboards that can't inform actual instructional decisions.
The fact that even the "positive" Johns Hopkins findings show no meaningful impact on reading and a clinically irrelevant difference on math tells you everything about the field's willingness to accept "statistically significant" as a substitute for "actually helpful."
What the author of this post failed to include, which doesn’t surprise me since he doesn’t appear to understand evaluation research, is that in addition to statistically significant differences on mathematics scores, the effect sizes were of values that are considered of medium effects. We don’t interpret effects by a percentage difference. Further, comparison students scored significantly higher than treatment students on pretest, so the treatment students had a somewhat greater uphill battle that can’t be captured the way this author reported. Lots more to the story
I agree with you that if you are going to reanalyze the data away from how education research is typically analyzed, this needs to be both noted and explained, but I see no evidence of that in this substack.
However, you are right that there is a lot more to the story, including this being a badly designed and poorly timed study. A quasi-experimental design where the experimental and control groups are not very similar is bad enough but understandable at times (though less understandable when the funding agency is a massive corporation), but then collecting the data during the 2020-2021 school year in Massachusetts was even worse. The students would have all been virtual for nearly the entire school year.
And since they were using readily available school data, why did they not compare the students' baseline MCAS scores from the prior year to the MCAS scores the following spring after the intervention, instead of comparing pre/post test on two different measures - iReady for the fall and then MCAS for the spring? Because the MCAS wasn't administered in Spring 2020 due to the pandemic is likely one reason. But the whole paper is weirdly silent on the whole COVID thing throwing a big wrench into their study and all of the resulting limitations to the study and any ability at all to generalize the results because of that...
Additionally, there were actually low rates of student usage in most of the grades they highlight as having significant results, so it's not clear to me how they can take credit for improvements in say, 6th grade math scores when only 29% of the students actually used the program as intended...
But are medium effects worth this much money?
Fair question. Kraft (2020) outlines small, medium, and large effects and honestly, obtaining a "large" effect in educational research is very challenging/rare. It looks like you can read his article here: https://static1.squarespace.com/static/6297c2b5c8bc35721cc7a65c/t/6859ae626e3b4a45dec1ded0/1750707811565/Kraft+2020+Interpreting+Effect+Sizes+ER.pdf and that will provide better context to the effects we regularly find in educational research. So many factors play in - a larger sample size may actually result in a lower effect size!
@John Allen Wooden - fantastic deep dive on your end too. The five Johns Hopkins commissioned studies point is huge - CRRE has been churning out vendor-funded papers for ed-tech publishers for years. And your point about the "Research Library" being an SEO-optimized facade for Curriculum Associates' in-house science-washing is spot on.
The pattern here is clear: we're seeing decades of what looks like "peer-reviewed" or "expert-validated" EdTech that's really just vendor-funded marketing dressed up as research. It's not just i-Ready - I'm seeing this across the field.
Question for you: in your series, have you found any cases where districts broke with i-Ready after realizing the data gap? Would love to see examples of successful exits from the i-Ready trap.
Tech-based "personalized learning," sans any supporting evidence, has contributed to the near-death of actual teaching--but still has incantatory power. It points up Daniel Kahneman's lament that even the most groundless ideas will acquire "the illusion of validity" if they are endlessly and enthusiastically repeated.
Texas uses I ready- and the state politicians get kick backs
See the state decline in outcomes should be enough. But TX children are commodities and data is gold. Ready for you to come to TX- we are alone out here in the tumbleweed fighting for 6 years to save 10% of the nations children from mind altering decay.
Thanks for statement in Center Sq w/ B Blankley great article!!
As a teacher, I pointed out your exact conclusion about formative assessment needing to inform and drive teacher decisions…and that the inability of teachers to analyze for misconceptions or weaknesses in an area made it impossible to react properly to the data! I left teaching ELA because of the sham…. BUY the modules from the same company that provides the test… but no info from iReady for what a non-module school could do to analyze data… It’s just a box-checking money maker at this point… money schools need for other things!
Thank you for this interesting and informative piece. It would be interesting to examine whether or how I-Ready changes instructional practices by teachers -- more reliance on computer-based lessons, less partner work, etc. No doubt difficult to obtain such data, given that it seems no one has bothered to examine the direct effects of I-Ready!
Hiya Jared. I was hoping someone with your credentials and expertise would dig into the i-Ready data and studies (or lack thereof). You can see my own recent i-Ready deep dive here: https://johnallenwooden.substack.com/p/i-ready-edtech-snake-oil-by-curriculum-associates
Two points therein relevant to your piece here:
1. Re: Johns Hopkins Commissioned Marketing Studies: There have actually been at least five of these for i-Ready, not just two. The Hopkins CRRE group is prolific in producing such commissioned studies for many ed-tech publishers. You're right that the Hopkins name carries weight, and the degree to which they trade on it to produce "studies" in "partnership" with their paying clients represents a serious scandal in the making.
2. The role that i-Ready's own painstakingly search-optimized "Research Library" (https://www.curriculumassociates.com/research-and-efficacy/research-library) plays in this data efficacy ecosystem (and hoodwinking school administrators) cannot be overstated. These officiously packaged documents rank at the top of most Google searches for information about the platform's efficacy. Most are products of Curriculum Associates' own in-house skunk works dedicated to science-washing i-Ready on a state-by-state level, with their own credentialed employees churning out officious "research" and "studies" which all declare i-Ready the cat's meow. Shocker!
The fox in charge of the henhouse...
Key understanding here: formative assessment is only helpful when teachers can use it to inFORM instruction. These tools are notorious for flooding teachers with unusable data. I think some administrators make the mistake of assuming that the data/insights will be easy to visualize--and that their teachers know how to use them. They don't. I don't even think most principals have a system for using this data.
Yes. This is such a great point. Data is only useful when it is digestible and actionable. Even trained researchers can get overwhelmed by too much data.
I think these tools also send a signal that their data should take precedence over a teachers own intuition. An experienced and well trained teacher might not be able to fully explain all of what they know about a student, but they will have a million nuanced data points and comparisons that help them guide a student effectively. We don't even know what we don't know, how can we program that full understanding of a good teacher into a digital tool?
Wow, that is such a great point. I’ve never thought of it that way. It’s like… How can we expect something that has been programmed by humans to be as smart as humans, when humans are constantly growing their understanding of students and their needs?
This is exactly what I mean when I say teachers deserve better. 13 million students. Zero meaningful evidence. But teachers are still being handed these tools and told to use them — while their professional judgment gets completely sidelined. This is why we built TDI: so educators have a space to cut through the noise and get PD that actually works. Glad someone is asking the hard questions. Keep going.
Thank you for breaking down the data (or lack there of), Jared! I have sat next to a number of students using the program and witnessed why a response was incorrect, impacted by so many factors that the program does not pick up (because it is not a human) much to the frustration of the student. So many students get frustrated and end up guessing in order to get through the assignment, so essentially data provided by the program does not accurately reflect their true abilities. There are countless other edtech programs being used that have all the same issues.
@jared What do we do about state requirements? Just got home from a school district tech task force meeting and they said Oregon is now going to require K-2 to take assessments on a device starting this fall. Previously they used an assessment called Dibels that was an in-person assessment and apparently the state does not have any device-free assessments on their "approved list" starting this fall. WHAT?! Are you aware of this? We need to contact our state legislator and the governor. Ridiculous. I was ready to help get the iPads out of K-2 and now the word is we have to prepare the kids to take assessments on devices. I can't believe it. That's backward from where we need to go...Would love your thoughts!!
Yes, Oregon just passed new legislation that all k-8 students in public schools will be tested 3 times per year with one of the approved online tests: MAP, i-ready, Star, or SBAC interim. Schools have no choice in the matter.
This is so frustrating--schools will now feel forced to cater to a curriculum that pushes online learning just so the kids can pass a test 3x per year. UGH.
DIBELS is an excellent benchmark screener for reading. As a reading specialist, I administered it 3 times a year school-wide K-5, and it provided such important insight for our MTSS process, with very little time taken from the students. What a shame that it is not on the approved list.
Just for clarification purposes, I believe you are referring to some kind of benchmark test taken 3 times a year. It’s frequently called “I-ready” or “I-ready diagnostic”. I believe the article is referencing both that AND the daily lessons students are expected to do on their devices, spending an hour and a half WEEKLY on math and reading lessons. This “iready” program is also called “my path”. In my state we’re required to give all students “universal reading screeners” three times a year do identify characteristics of dyslexia. I-ready diagnostic qualifies for that, but other programs do as well.
Has anyone looked into how districts tie teacher bonuses and monetary incentives to i-ready scores? Feels like an episode of black mirror.
Definitely a risk.
This is the pattern. Every tool that could inform teacher judgment gets converted into a replacement for it. The mechanism you're describing — formative assessment works because it changes what the teacher does, and i-Ready removes the teacher from that loop — is the same thing that happened to the LMS, the rubric, and the standardized test. The system doesn't adopt tools to improve learning. It adopts tools to remove judgment from the process, because judgment is expensive, unscalable, and impossible to audit. $535,000 for a 0.8% difference in math with zero impact on reading is not a failure of the tool. It's the tool performing exactly as the system needs it to — generating data, demonstrating compliance, and leaving learning to fend for itself.
This thread has been a masterclass in EdTech skepticism. John Allen Wooden's documentation of the CRRE/JHU relationship is particularly damning - five commissioned studies, all vendor-funded, all bearing Johns Hopkins' imprimatur. That's not just weak evidence, that's the academic equivalent of a company writing its own press release.
Molly, Erica, Manju - thanks for reading. The pattern here is so familiar: we've normalized "evidence" that's really just marketing dressed up in peer review clothing. What's your experience been? Are you seeing districts break free from these tools once they've done the audit? Or is the momentum so strong that escape is harder than it should be?
The bigger question: if we applied clinical significance standards (vs just statistical significance) to EdTech adoption, what would survive? The $535K for 0.8% math gains doesn't pass. But neither does any other adaptive tool I've reviewed that lacks independent, third-party RCTs.
I’m all about student growth and strategies to help teachers help students grow. I-ready provides an extensive report that breakdowns student strengths and areas of growth which are aligned with our state standards. This is something that can be given to parents and teachers can use this information to tailor lessons and/or create small groups to a. See if the student is truly struggling b. Provide targeted support for students because it identifies strengths and areas of growth. It’s a tool that can be used to support enrichment and remediation. This is how we used iReady as a formative assessment tool. Teachers are able to see the questions that students were asked (unless this has changed)
I’m only speaking from my experience with using iReady as a tool.
I appreciate this article because it does highlight the over reliance on programs and the drive for profit with Ed tech. I cannot speak to national statistics or data, but I can tell you from experience that iReady is a tool that can improve student performance if the data is used with intention along with systems in place that ensure consistency and accountability. IReady is a useful tool for teachers, students, parents, and school leaders. When schools use IReady just to put kids on a program, they are not utilizing all that IReady has to offer to support student growth and inform instruction. I’m not an iReady rep,
I’m just speaking from experience when I used it as a teacher, instructional coach, and district coordinator.
We use iready and the data reports never match what we see in the classroom. Also, the test is used as a bludgeon for districts to purchase their curriculum. It's a giant circular fire hose of money. It also takes SO MUCH TIME away from actual instruction. I could go on for hours how awful it is.
Everyone is entitled to their opinion. I can only speak about my experience. I’m sorry IReady isn’t as effective in your school/district.
What does that word salad even mean? Sounds like someone gets kickback.
I think your tone needs to kick back a bit. She is entitled to her opinion, whether it disagrees with yours or not. Implying that she is bought and paid for just because she had a better experience is insulting.