Right now your "digital inflection" data across all states looks like it predominantly reflects adoption of computerized use of tech by students in the classroom. Do you have more data than just this? What exactly is the correlation you are purporting, and where is the dataset for those correlations?
And yes - this is tech infrastructure, so student-facing. I have not done any analyses exploring teacher-facing tech or administrative utilization. I am primarily interested on the impact of student-use on learning.
Ha - This is exactly the tired data science whataboutism Horvath is calling out! He could (and can, and does) roll out a veritable mountain of data proving ed-tech is brain-rotting snake oil, but the latter day tobacco execs will always insist it's either insufficient or missing some other bit of data minutiae -- in a cynical, totally self-serving attempt to undermine what they know is a slam-dunk argument. #FAIL
“Dr. Tilak’s research focuses on the creation of a VR assisted online program”. — Quoth another EdTech shill devoted to strapping screens to children’s faces. Your bread is buttered in EdTech dollars, so forgive us if we don’t believe a word you say.
I have not read the data for every state, but the ones that I did read (OH, ND, MA, RI, OR) noted that the inflection points correspond to the time that online TESTING was implemented. NOT when online INSTRUCTION was implemented, or became widespread. These data don't support an assertion that digital instruction is failing our kids. They do support an assertion that digital TESTING shows some performance differences. I think we need to continue to dig.
These correlations are directly contradicted by experimental studies showing positive effects of technological applications on student performance in various subject areas such as literacy, mathematics, and science. See:
I’d argue that this data trumps “experimental studies” on the simple basis that there are real numbers and real students being studied here. Sometimes experimental data does not hold up in practice, which this appears to demonstrate. That first study appears to be calling for the introduction of the Internet into the classrooms of Indian schools. American schools are far beyond that.
Florida teacher here. Can you speak more to the Florida/Texas outermost exclusion points? Were these states that were the first and last to implement NAEP testing? Thanks!
I am thoroughly enjoying reading these articles as a sober second look in the face of so much EdTech hype, which I am constantly drowning in since I work in a digital learning role. My question is about ascribing the issues to the tools/EdTech as a concept, rather than the methodology or employment of those tools? What are these studies measuring? Virtual delivery that completely replaced traditional western methods of teaching to save money on teachers? Use of a platform or tool focused on the bottom line? Or, is this selecting research and evidence looking into meaningful, intentional, pedagogically-sound integrations? My personal feeling is when EdTech approaches are undertaken for economic or political reasons rather than pedagogical ones (which, let's be honest, is probably 80% of the time if not more), or out of a fear of falling behind, then, yeah. You're going to see these results.
And that is correlation that indicates a causation of sorts. Throwing a tool at something, just because you can or because it will save you money, rarely improves knowledge, but it is a common occurrence in a capitalist society.
Just my thoughts to chew on, and perhaps to discuss in a future post?
I looked at the NAEP data for Maine, which was the first state to introduce laptops to the classroom, starting in 2002 (Maine Learning Technology Initiative). Maine gives data on how many laptops were introduced to their classroom through this program. I found an article that said all 7th and 8th graders had a laptop by 2016. This was before the pandemic, and Maine should have been relatively similar to the rest of the country as far as smartphones and tablets outside of school. Before 2002, Maine outperformed the national average by 9.25 points across Math and ELA. In 2017, they only outperformed the nation by 1.75 points, a relative drop of 7.5 points. For reference, in the last year of results, the best state in the country was an average of 10 points better than the national average. Their ELA numbers fell in absolute terms, not just relative to the rising national scores. Maine's governor at the time called it a "massive failure" and said "It's been a failure and we all know it, but we keep doing it because we're used to doing it."
I think your data analysis is brilliant. I can only give my personal experience as a student and now as a parent. When I was a student we all had textbooks. We had a computer class. We had typing lessons. Yet I always took paper tests that reflected a cumulative review of what I learned and sometimes didn't learn in the course of the year. I'll be honest schools took credit for gifted kids like myself who knew how to read thanks to phonics and had critical thinking skills to answer the test without always knowing the information. I saw this shift to computer testing when my state stopped using the IOWA CRCT tests and created their own. This abruptly forced young kids to shift from handwriting to typing. It was terrible but they kept moving on. Today we assume all kids have technology at home. They don't. Then schools who rent out technology often put families into debt when a child breaks that technology and parents are on the hook. It's not the same as losing a $60 textbook. It's a $200 Chromebook today. My question is that at what point do we stop the freight train? Kids today don't know how to effectively type, code, or do anything other than stare and tap at a screen. Why are we allowing screens to have priority over an actual human that we have eons of history showing humans learn best from humans? It doesn't make sense.
Please include DoDEA as they are all in on EdTech and "21st century classroom" learning. Kids are given a chromebook in 3rd grade and almost everything is online.
Thank you for your work on this front. To anyone not trapped in a research rut of obstinacy, the reality of what you are showing is plain as day. Educators know it, parents know it.
(PS - having perused your scholar account, I notice your highest cited work is neither applied nor field work).
Shantanu - you seem to be mistaking short data dumps on Substack with broader work. I'd like to assume you exist beyond the short comments you are leaving here. I do as well, and so does my understanding and my work.
If you dislike me and my work, fine: but it's worth first understanding my work and research.
Brilliant - my applied work is on learning and pedagogy. It appears we can both be doing field work but focused on different things (that speak to the same outcome).
I love that you run a lab school: one of the most important contexts to exist out there.
Keep up the awesome work and the good thinking. And I agree - a lot can be done to demystify the arguments...which is why I wrote a 100,000 word book. You may think it's less rigorous than research (and you may be right), but it allows for the space and room to make more sense of things than short posts and podcasts do.
I hope you write a book someday and collate all you've learned - I would love to read it!
Right now your "digital inflection" data across all states looks like it predominantly reflects adoption of computerized use of tech by students in the classroom. Do you have more data than just this? What exactly is the correlation you are purporting, and where is the dataset for those correlations?
There's a link to all data in the article (along with, what I've been told, is a 'far too long' table explaining all inflection points). https://www.nationsreportcard.gov/ndecore/landing
And yes - this is tech infrastructure, so student-facing. I have not done any analyses exploring teacher-facing tech or administrative utilization. I am primarily interested on the impact of student-use on learning.
Ha - This is exactly the tired data science whataboutism Horvath is calling out! He could (and can, and does) roll out a veritable mountain of data proving ed-tech is brain-rotting snake oil, but the latter day tobacco execs will always insist it's either insufficient or missing some other bit of data minutiae -- in a cynical, totally self-serving attempt to undermine what they know is a slam-dunk argument. #FAIL
So simply asking for the actual data is whataboutism?
“Dr. Tilak’s research focuses on the creation of a VR assisted online program”. — Quoth another EdTech shill devoted to strapping screens to children’s faces. Your bread is buttered in EdTech dollars, so forgive us if we don’t believe a word you say.
I have not read the data for every state, but the ones that I did read (OH, ND, MA, RI, OR) noted that the inflection points correspond to the time that online TESTING was implemented. NOT when online INSTRUCTION was implemented, or became widespread. These data don't support an assertion that digital instruction is failing our kids. They do support an assertion that digital TESTING shows some performance differences. I think we need to continue to dig.
These correlations are directly contradicted by experimental studies showing positive effects of technological applications on student performance in various subject areas such as literacy, mathematics, and science. See:
https://www.sciencedirect.com/science/article/abs/pii/S0360131516302172
https://www.sciencedirect.com/science/article/abs/pii/S1747938X12000401
https://www.sciencedirect.com/science/article/abs/pii/S1747938X13000031
https://www.sciencedirect.com/science/article/pii/S0360131515300804
https://journals.sagepub.com/doi/abs/10.3102/00346543241261073?ai=1gvoi&mi=3ricys&af=R
And Hattie's Visible Learning: https://www.amazon.com/Visible-Learning-Sequel-John-Hattie/dp/1032462035
In your next post, can you please explain why all of these researchers are wrong?
I’d argue that this data trumps “experimental studies” on the simple basis that there are real numbers and real students being studied here. Sometimes experimental data does not hold up in practice, which this appears to demonstrate. That first study appears to be calling for the introduction of the Internet into the classrooms of Indian schools. American schools are far beyond that.
Mr. Horvath, have you studied student performance in schools where technology has NOT been widely adopted? Looking forward to reading your book.
Florida teacher here. Can you speak more to the Florida/Texas outermost exclusion points? Were these states that were the first and last to implement NAEP testing? Thanks!
I am thoroughly enjoying reading these articles as a sober second look in the face of so much EdTech hype, which I am constantly drowning in since I work in a digital learning role. My question is about ascribing the issues to the tools/EdTech as a concept, rather than the methodology or employment of those tools? What are these studies measuring? Virtual delivery that completely replaced traditional western methods of teaching to save money on teachers? Use of a platform or tool focused on the bottom line? Or, is this selecting research and evidence looking into meaningful, intentional, pedagogically-sound integrations? My personal feeling is when EdTech approaches are undertaken for economic or political reasons rather than pedagogical ones (which, let's be honest, is probably 80% of the time if not more), or out of a fear of falling behind, then, yeah. You're going to see these results.
And that is correlation that indicates a causation of sorts. Throwing a tool at something, just because you can or because it will save you money, rarely improves knowledge, but it is a common occurrence in a capitalist society.
Just my thoughts to chew on, and perhaps to discuss in a future post?
I looked at the NAEP data for Maine, which was the first state to introduce laptops to the classroom, starting in 2002 (Maine Learning Technology Initiative). Maine gives data on how many laptops were introduced to their classroom through this program. I found an article that said all 7th and 8th graders had a laptop by 2016. This was before the pandemic, and Maine should have been relatively similar to the rest of the country as far as smartphones and tablets outside of school. Before 2002, Maine outperformed the national average by 9.25 points across Math and ELA. In 2017, they only outperformed the nation by 1.75 points, a relative drop of 7.5 points. For reference, in the last year of results, the best state in the country was an average of 10 points better than the national average. Their ELA numbers fell in absolute terms, not just relative to the rising national scores. Maine's governor at the time called it a "massive failure" and said "It's been a failure and we all know it, but we keep doing it because we're used to doing it."
https://commons.trincoll.edu/edreform/2017/05/from-lunchboxes-to-laptops-maines-laptop-initiative/
I think your data analysis is brilliant. I can only give my personal experience as a student and now as a parent. When I was a student we all had textbooks. We had a computer class. We had typing lessons. Yet I always took paper tests that reflected a cumulative review of what I learned and sometimes didn't learn in the course of the year. I'll be honest schools took credit for gifted kids like myself who knew how to read thanks to phonics and had critical thinking skills to answer the test without always knowing the information. I saw this shift to computer testing when my state stopped using the IOWA CRCT tests and created their own. This abruptly forced young kids to shift from handwriting to typing. It was terrible but they kept moving on. Today we assume all kids have technology at home. They don't. Then schools who rent out technology often put families into debt when a child breaks that technology and parents are on the hook. It's not the same as losing a $60 textbook. It's a $200 Chromebook today. My question is that at what point do we stop the freight train? Kids today don't know how to effectively type, code, or do anything other than stare and tap at a screen. Why are we allowing screens to have priority over an actual human that we have eons of history showing humans learn best from humans? It doesn't make sense.
There is an elephant in the living room: how is this relevant when at least one of the tests, Smarter Balanced has no external validation?
Please include DoDEA as they are all in on EdTech and "21st century classroom" learning. Kids are given a chromebook in 3rd grade and almost everything is online.
Thank you for your work on this front. To anyone not trapped in a research rut of obstinacy, the reality of what you are showing is plain as day. Educators know it, parents know it.
Dig deeper? You mean like writing a book? :-) Remember - a 900 word post is different than a 20 year career.
You mean like the applied work I have been doing since 2011?
https://scholar.google.com/citations?user=iOqYqV4AAAAJ&hl=en
(PS - having perused your scholar account, I notice your highest cited work is neither applied nor field work).
Shantanu - you seem to be mistaking short data dumps on Substack with broader work. I'd like to assume you exist beyond the short comments you are leaving here. I do as well, and so does my understanding and my work.
If you dislike me and my work, fine: but it's worth first understanding my work and research.
Brilliant - my applied work is on learning and pedagogy. It appears we can both be doing field work but focused on different things (that speak to the same outcome).
I love that you run a lab school: one of the most important contexts to exist out there.
Keep up the awesome work and the good thinking. And I agree - a lot can be done to demystify the arguments...which is why I wrote a 100,000 word book. You may think it's less rigorous than research (and you may be right), but it allows for the space and room to make more sense of things than short posts and podcasts do.
I hope you write a book someday and collate all you've learned - I would love to read it!