According to a new study by researchers at Carnegie Mellon, MIT, Oxford, and UCLA,
Study should be solid I guess.
participants who were given AI assistants (in this case, a chatbot powered by OpenAI’s GPT-5 model) would have the aid pulled from them without warning during the test
Wow, interesting idea. 👍
where they had their assistant removed, the AI group saw the solve rate fall off a cliff. They had a solve rate about 20% lower
And even worse IMO:
They also had nearly double the skip rate, meaning they simply chose not to solve the questions.
This seems very alarming IMO, because this indicates they lost some of their ability to think constructively on how to actually solve a problem!
I know there have always been some who cried wold every time new technology has become available, like calculators and computers. Even dictionaries were once claimed to be harmful once!
But maybe this time there is a real danger, because AI takes away a lot of the need to actually think creatively and constructively. And that’s an ability we must not lose.
The last paragraph of the article is even worse. As it mentions 2 studies that show these effects are also long term!!!
When driving somewhere, if I set out with the mindset that I can’t rely on gps I can usually wing it and figure out where to go when a hiccup occurs. If I don’t, then I have a lot of trouble getting into that path finding mode when needed… similar to this maybe?
Yeah exactly, because although it’s possible to do more with technology sometimes, you’re actively de-skilling at the same time. When we invented the written word yes it legitimately made everything better, but also we lost oral traditions and the capacity to memorize large volumes of storytelling, songs, and histories. Now you can burn the books, and the knowledge dies. It’s a real risk.
Everything is like this. Every technology has a cost beyond its price, and making a decision of whether to use it or not will always be in error unless you think about what you’re losing in the process.
If I use AI for my personal coding projects I’ve found that if the task is unsolvable by the ai model, I’m not able to sit down and do it myself until the next day. It’s like I’ve got to reset my brain.
If I want to save time and use AI for a specific part of the code, it probably saves me 5 hours of work. But then I spend five hours yelling at the ai to try to get it to actually solve it. Next day I’ll just fix it myself in 2 hours.
Changing the terms of the test in the middle of it, without warning, is disruptive. I’m not convinced it “fried their brains.” The same would happen with a calculator suddenly removed during the middle of an exam.
Or any task change really. You tell me that I’m here for a writing task, then halfway through it becomes a math test? There’s no way I’m doing anywhere near as well as if they told me what was happening ahead of time.
there have always been some who cried wold every time new technology has become available, like calculators and computers
and they kinda have a point, really. people got worse at memorizing stuff by heart when writing was invented, and people got worse at mental calculus when calculators when invented.
but they allowed many things that were simply not possible. a calculation that takes me 2 minutes in wolfram alpha could take hours if not days to solve by hand!
ai, meanwhile, or at least the ai we’re sold, does not offer significant advantages (at best it saves a few minutes), at the cost of making us worse at thinking, a skill that is absolutely essential to have… and of course, that’s the point. the tech oligarchs want us to be dependent on their extremely expensive products.
No the test is not training, that’s a weird thing to claim. The switch is what is tested, and you disregard that 2 other tests have shown similar results. An actual decline in critical and problem solving thinking.
No the test is not training, that’s a weird thing to claim.
The control group solved 12 questions manually and then the 3 test questions manually. The AI grouped solved 0 questions manually and the 3 test questions manually. One group had 12 more manual math tasks to prepare for the manual math test the other group had 0 and also had to context switch.
The AI-assisted group was dealt a context switch, which results in a pretty severe performance loss. A context switch causes performance loss of around 40% according to this paper, which was peer-reviewed and published and is also the most cited paper on the topic, in the APA: https://www.apa.org/pubs/journals/releases/xhp274763.pdf
The AI-assisted group also did not have 12 questions to adjust to the new context, like the control group did. If they wanted to wipe out the context switching performance loss they should have kept asking questions to see if, after 12 questions, the AI-assisted group had a similar performance.
The switch is what is tested, and you disregard that 2 other tests have shown similar results.
No, they did not switch what was tested. Here is an image from the actual paper.
They were given 12 tasks with one group using AI and another doing mental math and then 3 tasks doing mental math. One group had 12 more tasks worth of preparation than the other.
Nothing, not even the article in theOP, says that they did math and swapped to reading to test.
They did 3 different experiments, in each experiment they gave 12 tasks and then disabled the AI for one group and gave 3 more tasks as a test. At no point did they ask 12 math questions and then finish with 3 reading questions or vice versa. They did 2 experiments using math tasks and 1 experiment using reading comprehension tasks.
So one group had 15 math tasks and one group had 12 ‘how to ask an AI’ tasks and then 3 math questions.
They also did not control for context switching losses, which is a well documented (see the APA paper) effect. The proper control would be to continue asking questions so the AI group also had 12 math tasks before the test.
There’s a reason that this is published on arXiv and not in a peer-reviewed journal. Designing a poor quality experiment doesn’t tell you anything useful even if you do multiple different versions of the same experiment.
This paper demonstrates a lack of a proper control group, specifically a failure to control for context switching performance loss.
The picture you post contradict your claims. The 2 groups are getting the same question, but one has AI assistance, the other has not.
Again you fail to show anything to support your claims.
Study should be solid I guess.
Wow, interesting idea. 👍
And even worse IMO:
This seems very alarming IMO, because this indicates they lost some of their ability to think constructively on how to actually solve a problem!
I know there have always been some who cried wold every time new technology has become available, like calculators and computers. Even dictionaries were once claimed to be harmful once!
But maybe this time there is a real danger, because AI takes away a lot of the need to actually think creatively and constructively. And that’s an ability we must not lose.
The last paragraph of the article is even worse. As it mentions 2 studies that show these effects are also long term!!!
When driving somewhere, if I set out with the mindset that I can’t rely on gps I can usually wing it and figure out where to go when a hiccup occurs. If I don’t, then I have a lot of trouble getting into that path finding mode when needed… similar to this maybe?
Yeah exactly, because although it’s possible to do more with technology sometimes, you’re actively de-skilling at the same time. When we invented the written word yes it legitimately made everything better, but also we lost oral traditions and the capacity to memorize large volumes of storytelling, songs, and histories. Now you can burn the books, and the knowledge dies. It’s a real risk.
Everything is like this. Every technology has a cost beyond its price, and making a decision of whether to use it or not will always be in error unless you think about what you’re losing in the process.
If I use AI for my personal coding projects I’ve found that if the task is unsolvable by the ai model, I’m not able to sit down and do it myself until the next day. It’s like I’ve got to reset my brain.
If I want to save time and use AI for a specific part of the code, it probably saves me 5 hours of work. But then I spend five hours yelling at the ai to try to get it to actually solve it. Next day I’ll just fix it myself in 2 hours.
Changing the terms of the test in the middle of it, without warning, is disruptive. I’m not convinced it “fried their brains.” The same would happen with a calculator suddenly removed during the middle of an exam.
Or any task change really. You tell me that I’m here for a writing task, then halfway through it becomes a math test? There’s no way I’m doing anywhere near as well as if they told me what was happening ahead of time.
Also and this is the big one for me. It’s 10% wrong on average. That’s really bad. 1 in 10 google Gemini answers is bullshit
and they kinda have a point, really. people got worse at memorizing stuff by heart when writing was invented, and people got worse at mental calculus when calculators when invented.
but they allowed many things that were simply not possible. a calculation that takes me 2 minutes in wolfram alpha could take hours if not days to solve by hand!
ai, meanwhile, or at least the ai we’re sold, does not offer significant advantages (at best it saves a few minutes), at the cost of making us worse at thinking, a skill that is absolutely essential to have… and of course, that’s the point. the tech oligarchs want us to be dependent on their extremely expensive products.
But they’re using the hell out of it, too, right? They’re exactly the types of people that love and use it the most: managers and owners.
This paper shows that a person who has performed a task 12 times performs better than a person who has never performed the same task.
They also do not properly control for performance loss due to context switching which is a well known contributor to performance loss.
It’s a paper on arXiv, it hasn’t been peer reviewed or published.
No the test is not training, that’s a weird thing to claim. The switch is what is tested, and you disregard that 2 other tests have shown similar results. An actual decline in critical and problem solving thinking.
Here is the paper: https://ai-project-website.github.io/AI-assistance-reduces-persistence/
The control group solved 12 questions manually and then the 3 test questions manually. The AI grouped solved 0 questions manually and the 3 test questions manually. One group had 12 more manual math tasks to prepare for the manual math test the other group had 0 and also had to context switch.
The AI-assisted group was dealt a context switch, which results in a pretty severe performance loss. A context switch causes performance loss of around 40% according to this paper, which was peer-reviewed and published and is also the most cited paper on the topic, in the APA: https://www.apa.org/pubs/journals/releases/xhp274763.pdf
The AI-assisted group also did not have 12 questions to adjust to the new context, like the control group did. If they wanted to wipe out the context switching performance loss they should have kept asking questions to see if, after 12 questions, the AI-assisted group had a similar performance.
No, they did not switch what was tested. Here is an image from the actual paper.
They were given 12 tasks with one group using AI and another doing mental math and then 3 tasks doing mental math. One group had 12 more tasks worth of preparation than the other.
Nothing, not even the article in theOP, says that they did math and swapped to reading to test.
They did 3 different experiments, in each experiment they gave 12 tasks and then disabled the AI for one group and gave 3 more tasks as a test. At no point did they ask 12 math questions and then finish with 3 reading questions or vice versa. They did 2 experiments using math tasks and 1 experiment using reading comprehension tasks.
So one group had 15 math tasks and one group had 12 ‘how to ask an AI’ tasks and then 3 math questions.
They also did not control for context switching losses, which is a well documented (see the APA paper) effect. The proper control would be to continue asking questions so the AI group also had 12 math tasks before the test.
There’s a reason that this is published on arXiv and not in a peer-reviewed journal. Designing a poor quality experiment doesn’t tell you anything useful even if you do multiple different versions of the same experiment.
This paper demonstrates a lack of a proper control group, specifically a failure to control for context switching performance loss.
The picture you post contradict your claims. The 2 groups are getting the same question, but one has AI assistance, the other has not.
Again you fail to show anything to support your claims.
I also wrote text.
If you’re just going to cherry pick a single point and dismiss everything else then we’re done here.
Maybe they’re unable to switch contexts
Wow. Now do this with a calculator.