Illinois researchers put AI to the test in a recent study
-
Findings from a recent study found that students who use ChatGPT as their only study tool can still pass a class with a B grade.
-
ChatGPT excelled at structured math questions, but struggled with openended analysis.
-
This study exposes course design implications as AI becomes part of education.
At the University of Illinois at UrbanaChampaign, researchers in aerospace engineering asked a simple but provocative question: Can you pass a demanding controlsystems course using only ChatGPT, with minimal effort?
They put a free version of ChatGPT to the test by feeding it exactly the same assignments students got in AE353: Aerospace Control Systems, covering math problems, code projects, and analytical writing.
The result: ChatGPT earned a B (82.24%) close to the class average of around 84.99%, but with very little actual learning.
We found ChatGPT technology can get an A on structured, straightforward questions, Ph.D. student Gokul Puthumanaillam said in a news release.
On open-ended questions it got a 62% bringing ChatGPTs semester grade down to an 82%, a low B. The class average for the human students was 84.85% because they could handle the problems that required higher-level reasoning.
How did they test it?
The researchers ran ChatGPT through all 115 homework sets, exams, and programming assignments from the semester-long course, simulating a realistic usage style just the bare minimum effort a timepressured student might use.
The model tackled structured, autograded multiplechoice and math problems, wrote Python code, and wrote longer analytical answers all using only the free ChatGPT interface and no additional instructor guidance or corrections beyond those prompts. They recorded scores across different task types to understand where ChatGPT performed well and where it fell short.
Results: What did they find?
On routine, structured math questions, ChatGPT delivered near-perfect answers earning top marks with minimal prompting. But as questions grew more openended and analytical especially longer essays or designoriented tasks its performance dropped sharply, sometimes earning Ds.
Despite the fact that we provided all of the course material needed to ChatGPT, it still hallucinated, using words that were never used in the class, in the lectures, or course materials, said Puthumanaillam.
Overall, the model earned 82.24%, a solid B, though slightly below the human student average of about 84.99%.
That means, in practical terms, a lazy student could rely on ChatGPT, breeze through math assignments, get a passing grade, and still barely understand the course.
Why it matters
This study isnt just a curiosity it raises important questions for educators.
Like calculators in math classes, ChatGPT is a tool thats here to stay and that students will use, researcher Melkior Ornik said in the news release. What the results of this study pointed out to me is that I need to adjust as an educator.
I plan to consider how I design my courses so that, over time, I include more higher-level questions, perhaps including project-based assignments. Students will still use programs like ChatGPT to do the simpler math problems, but by adding more open-ended questions, theyll also reach a higher level of critical thinking and truly learn the material.
Posted: 2025-08-04 13:55:09
