AI in Education: Insights from a GPT-4 High School Experiment

Generative AI like GPT-4 is shaking up education, but are we really ready for the changes?

I came across a fascinating study titled "Generative AI Can Harm Learning " led by Hamsa Bastani and colleagues. This research explored how AI, specifically GPT-4, impacts learning in high school math classes. Here’s a breakdown of what they found:

Study Setup and Structure

The researchers conducted a field experiment in a Turkish high school with nearly 1,000 students from grades 9 to 11. They tested two AI tutors:

• GPT Base - Mimicking ChatGPT
• GPT Tutor - Equipped with additional safeguards.

These AI tutors covered about 15% of the math curriculum over four 90-minute sessions. Each session had three parts: a review lecture, an assisted practice period (with AI help), and an unassisted exam.

The goal was to measure student performance in both assisted and unassisted settings.

Results and Analysis

Boosted Immediate Performance: GPT-4 improved student performance by 48% with GPT Base and a whopping 127% with GPT Tutor during practice sessions.

Long-term Impact of GPT Base: When the AI was taken away, students using GPT Base scored 17% worse than those who never had it. It seems they relied too much on the AI.

Long-term Impact of GPT Tutor: The GPT Tutor, with its extra learning safeguards, helped maintain performance levels.

Student Perception: Interestingly, students felt they performed better and valued practice sessions more with the GPT Tutor, even though there weren't significant performance gains.

Conclusion

Immediate Gains: Both GPT Base and GPT Tutor significantly boosted immediate performance in practice sessions compared to traditional methods, with GPT Tutor showing the highest improvement.

Long-term Learning: Traditional methods and GPT Tutor maintained consistent performance in exams, while GPT Base led to a decline, indicating overreliance and insufficient skill retention.

Student Engagement: GPT Tutor was perceived positively by students, enhancing their engagement and perceived value of learning, unlike GPT Base, which led to false confidence.

Generative AI can be a game-changer, but its design and implementation are crucial. GPT Tutor, with its structured prompts and incremental hints, supports both immediate and long-term learning. It's a more effective tool compared to the standard GPT Base and traditional methods. While we should embrace tech in education, we must do so wisely to ensure students continue to build critical skills.

Let’s embrace tech! We still have much to learn.

Supported by UX Design Lab, LLC

Originally published: https://uxdesignlab.com/ai-in-education-insights-from-a-gpt-4-high-school-experiment/