- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Performance of ChatGPT in solving wave problems

Muhammad Aswin Rangkuti1,2,*, Wawan Bunawan1,2, Hendra Y. Agustian3, Tuti Hardianti1,2, Dewi Wulandari1, and Nazwa Hudaiby1

  • *Contact author: aswin_ray@unimed.ac.id
PDF

Phys. Rev. Phys. Educ. Res. 22, 020113 – Published 6 August, 2026

DOI: https://doi.org/10.1103/4dgy-1dg1

Abstract

The ability of ChatGPT, especially its latest versions, attracts significant attention from educators and researchers in fields such as physics. Several studies on this topic explore the performance of ChatGPT in answering physics questions across different versions of ChatGPT. In this study, we evaluate the development of ChatGPT in answering physics questions using the Mechanical Waves Conceptual Survey (MWCS) to examine whether its capabilities improve across different versions. A total of 75 responses per item are generated using ChatGPT versions 4, 4.o, and 5.2, resulting in 4950 responses for comprehensive analysis. Based on the comparison between the three versions of ChatGPT, we find that some difficult types of questions that are answered incorrectly by previous versions, as reported in earlier studies, can now be answered correctly, although there are still minor mistakes, especially when examining the reasoning. A generalized linear mixed model (GLMM) analysis, together with post hoc comparisons, indicates that ChatGPT 5.2 significantly outperforms both ChatGPT 4 and 4.o, suggesting that the capabilities of the latest version included in this study significantly improve. These findings highlight a future direction for this type of research, as our study shows that the latest version of ChatGPT can correctly answer more types of questions in MWCS.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (58)

  1. E. A. Alasadi and C. R. Baiz, Generative AI in education and research: Opportunities, concerns, and solutions, J. Chem. Educ. 100, 2965 (2023).
  2. Y.-J. Lan and N.-S. Chen, Teachers’ agency in the era of LLM and generative AI, Educ. Technol. Soc. 27, I (2024).
  3. H. Lin and Q. Chen, Artificial intelligence (AI)-integrated educational applications and college students’ creativity and academic emotions: Students and teachers’ perceptions and attitudes, BMC Psychol. 12, 487 (2024).
  4. S. Habib, T. Vogel, X. Anli, and E. Thorne, How does generative artificial intelligence impact student creativity?, J. Creativity 34, 100072 (2024).
  5. M. Giannakos, R. Azevedo, P. Brusilovsky, M. Cukurova, Y. Dimitriou, D. Hernandez-Leo, S. Järvelä, M. Mavrikis, and B. Rienties, The promise and challenges of generative AI in education, Behaviour Inf. Technol. 1 (2024).
  6. C. K. Y. Chan, A comprehensive AI policy education framework for university teaching and learning, Int. J. Educ. Technol. Higher Educ. 20, 38 (2023).
  7. A. Guilherme, AI and education: The importance of teacher and student relations, AI & Society 34, 47 (2019).
  8. L. Chen, P. Chen, and Z. Lin, Artificial intelligence in education: A review, IEEE Access 8, 75264 (2020).
  9. G. Polveriniand B. Gregorcic, Performance of ChatGPT on the test of understanding graphs in kinematics, Phys. Rev. Phys. Educ. Res. 20, 010109 (2024).
  10. F. Kieser, P. Wulff, J. Kuhn, and S. Küchemann, Educational data augmentation in physics education research using ChatGPT, Phys. Rev. Phys. Educ. Res. 19, 020150 (2023).
  11. D. Tong, Y. Tao, K. Zhang, X. Dong, Y. Hu, S. Pan, and Q. Liu, Investigating ChatGPT-4’s performance in solving physics problems and its potential implications for education, Asia Pac. Educ. Rev. 25, 1379 (2023).
  12. V. López-Simó and M. F. Rezende, Jr., Challenging ChatGPT with different types of physics education questions, Phys. Teach. 62, 290 (2024).
  13. S. Aldazharova, G. Issayeva, S. Maxutov, and N. Balta, Assessing AI’s problem solving in physics: Analyzing reasoning, false positives and negatives through the force concept inventory, Contemp. Educ. Technol. 16, ep538 (2024).
  14. A. Tongchai, M. D. Sharma, I. D. Johnston, K. Arayathanitkul, and C. Soankwan, Developing, evaluating and demonstrating the use of a conceptual survey in mechanical waves, Int. J. Sci. Educ. 31, 2437 (2009).
  15. P. Barniola and G. Zavala, The mechanical waves conceptual survey: An analysis of university students’ performance, and recommendations for instruction, Eurasia J. Math. Sci. Technol. Educ. 13, 929 (2016).
  16. K. Zhang and A. B. Aslan, AI technologies for education: Recent research and future directions, Comput. Educ. 2, 100025 (2021).
  17. X. Chen, D. Zou, H. Xie, G. Cheng, and C. Liu, Two decades of artificial intelligence in education, Educ. Technol. Soc. 25, 28 (2022).
  18. W. Holmes, K. Porayska-Pomsta, K. Holstein, E. Sutherland, T. Baker, S. B. Shum, O. C. Santos, M. T. Rodrigo, M. Cukurova, and I. I. Bittencourt, Ethics of AI in education: Towards a community-wide framework, Int. J. Artif. Intell. Educ. 32, 504 (2022).
  19. S. Nemorin, A. Vlachidis, H. M. Ayerakwa, and P. Andriotis, AI hyped? A horizon scan of discourse on artificial intelligence in education (AIED) and development, Learn. Media Technol. 48, 38 (2023).
  20. X. Zhai, X. Chu, C. S. Chai, M. S. Y. Jong, A. Istenic, M. Spector, J.-B. Liu, J. Yuan, Y. Li, and N. Cai, A review of artificial intelligence (AI) in education from 2010 to 2020, Complexity 2021, 8812542 (2021).
  21. U. Mittal, S. Sai, and V. Chamola, A comprehensive review on generative AI for education, IEEE Access 12, 142733 (2024).
  22. F. Almasri, Exploring the impact of artificial intelligence in teaching and learning of science: A systematic review of empirical research, Res. Sci. Educ. 54, 977 (2024).
  23. E. A. Alasadi and C. R. Baiz, Multimodal generative artificial intelligence tackles visual problems in chemistry, J. Chem. Educ. 101, 2716 (2024).
  24. G.-G. Lee and X. Zhai, Using ChatGPT for science learning: A study on pre-service teachers’ lesson planning, IEEE Trans. Learn. Technol. 17, 1643 (2024).
  25. A. M. Bettayeb, M. Abu Talib, A. Z. Sobhe Altayasinah, and F. Dakalbab, Exploring the impact of ChatGPT: Conversational AI in education, Front. Educ. 9, 1379796 (2024).
  26. T. Wan and Z. Chen, Exploring generative AI assisted feedback writing for students’ written responses to a physics conceptual question with prompt engineering and few-shot learning, Phys. Rev. Phys. Educ. Res. 20, 010152 (2024).
  27. N. M. Nasri, N. Nasri, N. F. Nasri, and M. A. Abd Talib, The impact of integrating an intelligent personal assistant (IPA) on secondary school physics students’ scientific inquiry skills, IEEE Trans. Learn. Technol. 16, 232 (2023).
  28. W. Holmes and I. Tuomi, State of the art and practice in AI in education, Eur. J. Educ. 57, 542 (2022).
  29. B. Williamson and R. Eynon, Historical Threads, Missing Links, and Future Directions in AI in Education (Taylor & Francis, London, 2020).
  30. L. Krupp, S. Steinert, M. Kiefer-Emmanouilidis, K. E. Avila, P. Lukowicz, J. Kuhn, S. Küchemann, and J. Karolus, Challenges and opportunities of moderating usage of large language models in education, MUM’24: Proceedings of the International Conference on Mobile and Ubiquitous Multimedia (2024), pp. 249–254, 10.1145/3701571.3701590.
  31. X. Wang, L. Li, S. C. Tan, L. Yang, and J. Lei, Preparing for AI-enhanced education: Conceptualizing and empirically examining teachers’ AI readiness, Comput. Hum. Behav. 146, 107798 (2023).
  32. J. Kim, H. Lee, and Y. H. Cho, Learning design to support student-AI collaboration: Perspectives of leading teachers for AI in education, Educ. Inf. Technol. 27, 6069 (2022).
  33. S. Schorcht, N. Buchholtz, and L. Baumanns, Prompt the problem—investigating the mathematics educational quality of AI-supported problem solving by comparing prompt techniques, Front. Educ. 9, 1386075 (2024).
  34. M.-L. Tsai, C. W. Ong, and C.-L. Chen, Exploring the use of large language models (LLMs) in chemical engineering education: Building core course problem models with Chat-GPT, Educ. Chem. Eng. 44, 71 (2023).
  35. T. M. Clark, Investigating the use of an artificial intelligence chatbot with general chemistry exam questions, J. Chem. Educ. 100, 1905 (2023).
  36. T. M. Clark, E. Anderson, N. M. Dickson-Karn, C. Soltanirad, and N. Tafini, Comparing the performance of college chemistry students with ChatGPT for calculations involving acids and bases, J. Chem. Educ. 100, 3934 (2023).
  37. M. A. Rangkuti and R. Karam, Conceptual challenges with the graphical representation of the propagation of a pulse in a string, Phys. Rev. Phys. Educ. Res. 18, 020119 (2022).
  38. M. A. Rangkuti and R. Karam, Encouraging students to understand the 1D wave equation, Phys. Rev. Phys. Educ. Res. 19, 020161 (2023).
  39. L. C. McDermott, M. L. Rosenquist, and E. H. van Zee, Student difficulties in connecting graphs and physics: Examples, Am. J. Phys. 55, 503 (1987).
  40. M. Planinic, Z. Milin-Sipus, H. Katic, A. Susac, and L. Ivanjek, Comparison of student understanding of line graph slope in physics and mathematics, Int. J. Sci. Math. Educ. 10, 1393 (2012).
  41. W. Yeadon and T. Hardy, The impact of AI in physics education: A comprehensive review from GCSE to university levels, Phys. Educ. 59, 025010 (2024).
  42. G. Polverini and B. Gregorcic, How understanding large language models can inform the use of ChatGPT in physics education, Eur. J. Phys. 45, 025701 (2024).
  43. K. D. Wang, E. Burkholder, C. Wieman, S. Salehi, and N. Haber, Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving, Front. Educ. 8, 1330486 (2024).
  44. P. Barniol and G. Zavala, Mechanical waves conceptual survey: Its modification and conversion to a standard multiple-choice test, Phys. Rev. Phys. Educ. Res. 12, 010107 (2016).
  45. S. Puttisanwimon and S. Wuttiprom, Mechanical wave concepts of Thai high school students: Comparing learned and unlearned groups, Int. J. Innovation Sci. Math. Educ. 31 (2023).
  46. T. Kubiszyn and G. D. Borich, Educational Testing and Measurement (John Wiley & Sons, New York, 2024).
  47. M. Kryjevskaia, M. K. R. Stetzer, and P. R. L. Heron, Student understanding of wave behavior at a boundary: The relationships among wavelength, propagation speed, and frequency, Am. J. Phys. 80, 339 (2012).
  48. X. Zhai, M. Nyaaba, and W. Ma, Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science?, Sci. Educ. 34, 649 (2025).
  49. G. Polverini, J. Melin, E. Önerud, and B. Gregorcic, Performance of ChatGPT on tasks involving physics visual representations: The case of the brief electricity and magnetism assessment, Phys. Rev. Phys. Educ. Res. 21, 010154 (2025).
  50. G. Polverini and B. Gregorcic, Evaluating vision-capable chatbots in interpreting kinematics graphs: A comparative study of free and subscription-based models, Front. Educ. 9, 1452414 (2024).
  51. Y. Liang, D. Zou, H. Xie, and F. L. Wang, Exploring the potential of using ChatGPT in physics education, Smart Learn. Environ. 10, 52 (2023).
  52. D. M. Colaco and D. Antao, Exploring self-learning efficacy: A comparative analysis of ChatGPT and Gemini, Interact. Technol. Smart Educ. 22, 688 2025).
  53. M. E. Frenkel and H. Emara, ChatGPT-3.5 and-4.0 and mechanical engineering: Examining performance on the FE mechanical engineering and undergraduate exams, Comput. Appl. Eng. Educ. 32, e22781 (2024).
  54. C. K. Y. Chan and L. H. Y. Tsi, Will generative AI replace teachers in higher education? A study of teacher and student perceptions, Stud. Educ. Eval. 83, 101395 (2024).
  55. H. Jang and H. Choi, A double-edged sword: Physics educators’ perspectives on utilizing ChatGPT and its future in classrooms, J. Sci. Educ. Technol. 34, 267 (2025).
  56. G. Kortemeyer, M. Babayeva, G. Polverini, R. Widenhorn, and B. Gregorcic, Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories, Phys. Rev. Phys. Educ. Res. 21, 020101 (2025).
  57. G. Revalde, M. Zholdakhmet, A. Abola, and A. Murzagaliyeva, Can ChatGPT pass a physics test?, Technol. Knowl. Learn. 30, 2459 (2025).
  58. M. N. Dahlkemper, S. Z. Lahme, and P. Klein, How do physics students evaluate artificial intelligence responses on comprehension questions? A study on the perceived scientific accuracy and linguistic quality of ChatGPT, Phys. Rev. Phys. Educ. Res. 19, 010142 (2023).

Outline

Information

Phys. Rev. Phys. Educ. Res. 22, 020113– Published 6 August, 2026
  • Received 10 November 2025
  • Accepted 15 July 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation