- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

When AI evaluates its own work: Validating learner-initiated, AI-generated physics practice problems

Tobias Geisler*

Gerd Kortemeyer†,‡

  • Department of Information Technology and Electrical Engineering, ETH Zurich, 8092 Zurich, Switzerland

  • Rectorate and ETH AI Center, ETH Zurich, 8092 Zurich, Switzerland

  • *Contact author: dblthere@gmail.com
  • †Contact author: kgerd@ethz.ch
  • ‡Also at Michigan State University, East Lansing, Michigan 48823, USA.
PDF

Phys. Rev. Phys. Educ. Res. 22, 010134 – Published 9 April, 2026

DOI: https://doi.org/10.1103/9hh9-vt4d

Abstract

Large language models (LLMs) can now generate physics practice problems in real time, yet the educational value of these items hinges on rapid, reliable postgeneration vetting. In this exploratory study, we investigated which automated checks are both technically feasible and pedagogically meaningful when exercises are produced on demand within a chatbot interface. A cohort of 34 introductory-physics students generated and attempted 543 practice problems during exam preparation. Each item was labeled by an expert on a wide range of quality attributes and presented to the learners in pairs to record their preference. We then (i) benchmarked three commodity LLMs as “judges” against the expert labels, (ii) quantified which attributes predict student choice via random-forest models, and (iii) triangulated these results with free-form exit surveys. Only a small subset of the original metric items proved necessary to reliably address student preferences either directly or by proxy. The study demonstrates that scalable formative assessment does not require exhaustive scoring: a carefully curated core of structural and learner-visible checks is sufficient to ensure both technical soundness and user appeal. The findings provide a practical blueprint for deploying real-time, AI-generated practice in physics and other quantitative disciplines.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (82)

  1. OpenAI, Introducing gpt-5.4, https://openai.com/index/introducing-gpt-5-4/ (2025) [accessed 2026-03-12].
  2. V. Shekoyan and E. Etkina, Using cognitive apprenticeship framework and multiple-possibility problems to enhance epistemic cognition, AIP Conf. Proc. 1179, 269 (2009).
  3. D. Kagan, E. Ayars, and D. Brookes, The problem with some problems, Phys. Teach. 57, 76 (2019).
  4. F. Reif, J. H. Larkin, and G. C. Brackett, Teaching general learning and problem-solving skills, Am. J. Phys. 44, 212 (1976).
  5. A. B. Arons, Teaching Introductory Physics (John Wiley & Sons, Hoboken, NJ, 1996).
  6. N. R. Council et al., How People Learn: Brain, Mind, Experience, and School: Expanded Edition (National Academies Press, Washington, DC, 2000).
  7. R. J. Dufresne and W. J. Gerace, Assessing-to-learn: Formative assessment in physics instruction, Phys. Teach. 42, 428 (2004).
  8. C. Wagner and A. Vaterlaus, Promoting formative assessment in high school teaching of physics, Lat. Am. J. Phys. Educ. 6, 410 (2012), http://www.lajpe.org/icpe2011/80_Wagner_Clemens.pdf.
  9. D. A. Kashy, G. Albertelli, W. Bauer, E. Kashy, and M. Thoennessen, Influence of non-moderated and moderated discussion sites on student success, J. Asynchronous Learn. Netw. 7, 31 (2003).
  10. G. Kortemeyer, E. Kashy, W. Benenson, and W. Bauer, Experiences using the open-source learning content management and assessment system LON-CAPA in introductory physics courses, Am. J. Phys. 76, 438 (2008).
  11. J. Risley, Motivating students to learn physics using an online homework system, Newslett. APS Forum Educ. Fall 2001, 3 (2001), https://higherlogicdownload.s3.amazonaws.com/APS/379bd548-10a1-4054-aec2-911470db8df9/UploadedImages/Documents/fall01.pdf.
  12. G. Tang and A. Titus, Increasing student’s time on task in calculus and general physics courses through WebAssign, in ASEE Annual Conference and Exposition, Montreal, Canada (American Society for Engineering Education, Washington, DC, 2002), pp. 7.660.1–7.660.20.
  13. S. W. Bonham, D. L. Deardorff, and R. J. Beichner, Comparison of student performance using web and paper-based homework in college-level physics, J. Res. Sci. Teach. 40, 1050 (2003).
  14. B. Gutmann, G. Gladding, M. Lundsgaard, and T. Stelzer, Mastery-style homework exercises in introductory physics courses: Implementation matters, Phys. Rev. Phys. Educ. Res. 14, 010128 (2018).
  15. S. P. Bates, R. K. Galloway, J. Riise, and D. Homer, Assessing the quality of a student-generated question repository, Phys. Rev. ST Phys. Educ. Res. 10, 020105 (2014).
  16. A. Sperling and J. Lincoln, Artificial intelligence and high school physics, Phys. Teach. 62, 314 (2024).
  17. P. Wattanakasiwich, K. Kaewkhong, and D. Katwibun, Physics instructors’ acceptance and implementation of generative AI, Phys. Rev. Phys. Educ. Res. 21, 010155 (2025).
  18. S. Kilde-Westberg, A. Johansson, and J. Enger, Generative AI as a lab partner: A case study, Phys. Rev. Phys. Educ. Res. 21, 020119 (2025).
  19. G. Kortemeyer, M. Babayeva, G. Polverini, R. Widenhorn, and B. Gregorcic, Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories, Phys. Rev. Phys. Educ. Res. 21, 020101 (2025).
  20. J. C. Dunlap, R. Sissons, and R. Widenhorn, Descending an inclined plane with a large language model, Phys. Rev. Phys. Educ. Res. 21, 010153 (2025).
  21. S. Küchemann, S. Steinert, N. Revenga, M. Schweinberger, Y. Dinc, K. E. Avila, and J. Kuhn, Can ChatGPT support prospective teachers in physics task development?, Phys. Rev. Phys. Educ. Res. 19, 020128 (2023).
  22. J. Lademann, J. Henze, and S. Becker-Genschow, Augmenting learning environments using AI custom chatbots: Effects on learning performance, cognitive load, and affective variables, Phys. Rev. Phys. Educ. Res. 21, 010147 (2025).
  23. P. Bitzenbauer, ChatGPT in physics education: A pilot study on easy-to-implement activities, Contemp. Educ. Technol. 15, ep430 (2023).
  24. G. Kortemeyer, Ethel: A virtual teaching assistant, Phys. Teach. 62, 698 (2024).
  25. G. Kortemeyer and J. Nöhl, Assessing confidence in AI-assisted grading of physics exams through psychometrics: An exploratory study, Phys. Rev. Phys. Educ. Res. 21, 010136 (2025).
  26. Z. Chen and T. Wan, Grading explanations of problem-solving process and generating feedback using large language models at human-level accuracy, Phys. Rev. Phys. Educ. Res. 21, 010126 (2025).
  27. B. Gregorcic, G. Polverini, and A. Sarlah, ChatGPT as a tool for honing teachers’s Socratic dialogue skills, Phys. Educ. 59, 045005 (2024).
  28. M. A. R. Vasconcelos and R. P. dos Santos, Enhancing STEM learning with ChatGPT and Bing chat as objects to think with: A case study, Eurasia J. Math. Sci. Technol. Educ. 19, em2296 (2023).
  29. L. Ding, T. Li, S. Jiang, and A. Gapud, Students’ perceptions of using ChatGPT in a physics class as a virtual tutor, Int. J. Educ. Technol. Higher Educ. 20, 63 (2023).
  30. F. Balabdaoui, N. Dittmann-Domenichini, H. Grosse, C. Schlienger, and G. Kortemeyer, A survey on students’ use of AI at a technical university, Discov. Educ. 3, 51 (2024).
  31. G. Kortemeyer, Could an artificial-intelligence agent pass an introductory physics course?, Phys. Rev. Phys. Educ. Res. 19, 010132 (2023).
  32. K. A. Pimbblet and L. J. Morrell, Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degrees, Eur. J. Phys. 46, 015702 (2024).
  33. G. Polverini and B. Gregorcic, Performance of ChatGPT on the test of understanding graphs in kinematics, Phys. Rev. Phys. Educ. Res. 20, 010109 (2024).
  34. Y. Niu and H. Xue, Exercise generation and student cognitive ability research based on ChatGPT and Rasch model, IEEE Access 11, 116695 (2023).
  35. S. Maity, A. Deroy, and S. Sarkar, How effective is gpt-4 turbo in generating school-level questions from textbooks based on Bloom’s revised taxonomy?, arXiv:2406.15211.
  36. P. A. Kirschner, J. Sweller, and R. E. Clark, Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching, Educ. Psychol. 41, 75 (2006).
  37. S. El-Adawy, I. Liao, V. Lad, M. Abdelhafez, and P. Dourmashkin, Streamlining physics problem generation to support physics teachers in using generative artificial intelligence, Phys. Teach. 62, 595 (2024).
  38. J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu et al., A survey on LLM-as-a-judge, arXiv:2411.15594.
  39. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., Retrieval-augmented generation for knowledge-intensive NLP tasks, in Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada (2020), https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf.
  40. M. Scott, T. Stelzer, and G. Gladding, Evaluating multiple-choice exams in large introductory physics courses, Phys. Rev. ST Phys. Educ. Res. 2, 020102 (2006).
  41. W. Fakcharoenphol, E. Potter, and T. Stelzer, What students learn when studying physics practice exam problems, Phys. Rev. ST Phys. Educ. Res. 7, 010107 (2011).
  42. R. Gautreau and L. Novemsky, Concepts first–A small group approach to physics learning, Am. J. Phys. 65, 418 (1997).
  43. W. Fakcharoenphol and T. Stelzer, Physics exam preparation: A comparison of three methods, Phys. Rev. ST Phys. Educ. Res. 10, 010108 (2014).
  44. M. Rodriguez and G. Potvin, Frequent small group interactions improve student learning gains in physics: Results from a nationally representative pre-post study of four-year colleges, Phys. Rev. Phys. Educ. Res. 17, 020131 (2021).
  45. M. J. Gierl, O. Bulut, Q. Guo, and X. Zhang, Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review, Rev. Educ. Res. 87, 1082 (2017).
  46. T. F. Scott and D. Schumayer, Central distractors in force concept inventory data, Phys. Rev. Phys. Educ. Res. 14, 010106 (2018).
  47. P. Heller, R. Keith, and S. Anderson, Teaching problem solving through cooperative grouping, Am. J. Phys. 60, 627 (1992).
  48. A. R. Mota, N. Didiş Körhasan, K. Miller, and E. Mazur, Homework as a metacognitive tool in an undergraduate physics course, Phys. Rev. Phys. Educ. Res. 15, 010136 (2019).
  49. C. Walkington and M. L. Bernacki, Personalizing algebra to students’ individual interests in an intelligent tutoring system: Moderators of impact, Int. J. Artif. Intell. Educ. 29, 58 (2019).
  50. G. Taasoobshirazi and M. Carr, A review and critique of context-based physics instruction and assessment, Educ. Res. Rev. 3, 155 (2008).
  51. Z. Dulger and F. Ogan-Bekiroglu, Students’ metacognition knowledge and skills during physics problem-solving process, Phys. Rev. Phys. Educ. Res. 21, 020106 (2025).
  52. C. Harrison, C. P. Constantinou, C. F. Correia, M. Grangeat, M. Hähkiöniemi, M. Livitzis, P. Nieminen, N. Papadouris, E. Rached, N. Serret et al., Assessment on-the-fly: Promoting and collecting evidence of learning through dialogue, in Transforming Assessment: Through an Interplay between Practice, Research and Policy (2018), pp. 83–107, 10.1007/978-3-319-63248-3.
  53. D. Sadigh, S. A. Seshia, and M. Gupta, Automating exercise generation: A step towards meeting the MOOC challenge for embedded systems, in Proceedings of the Workshop on Embedded and Cyber-Physical Systems Education (Association for Computing Machinery, New York, NY, 2012), pp. 1–8.
  54. M. N. Demaidi, M. M. Gaber, and N. Filer, Evaluating the quality of the ontology-based auto-generated questions, Smart Learn. Environ. 4, 7 (2017).
  55. V. Nentwich, N. Fischer, A. C. Sonnenbichler, and A. Geyer-Schulz, Computer aided exercise generation—A framework for human interaction in the automated exercise generation process, in Proceedings of the 13th International Joint Conference on e-Business and Telecommunications (SCITEPRESS, Setúbal, Portugal, 2016), pp. 57–63.
  56. I. Aldabe, M. L. De Lacalle, M. Maritxalar, E. Martinez, and L. Uria, Arikiturri: An automatic question generator based on corpora and NLP techniques, in Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, Jhongli, Taiwan (Springer, New York, NY, 2006), pp. 584–594.
  57. T. Freitas, Á. Neto, M. J. Pereira, and P. Henriques, NLP/AI based techniques for programming exercises generation, in Proceedings of the 4th International Computer Programming Education Conference (ICPEC 2023) (University of Minho, Porto, 2023), Vol. 112, 10.4230/OASIcs.ICPEC.2023.9.
  58. G. Chen, J. Yang, C. Hauff, and G.-J. Houben, LearningQ: A large-scale dataset for educational question generation, in Proceedings of the International AAAI Conference on Web and Social Media (Association for the Advancement of Artificial Intelligence, Washington, DC, 2018), Vol. 12.
  59. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., Training verifiers to solve math word problems, arXiv:2110.14168.
  60. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, Measuring mathematical problem solving with the MATH dataset, arXiv:2103.03874.
  61. J. Macina, N. Daheim, S. P. Chowdhury, T. Sinha, M. Kapur, I. Gurevych, and M. Sachan, Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems, arXiv:2305.14536.
  62. J. Doughty, Z. Wan, A. Bompelli, J. Qayum, T. Wang, J. Zhang, Y. Zheng, A. Doyle, P. Sridhar, A. Agarwal et al., A comparative study of AI-generated (gpt-4) and human-crafted MCQs in programming education, in Proceedings of the 26th Australasian Computing Education Conference (Association for Computing Machinery, New York, NY, 2024), pp. 114–123.
  63. G. Kortemeyer, J. Nöhl, and D. Onishchuk, Grading assistance for a handwritten thermodynamics exam using artificial intelligence: An exploratory study, Phys. Rev. Phys. Educ. Res. 20, 020144 (2024).
  64. M. X. Liu, F. Liu, A. J. Fiannaca, T. Koo, L. Dixon, M. Terry, and C. J. Cai, “We need structured output”: Towards user-centered constraints on large language model output, in Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery, New York, NY, 2024), pp. 1–9.
  65. J. L. Docktor, J. Dornfeld, E. Frodermann, K. Heller, L. Hsu, K. A. Jackson, A. Mason, Q. X. Ryan, and J. Yang, Assessing student written problem solutions: A problem-solving rubric with application to introductory physics, Phys. Rev. Phys. Educ. Res. 12, 010130 (2016).
  66. R. V. Hogg, E. A. Tanis, and D. L. Zimmerman, Probability and Statistical Inference, 9th ed. (Pearson, Boston, MA, 2019), Sec. 5.6, p. 202.
  67. L. A. Orawo, Confidence intervals for the binomial proportion: A comparison of four methods, Open J. Stat. 11, 806 (2021).
  68. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, Survey of hallucination in natural language generation, ACM Comput. Surv. 55, 1 (2023).
  69. T. M. Fruchterman and E. M. Reingold, Graph drawing by force-directed placement, Software 21, 1129 (1991).
  70. C. Spearman, The proof and measurement of association between two things, Am. J. Psychol. 15, 72 (1904).
  71. G. Kortemeyer, Virtual-reality graph visualization based on Fruchterman-Reingold using unity and SteamVR, Inf. Vis. 21, 143 (2022).
  72. OpenAI, OpenAI, https://openai.com/ [accessed July 2025].
  73. Microsoft, azure ai services, https://azure.microsoft.com/en-us/products/ai-services [accessed June 2024].
  74. OpenAI, Hello GPT-4o, https://openai.com/index/hello-gpt-4o/ [accessed June 2024].
  75. OpenAI, GPT-4o mini: Advancing cost-efficient intelligence, https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ [accessed February 2025].
  76. OpenAI, OpenAI o3-mini, https://openai.com/index/openai-o3-mini/ [accessed February 2025].
  77. L. Breiman, Random forests, Mach. Learn. 45, 5 (2001).
  78. OpenAI, Introducing gpt-5, https://openai.com/index/introducing-gpt-5/ (2025) [accessed 2026-02-05].
  79. G. Kortemeyer, The boiling-frog problem of physics education, Phys. Teach. 64, 8 (2026).
  80. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., Chain-of-thought prompting elicits reasoning in large language models, in Proceedings of the 36th Conference on Neural Information Processing Systems (NeuIPS 2022), New Orleans, Louisiana, USA (2022), https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
  81. E. Kim and S.-J. Pak, Students do not overcome conceptual difficulties after solving 1000 traditional problems, Am. J. Phys. 70, 759 (2002).
  82. https://github.com/banditrider/mt-exercise-eval-pipeline.

Outline

Information

Phys. Rev. Phys. Educ. Res. 22, 010134– Published 9 April, 2026
  • Received 6 August 2025
  • Accepted 26 March 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation