- Open Access
Measuring student learning with item response theory
Phys. Rev. ST Phys. Educ. Res. 4, 010102 – Published 31 January, 2008
DOI: https://doi.org/10.1103/PhysRevSTPER.4.010102
Abstract
We investigate short-term learning from hints and feedback in a Web-based physics tutoring system. Both the skill of students and the difficulty and discrimination of items were determined by applying item response theory (IRT) to the first answers of students who are working on for-credit homework items in an introductory Newtonian physics course. We show that after tutoring a shifted logistic item response function with lower discrimination fits the students’ second responses to an item previously answered incorrectly. Student skill decreased by 1.0 standard deviation when students used no tutoring between their (incorrect) first and second attempts, which we attribute to “item-wrong bias.” On average, using hints or feedback increased students’ skill by 0.8 standard deviation. A skill increase of 1.9 standard deviation was observed when hints were requested after viewing, but prior to attempting to answer, a particular item. The skill changes measured in this way will enable the use of IRT to assess students based on their second attempt in a tutoring environment.
Article Text
References (13)
- A. Collins, J. S. Brown, and S. E. Newman, Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics, in Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser, edited by L. B. Resnick (Lawrence Erlbaum, Hillsdale, NJ, 1990), p. 453.
- http://www.masteringphysics.com/ was made by Effective Educational Technologies, a company started by the family of one of the authors (D.E.P.). It has been purchased by Pearson Education.
- N. Chudowsky, R. Glaser, and J. W. Pellegrino, Knowing What Students Know: The Science and Design of Educational Assessment (National Academies Press, Washington, DC, 2001).
- F. B. Baker and S.-H. Kim, Item Response Theory: Parameter Estimation Techniques (Marcel Dekker, New York, 2004).
- R. K. Hambleton and H. Swaminathan, Item Response Theory: Principles and Applications (Kluwer Nijhoff, Boston, 1985).
- F. M. Lord, Applications of Item Response Theory to Practical Testing Problems (Erlbaum, Mahwah, NJ, 1980).
- Computer code BILOG-MG (Version 3.0) (Assessment Systems Corporation; St. Paul, MN, 2003).
- S. Kim, F. B. Baker, and M. J. Subkoviak, The rules in the minimum logit chi-square estimation procedure when small samples are used, Br. J. Math. Stat. Psychol. 42, 113 (1989).
We also examined if subtasks were more beneficial than hints, considering the fact that subtasks provide much more specific information than hints, but found no statistical evidence.
This “hints first” route was followed in 11.0% of all cases, and the median, mode, and maximum use of this route by students were 8.6%, 0.01%, and 71%, respectively.
- E.-S. Morote and D. E. Pritchard, Technology closes the gap between students' individual skills and background difference, in Proceedings of the Society for Information Technology and Teacher Education: International Conference, Atlanta, GA edited by C. Crowford, D. A. Willis, R. Carlsen, I. Gibson, K. McFerrin, J. Price, and R. Weber (AACE, Chesapeake, VA, 2004), p. 826.
- R. Warnakulasooriya and D. E. Pritchard, Time to completion reveals problem-solving transfer, in Proceedings of the Physics Education Research Conference, Sacramento, CA, edited by J. Max, P. Heron, and S. Franklin (AIP, Melville, NY, 2004), p. 205.
- R. Warnakulasooriya, D. J. Palazzo, and D. E. Pritchard, Evidence of problem-solving transfer in web-based Socratic tutor, in Proceedings of the Physics Education Research Conference, Salt Lake City, UT, edited by P. Heron, L. McCullough, and J. Max (AIP, Melville, NY, 2005), p. 41.
