Showing 1–13 of 13 results for author: Smith, T I

Comparing item response theory models for ranking incorrect response options

Authors: Trevor I. Smith, Mohammad A. Kazmi, Richard R. Sarles III, Joshua A. Sbrana, Cody W. Soper, Nasrine Bendjilali

Abstract: Previous work has shown that item response theory may be used to rank incorrect response options to multiple-choice items on commonly used assessments. This work has shown that, when the correct response to each item is specified, a nominal response model (NRM) may be used to rank the incorrect options. We seek to determine the robustness of these results by applying the NRM to all response choice… ▽ More Previous work has shown that item response theory may be used to rank incorrect response options to multiple-choice items on commonly used assessments. This work has shown that, when the correct response to each item is specified, a nominal response model (NRM) may be used to rank the incorrect options. We seek to determine the robustness of these results by applying the NRM to all response choices, without specifying the correct response. We apply these analyses to multiple data sets (each with more than 9,000 response sets), including pre-instruction and post-instruction responses. We find that the rankings generated without specifying the correct response are consistent with the previously published rankings for one data set; however, we find noticeable differences between rankings generated from different data sets. We provide evidence that discrepancies may result from differences in response rates for less commonly chosen responses. △ Less

Submitted 7 October, 2024; originally announced October 2024.

Replicating analyses of item response curves using data from the Force and Motion Conceptual Evaluation

Authors: Connor J. Richardson, Trevor I. Smith, Paul J. Walter

Abstract: Ishimoto, Davenport, and Wittmann have previously reported analyses of data from student responses to the Force and Motion Conceptual Evaluation (FMCE), in which they used item response curves (IRCs) to make claims about American and Japanese students' relative likelihood to choose certain incorrect responses to some questions. We have used an independent data set of over 6,500 American students'… ▽ More Ishimoto, Davenport, and Wittmann have previously reported analyses of data from student responses to the Force and Motion Conceptual Evaluation (FMCE), in which they used item response curves (IRCs) to make claims about American and Japanese students' relative likelihood to choose certain incorrect responses to some questions. We have used an independent data set of over 6,500 American students' responses to the FMCE to generate IRCs to test their claims. Converting the IRCs to vectors, we used dot product analysis to compare each response item quantitatively. For most questions, our analyses are consistent with Ishimoto, Davenport, and Wittmann, with some results suggesting more minor differences between American and Japanese students than previously reported. We also highlight the pedagogical advantages of using IRCs to determine the differences in response patterns for different populations to better understand student thinking prior to instruction. △ Less

Submitted 17 April, 2021; originally announced April 2021.

Journal ref: Phys. Rev. Phys. Educ. Res. 17, 020127 (2021)

The Physics Inventory of Quantitative Literacy: A tool for assessing mathematical reasoning in introductory physics

Authors: Suzanne White Brahmia, Alexis Olsho, Trevor I. Smith, Andrew Boudreaux, Philip Eaton, Charlotte Zimmerman

Abstract: One desired outcome of introductory physics instruction is that students will develop facility with reasoning quantitatively about physical phenomena. Little research has been done regarding how students develop the algebraic concepts and skills involved in reasoning productively about physics quantities, which is different from either understanding of physics concepts or problem-solving abilities… ▽ More One desired outcome of introductory physics instruction is that students will develop facility with reasoning quantitatively about physical phenomena. Little research has been done regarding how students develop the algebraic concepts and skills involved in reasoning productively about physics quantities, which is different from either understanding of physics concepts or problem-solving abilities. We introduce the Physics Inventory of Quantitative Literacy (PIQL) as a tool for measuring quantitative literacy, a foundation of mathematical reasoning, in the context of introductory physics. We present the development of the PIQL and evidence of its validity for use in calculus-based introductory physics courses. Unlike concept inventories, the PIQL is a reasoning inventory, and can be used to assess reasoning over the span of students' instruction in introductory physics. Although mathematical reasoning associated with the PIQL is taught in prior mathematics courses, pre/post test scores reveal that this reasoning isn't readily used by most students in physics, nor does it develop as part of physics instruction--even in courses that use high-quality, research-based curricular materials. As has been the case with many inventories in physics education, we expect use of the PIQL to support the development of instructional strategies and materials--in this case, designed to meet the course objective that all students become quantitatively literate in introductory physics. △ Less

Submitted 9 June, 2021; v1 submitted 22 September, 2020; originally announced September 2020.

Toward a valid instrument for measuring physics quantitative literacy

Authors: Trevor I. Smith, Philip Eaton, Suzanne White Brahmia, Alexis Olsho, Andrew Boudreaux, Charlotte Zimmerman

Abstract: We have developed the Physics Inventory of Quantitative Literacy (PIQL) as a tool to measure students' quantitative literacy in the context of introductory physics topics. We present the results from various quantitative analyses used to establish the validity of both the individual items and the PIQL as a whole. We show how examining the results from classical test theory analyses, factor analysi… ▽ More We have developed the Physics Inventory of Quantitative Literacy (PIQL) as a tool to measure students' quantitative literacy in the context of introductory physics topics. We present the results from various quantitative analyses used to establish the validity of both the individual items and the PIQL as a whole. We show how examining the results from classical test theory analyses, factor analysis, and item response curves informed decisions regarding the inclusion, removal, or modification of items. We also discuss how the choice to include multiple-choice/multiple-response items has informed both our choices for analyses and the interpretations of their results. We are confident that the most recent version of the PIQL is a valid and reliable instrument for measuring students' physics quantitative literacy in calculus-based introductory physics courses at our primary research site. More data are needed to establish its validity for use at other institutions and in other courses. △ Less

Submitted 16 July, 2020; originally announced July 2020.

Comments: accepted for publication: 2020 Physics Education Research Conference

Using psychometric tools as a window into students' quantitative reasoning in introductory physics

Authors: Trevor I. Smith, Philip Eaton, Suzanne White Brahmia, Alexis Olsho, Andrew Boudreaux, Chris DePalma, Victor LaSasso, Scott Straguzzi, Christopher Whitener

Abstract: The Physics Inventory of Quantitative Literacy (PIQL), a reasoning inventory under development, aims to assess students' physics quantitative literacy at the introductory level. The PIQL's design presents the challenge of isolating types of mathematical reasoning that are independent of each other in physics questions. In its current form, the PIQL spans three principle reasoning subdomains previo… ▽ More The Physics Inventory of Quantitative Literacy (PIQL), a reasoning inventory under development, aims to assess students' physics quantitative literacy at the introductory level. The PIQL's design presents the challenge of isolating types of mathematical reasoning that are independent of each other in physics questions. In its current form, the PIQL spans three principle reasoning subdomains previously identified in mathematics and physics education research: ratios and proportions, covariation, and signed (negative) quantities. An important psychometric objective is to test the orthogonality of these three reasoning subdomains. We present results from exploratory factor analysis, confirmatory factor analysis, and module analysis that inform interpretations of the underlying structure of the PIQL from a student viewpoint, emphasizing ways in which these results agree and disagree with expert categorization. In addition to informing the development of existing and new PIQL assessment items, these results are also providing exciting insights into students' quantitative reasoning at the introductory level. △ Less

Submitted 1 October, 2019; v1 submitted 11 July, 2019; originally announced July 2019.

Comments: submitted to 2019 PERC Proceedings

Quantitatively ranking incorrect responses to multiple-choice questions using item response theory

Authors: Trevor I. Smith, Kyle J. Louis, Bartholomew J. Ricci IV, Nasrine Bendjilali

Abstract: Research-based assessment instruments (RBAIs) are ubiquitous throughout both physics instruction and physics education research. The vast majority of analyses involving student responses to RBAI questions have focused on whether or not a student selects correct answers and using correctness to measure growth. This approach often undervalues the rich information that may be obtained by examining st… ▽ More Research-based assessment instruments (RBAIs) are ubiquitous throughout both physics instruction and physics education research. The vast majority of analyses involving student responses to RBAI questions have focused on whether or not a student selects correct answers and using correctness to measure growth. This approach often undervalues the rich information that may be obtained by examining students' particular choices of incorrect answers. In the present study, we aim to reveal some of this valuable information by quantitatively determining the relative correctness of various incorrect responses. To accomplish this, we propose an assumption that allow us to define relative correctness: students who have a high understanding of Newtonian physics are likely to answer more questions correctly and also more likely to choose better incorrect responses, than students who have a low understanding. Analyses using item response theory align with this assumption, and Bock's nominal response model allows us to uniquely rank each incorrect response. We present results from over 7,000 students' responses to the Force and Motion Conceptual Evaluation. △ Less

Submitted 2 June, 2019; originally announced June 2019.

Journal ref: Phys. Rev. Phys. Educ. Res. 16, 010107 (2020)

A framework for the natures of negativity in introductory physics

Authors: Suzanne White Brahmia, Alexis Olsho, Trevor I. Smith, Andrew Boudreaux

Abstract: Mathematical reasoning skills are a desired outcome of many introductory physics courses, particularly calculus-based physics courses. Positive and negative quantities are ubiquitous in physics, and the sign carries important and varied meanings. Novices can struggle to understand the many roles signed numbers play in physics contexts, and recent evidence shows that unresolved struggle can carry o… ▽ More Mathematical reasoning skills are a desired outcome of many introductory physics courses, particularly calculus-based physics courses. Positive and negative quantities are ubiquitous in physics, and the sign carries important and varied meanings. Novices can struggle to understand the many roles signed numbers play in physics contexts, and recent evidence shows that unresolved struggle can carry over to subsequent physics courses. The mathematics education research literature documents the cognitive challenge of conceptualizing negative numbers as mathematical objects--both for experts, historically, and for novices as they learn. We contribute to the small but growing body of research in physics contexts that examines student reasoning about signed quantities and reasoning about the use and interpretation of signs in mathematical models. In this paper we present a framework for categorizing various meanings and interpretations of the negative sign in physics contexts, inspired by established work in algebra contexts from the mathematics education research community. Such a framework can support innovation that can catalyze deeper mathematical conceptualizations of signed quantities in the introductory courses and beyond. △ Less

Submitted 9 March, 2019; originally announced March 2019.

Journal ref: Phys. Rev. Phys. Educ. Res. 16, 010120 (2020)

Developing a reasoning inventory for measuring physics quantitative literacy

Authors: Trevor I. Smith, Suzanne W. Brahmia, Alexis Olsho, Andrew Boudreaux, Philip Eaton, Paul J. Kelly, Kyle J. Louis, Mitchell A. Nussenbaum, Louis J. Remy

Abstract: In an effort to improve the quality of citizen engagement in workplace, politics, and other domains in which quantitative reasoning plays an important role, Quantitative Literacy (QL) has become the focus of considerable research and development efforts in mathematics education. QL is characterized by sophisticated reasoning with elementary mathematics. In this project, we extend the notions of QL… ▽ More In an effort to improve the quality of citizen engagement in workplace, politics, and other domains in which quantitative reasoning plays an important role, Quantitative Literacy (QL) has become the focus of considerable research and development efforts in mathematics education. QL is characterized by sophisticated reasoning with elementary mathematics. In this project, we extend the notions of QL to include the physics domain and call it Physics Quantitative Literacy (PQL). We report on early stage development from a collaboration that focuses on reasoning inventory design and data analysis methodology for measuring the development of PQL across the introductory physics sequence. We have piloted a prototype assessment designed to measure students' PQL in introductory physics: Physics Inventory of Quantitative Literacy (PIQL). This prototype PIQL focuses on two components of PQL: proportional reasoning, and reasoning with negative quantities. We present preliminary results from approximately 1,000 undergraduate and 20 graduate students. △ Less

Submitted 10 January, 2019; originally announced January 2019.

Representing uncertainty on model analysis plots

Authors: Trevor I. Smith

Abstract: Model analysis provides a mechanism for representing student learning as measured by standard multiple-choice surveys. The model plot contains information regarding both how likely students in a particular class are to choose the correct answer and how likely they are to choose an answer consistent with a well-documented conceptual model. Unfortunately Bao's original presentation of the model plot… ▽ More Model analysis provides a mechanism for representing student learning as measured by standard multiple-choice surveys. The model plot contains information regarding both how likely students in a particular class are to choose the correct answer and how likely they are to choose an answer consistent with a well-documented conceptual model. Unfortunately Bao's original presentation of the model plot did not include a way to represent uncertainty in these measurements. I present details of a method to add error bars to model plots by expanding the work of Sommer and Lindell. I also provide a template for generating model plots with error bars. △ Less

Submitted 29 August, 2016; v1 submitted 7 April, 2016; originally announced April 2016.

Comments: Accepted for publication in Phys. Rev. Phys. Edu. Res. Published version will include supplementary files

Student understanding of the Boltzmann factor

Authors: Trevor I. Smith, Donald B. Mountcastle, John R. Thompson

Abstract: We present results of our investigation into student understanding of the physical significance and utility of the Boltzmann factor in several simple models. We identify various justifications, both correct and incorrect, that students use when answering written questions that require application of the Boltzmann factor. Results from written data as well as teaching interviews suggest that many st… ▽ More We present results of our investigation into student understanding of the physical significance and utility of the Boltzmann factor in several simple models. We identify various justifications, both correct and incorrect, that students use when answering written questions that require application of the Boltzmann factor. Results from written data as well as teaching interviews suggest that many students can neither recognize situations in which the Boltzmann factor is applicable, nor articulate the physical significance of the Boltzmann factor as an expression for multiplicity, a fundamental quantity of statistical mechanics. The specific student difficulties seen in the written data led us to develop a guided-inquiry tutorial activity, centered around the derivation of the Boltzmann factor, for use in undergraduate statistical mechanics courses. We report on the development process of our tutorial, including data from teaching interviews and classroom observations on student discussions about the Boltzmann factor and its derivation during the tutorial development process. This additional information informed modifications that improved students' abilities to complete the tutorial during the allowed class time without sacrificing the effectiveness as we have measured it. These data also show an increase in students' appreciation of the origin and significance of the Boltzmann factor during the student discussions. Our findings provide evidence that working in groups to better understand the physical origins of the canonical probability distribution helps students gain a better understanding of when the Boltzmann factor is applicable and how to use it appropriately in answering relevant questions. △ Less

Submitted 17 August, 2015; originally announced August 2015.

Identifying Student Difficulties with Entropy, Heat Engines, and the Carnot Cycle

Authors: Trevor I. Smith, Warren M. Christensen, Donald B. Mountcastle, John R. Thompson

Abstract: We report on several specific student difficulties regarding the Second Law of Thermodynamics in the context of heat engines within upper-division undergraduates thermal physics courses. Data come from ungraded written surveys, graded homework assignments, and videotaped classroom observations of tutorial activities. Written data show that students in these courses do not clearly articulate the co… ▽ More We report on several specific student difficulties regarding the Second Law of Thermodynamics in the context of heat engines within upper-division undergraduates thermal physics courses. Data come from ungraded written surveys, graded homework assignments, and videotaped classroom observations of tutorial activities. Written data show that students in these courses do not clearly articulate the connection between the Carnot cycle and the Second Law after lecture instruction. This result is consistent both within and across student populations. Observation data provide evidence for myriad difficulties related to entropy and heat engines, including students' struggles in reasoning about situations that are physically impossible and failures to differentiate between differential and net changes of state properties of a system. Results herein may be seen as the application of previously documented difficulties in the context of heat engines, but others are novel and emphasize the subtle and complex nature of cyclic processes and heat engines, which are central to the teaching and learning of thermodynamics and its applications. Moreover, the sophistication of these difficulties is indicative of the more advanced thinking required of students at the upper division, whose developing knowledge and understanding give rise to questions and struggles that are inaccessible to novices. △ Less

Submitted 17 August, 2015; originally announced August 2015.

Student Understanding of Taylor Series Expansions in Statistical Mechanics

Authors: Trevor I. Smith, John R. Thompson, Donald B. Mountcastle

Abstract: One goal of physics instruction is to have students learn to make physical meaning of specific mathematical ideas, concepts, and procedures in different physical settings. As part of research investigating student learning in statistical physics, we are developing curriculum materials that guide students through a derivation of the Boltzmann factor, using a Taylor series expansion of entropy. Usin… ▽ More One goal of physics instruction is to have students learn to make physical meaning of specific mathematical ideas, concepts, and procedures in different physical settings. As part of research investigating student learning in statistical physics, we are developing curriculum materials that guide students through a derivation of the Boltzmann factor, using a Taylor series expansion of entropy. Using results from written surveys, classroom observations, and both individual think-aloud and teaching interviews, we present evidence that many students can recognize and interpret series expansions, but they often lack fluency with the Taylor series despite previous exposures in both calculus and physics courses. We present students' successes and failures both using and interpreting Taylor series expansions in a variety of contexts. △ Less

Submitted 5 March, 2012; v1 submitted 18 December, 2011; originally announced December 2011.

Comments: 13 pages, 2 figures

Applying a resources framework to analysis of the Force and Motion Conceptual Evaluation

Authors: Trevor I. Smith, Michael C. Wittmann

Abstract: We suggest one redefinition of common clusters of questions used to analyze student responses on the Force and Motion Conceptual Evaluation (FMCE). Our goal is to move beyond the expert/novice analysis of student learning based on pre-/post-testing and the correctness of responses (either on the overall test or on clusters of questions defined solely by content). We use a resources framework, ta… ▽ More We suggest one redefinition of common clusters of questions used to analyze student responses on the Force and Motion Conceptual Evaluation (FMCE). Our goal is to move beyond the expert/novice analysis of student learning based on pre-/post-testing and the correctness of responses (either on the overall test or on clusters of questions defined solely by content). We use a resources framework, taking special note of the contextual and representational dependence of questions with seemingly similar physics content. We analyze clusters in ways that allow the most common incorrect answers to give as much, or more, information as the correctness of responses in that cluster. Furthermore, we show that false positives can be found, especially on questions dealing with Newton's Third Law. △ Less

Submitted 17 March, 2008; v1 submitted 12 November, 2007; originally announced November 2007.

Comments: 13 pages, 7 figures, submitted to Phys. Rev. ST Phys. Educ. Res; Revised: 12 pages, 9 figures, submitted to Phys. Rev. ST Phys. Educ. Res., altered content and focus