1st Edition
Research for Practical Issues and Solutions in Computerized Multistage Testing
This volume presents a comprehensive collection of the latest research findings supporting the current and future implementations and applications of computerized multistage testing (MST).
As a sequel to the widely acclaimed Computerized Multistage Testing: Theory and Applications (2014) by Yan, von Davier, and Lewis, this volume delves into the experiences, considerations, challenges, and lessons learned over the past years. It also offers practical approaches and solutions to the issues encountered. The topics covered include purposeful MST designs, practical approaches for optimal design, assembly strategies for accuracy and efficiency, hybrid designs, MST with natural language processing, practical routing considerations and methodologies, item calibration and proficiency estimation methods, routing and classification accuracy, added value of process data, prediction and evaluation of MST performance, cognitive diagnostic MST, differential item functioning, robustness of statistical methods, simulations, test security, the new digital large-scale Scholastic Aptitude Test, software for practical assessment and simulations, artificial intelligence impact, and the future of adaptive MST.
This volume is intended for students, faculty, researchers, practitioners, and education officers in the fields of educational measurement and evaluation in the United States and internationally.
Foreword
Nathan Thompson
Preface
Duanli Yan, Alina A. von Davier, and David J. Weiss
1. A brief history of computerized adaptive and multistage testing
David J. Weiss and Duanli Yan
PART I Multistage test design and assembly
2. Purposeful design for useful tests: Considering choices in multistage-adaptive testing
April L. Zenisky and Stephen G. Sireci
3. MST strategic design issues and implementation
Ric Luecht and Xiao Luo
4. Designing multistage tests to meet accuracy and efficiency goals
Unhee Ju, Mark D. Reckase, and Sewon Kim
5. Investigating hybrid test designs in testlet-based adaptive tests
Ye Ma, Deborah J. Harris, and Stephen B. Dunbar
6. A practical approach to finding an optimal multistage test design
Hwanggyu Lim and Tim Davey
7. Improving literacy by integrating advances in the learning sciences and natural language processing in multistage testing
Paul Deane, Tenaha O'Reilly, Duanli Yan, Zuowei Wang, and Jonathan Weeks
PART II MST routing, scoring, and estimation
8. Multistage testing with intersectional routing for short-length tests
Kyung (Chris) T. Han
9. Effect of routing errors on the psychometric properties of multistage tests
Robert Chapman, David J. Weiss, and King Yiu Suen
10. Item calibration in multistage tests
Paul A. Jewsbury and Peter W. van Rijn
11. Item response theory proficiency estimation methods under multistage testing
Sooyeon Kim and Tim Moses
12. Multistage tests under D-scoring approach
Kyung (Chris) T. Han, Dimiter Dimitrov, and Faisal Al-Mashary
13. Development and application of probability-weighted classification for multistage testing
Victoria Song, Duanli Yan, and Charles Lewis
14. Creating value from process data: Implications for multistage testing
Okan Bulut
Part III: MST evaluations
15. Evaluating multistage testing performance
Tim Davey
16. Module assembly and routing of cognitive diagnostic multistage adaptive test
Manqian Liao and Hong Jiao
17. Differential item functioning in multistage tests
Ru Lu and Paul A. Jewsbury
18. Navigating statistical challenges in the transition from linear to multistage adaptive testing
Usama S. Ali, Hyo Jeong Shin, and Peter W. Van Rijn
19. Conducting simulation studies for computerized MST research
Halil I. Sari and A. Corinne Huggins-Manley
20. Test security considerations for CAT and MST
Kirk Becker, J. Carl Setzer, Matthew Schultz, Yiqin Pan, and Kaiwen Man
PART IV Applications and Technologies
21. Considerations in the MST design for the new digital SAT suite of assessments
Thomas Proctor and Oliver Zhang
22. Build high-quality MST panels with mixed-integer programming in R
Xiao Luo and Richard M. Luecht
23. Bayesian inference for multistage and other incomplete designs
Jesse Koops, Timo Bechger, and Gunter Maris
24. An overview of computerized adaptive and multistage testing software
Ye Ma and Duanli Yan
25. How will AI change adaptive testing?
Alina A. von Davier
26. Afterword: The emergence of personalized ensemble testing
Alina A. von Davier
Biography
Duanli Yan is a Director of Computational Research at Educational Testing Services, Princeton, New Jersey, USA. She is also an adjunct professor at Rutgers University and Fordham University and has extensive experience in innovative psychometric research and development. She has published many books and received many awards, including the 2016 AERA D Significant Contribution to Educational Measurement and Research Methodology Award, and the 2022 and 2023 NCME Bradley Hanson Award.
Alina A. von Davier is the Chief of Assessment at Duolingo, Pittsburgh, Pennsylvania, USA. She leads the Duolingo English Test research and development area. She is a researcher in computational psychometrics, machine learning, and education. Von Davier is an innovator and an executive leader with over 20 years of experience in EdTech and in the assessment industry. In 2022, she joined the University of Oxford as an Honorary Research Fellow, and Carnegie Mellon University as a Senior Research Fellow.
David J. Weiss is a Professor of Psychology at University of Minnesota, Minnesota, USA. He has been continuously active in computerized adaptive testing (CAT) research since 1970, including hosting six international CAT conferences. He co-founded the International Association for Computerized Adaptive Testing, the Assessment Systems Corporation, and the Insurance Testing Corporation and was the founding editor of Applied Psychological Measurement and the Journal of Computerized Adaptive Testing.
