Volume 14 - SPECIAL ISSUE

April 2013

Incorporating Innovations in Psychometric and

Cognitive Theory into Operational Tests:

A special issue of the Journal of Applied Testing Technology

Click Here to read: Special Issue Overview by Matthew J. Burke, AICPA

Special Issue, Article 1

Implementing Assessment Engineering in the Uniform Certified Public Accountant (CPA) Examination by Matthew Burke, Ph.D., Richard Devore, Ed.D., Josh Stopek, CPA, MBA

Abstract

 This paper describes efforts to bring principled assessment design to a large-scale, high-stakes licensure examination by employing the frameworks of Assessment Engineering (AE), the Revised Bloom’s Taxonomy (RBT), and Cognitive Task Analysis (CTA).  The Uniform CPA Examination is practice-oriented and focuses on the skills of accounting. In preparation for a Practice Analysis, the authors are revising skill definitions to support the development of construct maps. Construct maps are the means by which distinct levels of proficiency are represented along a continuum and linked to tasks used to provide evidence of proficiency.  A modified CTA is employed to better understand the “moving parts” of existing complex performance exercises used in the examination. The RBT is used to provide a hierarchical structure to the cognitive skills underlying task performance. Construct maps provide the basis for a new type of test blueprint that incorporates psychometric concerns into the test development process. This research focuses on tackling a critical step in AE; writing skill statements to populate the construct map.

Special Issue, Article 2

Evidence-Centered Design: Recommendations for Implementation and Practice

by Amy Hendrickson, Maureen Ewing, Pamela Kaliski,The College Board and Kristen Huff, Regents Research Fund

Abstract

Evidence-centered design (ECD) is an orientation towards assessment development. It differs from conventional practice in several ways and consists of multiple activities. Each of these activities results in a set of useful documentation: domain analysis, domain modeling, construction of the assessment framework, and assessment implementation/delivery (Mislevy & Haertel, 2006). In this article we focus on the four primary challenges we have encountered in our work with ECD in the context of large-scale educational assessment. For each challenge, we identify potential mitigation strategies as well as research studies or other endeavors that we think are helpful in advancing the science of ECD.  The challenges discussed are: integrating learning theory into assessment design; identifying the appropriate levels of specificity with which to document the claims and evidence; developing and evaluating task models; and strategically incorporating iteration into the design process.

Special Issue, Article 3

Use of Bloom’s Taxonomy in Developing Reading Comprehension Specifications by Stephen Luebke and James Lorié, Law School Admission Council

Abstract

This article is a brief account of the use of Bloom’s Taxonomy of Educational Objectives (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956) by staff of the Law School Admission Council in the 1990 development of redesigned specifications for the Reading Comprehension section of the Law School Admission Test. Summary item statistics for the test items developed based on these specifications from 1991 to the present are also presented. These statistics offer evidence that this application of Bloom’s Taxonomy to the development of test specifications was useful and helped achieve testing goals, although with limitations.

Special Issue, Article 4

Assessment Engineering Task Model Maps, Task Models and Templates as a New Way to Develop and Implement Test Specifications by Richard M. Luecht, University of North Carolina at Greensboro

Abstract

Assessment engineering is a new way to design and implement scalable, sustainable and ideally lower-cost solutions to the complexities of designing and developing tests. It represents a merger of sorts between cognitive task modeling and engineering design principles—a merger that requires some new thinking about the nature of score scales, item difficulty, and content. This article summarizes some recent developments in developing AE task model maps, task models, and templates as alternative to more traditional test specifications, and discusses some of the necessary quality control mechanisms that can ensure the consistent production of high quality items and test forms over time.