Volume 9 
December 2008 
Issue #5

A Proposed Framework of Test Administration Methods, Nathan A. Thompson, Assessment Systems Corporation

Abstract

The widespread application of personal computers to educational and psychological testing has substantially increased the number of test administration methodologies available to testing programs. Many of these mediums are referred to by their acronyms, such as CAT, CBT, CCT, and LOFT. The similarities between the acronyms and the methods themselves can be a source of confusion to testing professionals. This purpose of this paper is to suggest a hierarchical classification system for test administration methods and clarify what each entails while also briefly discussing the similarities and differences of each.

Volume 9 
July 2008 
Issue # 4

Testing What Students in the Gap Can Do, Michael Russell, Lisa Famularo, Technology and Assessment Study Collaborative, Boston College 323 Campion Hall, Chestnut Hill, MA 02467

Abstract

Student assessment is an integral component of classroom instruction. Assessment is intended to help teachers identify what students are able to do and what content and skills students must develop further. State tests play an important role in guiding instruction. However, for some students, the tests may lead to inaccurate conclusions about student mastery of key content and skills. For other students, the tests may provide information about what they cannot do, but do not help inform teachers about what students can do.

Volume 9 
April 2008 
Issue #3

Examining Panelist Data from a Bilingual Standard Setting Study, Elaine M. Rodeck, Tzu-Yun Chin, Susan L. Davis, Barbara S. Plake, Buros Center for Testing, University of Nebraska-Lincoln

Abstract

This study examined the relationships between the evaluations obtained from standard setting panelists and changes in ratings between different rounds of a standard setting study that involved setting standards on different language versions of an exam We investigated panelists' evaluations to determine if their perceptions of the standard setting were related to adjustments they made in their recommended cut scores across rounds of the process. The standard setting was conducted for a high school mathematics test composed of multiple-choice and constructed response items. The test was designed for a population of students who speak and receive primary instruction in either English or French. Results indicated panelists' ratings of their ratings and their comfort with the process were related to how their ratings changed across sequential rounds of the process. Differences in the degree to which the evaluations influenced the standard setting judgments were observed across the English and French panelists, with the French group reporting increasing comfort across rounds in contrast to the English group that had relatively higher comfort at the beginning of the process. The results illustrate how standard setting evaluation data can provide insight into factors that affect panelists' ratings.

Volume 9 
February 2008 
Issue #2

A Non-Technical Approach for Illustrating Item Response Theory, Chong Ho Yu, Angel Jannasch-Pennell, & Samuel DiGangi, Arizona State University

Abstract

Since the introduction of the No Child Left Behind Act, assessment has become a pre-dominant theme in the US K-12 system. However, making assessment results understandable and usable for the K-12 teachers has been a challenge. While test technology offered by various vendors has been widely implemented, technology of training for test development seems to be under-developed. The objective of this presentation is to illustrate a well-designed interactive tutorial for understanding the complex concepts of Item Response Theory (IRT). The approach of this tutorial is to dissociate IRT from Classical Test Theory (CTT) because it is the belief of the authors that the mis-analogy between IRT and CTT could lead to misconceptions. Initial user feedback is collected as input for further refining the program.

Volume 9 
February 2008 
Issue #1

Matching the Judgmental Task with Standard Setting Panelist Expertise: The Item-Descriptor (ID) Matching Method. Steve Ferrara, CTB/McGraw-Hill; Marianne Perie, National Center for the Improvement of Educational Assessment; and Eugene Johnson, Independent Consultant

Abstract

Psychometricians continue to introduce new approaches to setting cut scores for educational assessments in an attempt to improve on current methods. In this paper we describe the Item-Descriptor (ID) Matching method, a method based on IRT item mapping. In ID Matching, test content area experts match items (i.e., their judgments about the knowledge and skills required to respond to an item) to the knowledge and skills described in performance level descriptors that are used for reporting test results. We argue that the cognitive-judgmental task of matching item response requirements to performance level descriptors is aligned closely with the experience and expertise of standard setting panelists, who are typically classroom teachers and other content area experts. Unlike other popular standard setting methods, ID Matching does not require panelists to make error-prone probability judgments, predict student performance, or imagine examinees who are just barely in a performance level. We describe applications of ID Matching in two educational testing programs and provide evidence of the effectiveness of this method. The entire process is described in the first section of the paper. Subsequent sections describe applications of ID Matching for two operational testing programs.