Abstracts

General Sessions

Keynote: Technology, Legal, and Usage Issues: Psychological Testing in the Wake of 911
David Arnold, Ph.D., Vice President - Development and Professional Compliance, NCS Assessments

This presentation will discuss the impact of the events of September 11 on various forms of psychological testing (clinical, industrial, education, and certification). Issues involving the demand for, and use of testing, the growing importance of technology-based platforms and legal/social standards will be discussed. Additionally, many of these issues will be evaluated in light of the recently enacted Aviation and Transportation Security Act.

Keynote Presentation: Dealing with the News Media: The Good, the Bad and the Ugly
Increasingly, companies in the testing business are finding themselves the subject of media scrutiny as a result of isolated errors, misleading information and the efforts of anti-testing groups to oversimplify testing issues and demonize testing organizations.

How does an organization work effectively with the news media to tell its story? What recourse is there when negative or misleading media coverage occurs? What role should the CEO of the company have in presenting the organization’s point of view? These issues will be explored by two executives from NCS Pearson, which has been the subject of news media coverage the past year.

Keynote Presentation: Distributed Simulation: The Need to Test for Teamwork Skills Using Technology-Based Assessments
Dr. Jack Thorpe is a consultant involved in the definition and planning of advanced technology development projects. His expertise is in Distributed Simulation, and he was the Program Manager at the Defense Advanced Research Projects Agency (DARPA) that initiated the SIMNET simulator networking project; micro-travel; desktop simulators; video arcade trainers; interactive history; the electronic sand table; seamless simulation; the Defense Simulation Internet; and the "60% solution" development methodology.

He served in the Air Force with 26 years active duty as an R&D officer, the last 12 years as a program manager and office director at DARPA. He retired in 1993 at the rank of Colonel, and continues as a consultant on advanced technology development programs.

Keynote Presentation: Showcase of Innovations in Computer-Based Testing
The showcase will highlight the cutting edge of computer based testing:

James Olsen will present educational performance assessment tools and national standards based assessments in discovery learning environments. Students use computer tools (pen, ruler, compass, spreadsheet, calculator) to classify or create plane and solid geometric objects; to measure sides, angles, and radii; perform computations, record results, and write extended explanations showing their understanding.

Dick Devore will overview new item types for a new CPA examination. The AICPA will move the Uniform CPA examination from paper and pencil to CBT delivery in 2003. This change will permit the inclusion of simulations incorporating a number of new and innovative item types and delivery techniques. This presentation will demonstrate some of the item types that have been developed by the AICPA to facilitate the assessment of accounting skills in an environment familiar to most CPAs.

Jerry Goldman will demonstrate an example of AMbER -- advanced multimedia-based experimental research. AMbER uses a web client to deliver decision-making experiences and provides full back-end control to investigators (e.g., controlling access, selecting variables for testing). The prototype relies on a criminal justice example. Attributes such as race, dress and affect may be systematically varied or held constant, depending on the objectives of the investigator. All data are gathered, analyzed and reported in real time. Click-through data assist in identifying outliers.

Keynote Presentation: Academic Standards for the 21st Century
Willard R. Daggett, Ed.D., is President of the International Center for Leadership in Education. He is recognized worldwide for his efforts to move education systems towards more rigorous and relevant skills and knowledge for all students.

Dr. Daggett has assisted several states and many school districts with their school reform initiatives as well as worked with education ministries in other countries. Dr. Daggett serves as the lead consultant for the International Center's network of highly successful model schools committed to rigorous and relevant curriculum for all students. These experiences give him a practical, broad-based perspective to the issues facing education.

Dr. Daggett is the creator of the Application Model, which provides a framework to determine the relevance of curriculum and assessment to real-world situations. The Application Model is part of the Rigor/Relevance Framework, which has become a cornerstone of reform initiatives throughout the United States and the world.

Dr. Daggett founded the International Center after serving in various management positions at the New York State Education Department, where he spearheaded a series of restructuring initiatives to focus the state's education system on the skills and knowledge students need in today's technological, information-based society. He has held both teaching and administrative positions at the secondary, two-year college, four-year liberal arts college and university levels.

Dr. Daggett is the author of 12 textbooks, four books on educational leadership, numerous research reports, and many articles in professional publications. He has addressed audiences in all 50 states and overseas. He also serves as a consultant to many business organizations.

Lifetime Achievement Award to Benjamin Wright
Join the Association of Test Publishers as we recognize Benjamin D. Wright, Ph.D. by honoring him with the Career Achievement Award in Computer-Based Testing.

Dr. Wright is Professor of Education and Psychology at the University of Chicago and Director of the Mesa Psychometric Laboratory. He has published 150 papers on Rasch measurement; authored 12 books, including Best Test Design and Rating Scale Analysis and developed the two most widely used Rasch measurement computer programs, WINSTEPS and Facets. Dr. Wright has taught annual workshops and courses on Rasch measurement since 1969 and chaired 70 doctoral dissertations on that topic. Many of his students are contemporary leaders in psychometrics. Dr. Wright's work serves as the foundation for a significant proportion of the computer based tests represented at this conference.

Accepting the award for Dr. Wright is Dr. A. Jackson Stenner who will present a paper co-authored with Dr. Wright titled
Readability, Reading Ability, and Comprehension.

Workshops

Building Computer-Based Performance Tests
Did you ever wonder what it would take to build a computer based performance exam which could be delivered to a national/international audience. This workshop will discuss the ins and outs of designing, building, and delivering computer based performance exams. It will present this from the perspective of several test publishers who have gone through the often agonizing and frustrating process of creating such exams. Topics included will be Job Analyses, task building, delivery alternatives (simulation, live application), and results reporting.

Computerized Adaptive Testing
What is computerized adaptive testing? Who is it appropriate for? How does one develop and maintain an appropriate item bank for CAT? How does one convert a paper and pencil test to an adaptive test? This all day session will fully introduce the attendees to the world of computerized adaptive testing and how it can be utilized in assessment. Time will be available to address issues pertinent to specific attendees.
Converting from Paper-Based Tests to Computer-Based Tests
Did you ever wonder what it would take to build a computer-based examination which could be delivered to a national/international audience. This workshop will discuss the ins and outs of designing, building, and delivering computer based performance exams. Topics included will be determining readiness for computer-based transition, marketing the conversion to candidates and employers, developing policies and procedures, delivery alternatives, and candidate-related issues.

Designing Criterion-Referenced Tests
Increasing competition, alignment, fluctuating job descriptions, and ever-changing technologies, require organizations to determine the competence of their members. All designers of instruction acknowledge the need for appropriate testing strategies; and non-instructional interventions also frequently require the assessment of human competence whether in the interest of needs assessment, the formation of effective work teams, or as a part of a service or product the organization sells. However, most professional development courses or workshops have typically avoided a complete exploration and resolution of the issues and procedures in testing in order to concentrate on instructional design or other human performance interventions.

The workshop is designed to answer 13 major questions in the Criterion Reference Test (CRT) development process. Presenters will alternate between the technical issues and the legal issues for each point on the agenda so the time estimates include both technical guidelines and legal ones.

TOPIC OUTLINE:
How do I decide whether to create a CRT or a norm-referenced test?
What does a test score mean?
What are the basic methods of test interpretation?
How will a content hierarchy help me plan a test?
What are the five types of tests most commonly used to assess human performance?
How many items should there be on a test?
What are the six most common types of items for cognitive tests? What are their strengths and weaknesses?
What are the four most common types of rating scales for performance tests? Which ones should be used?
What are the two most practical procedures for determining the cut-off (master/non-master) score?
How do I establish reliability for my test?
How do I establish the validity of my test?
What are the three basic item analysis techniques and how can they help?
What are the basic legal requirements for defending a test?

Evaluating Item Types for Computer-Based Testing

Evaluate existing item types - both paper and pencil as well as alternative formats for use in computer-based delivery.

Introduction to Item Response Theory
Applications of item response theory (IRT) models to computer-based credentialing examinations have been increasing in number. However, IRT models that can be used with dichotomously-scored and polytomously-scored exam items remain something of a mystery to many practitioners working in the credentialing field. The purposes of this six-hour workshop are (1) to introduce basic IRT models (e.g., logistic models, the graded response model) and concepts (e.g., item and ability parameter invariance, unidimensionality, test information), and consider the strengths and weaknesses of IRT and classical models and procedures, (2) to describe popular approaches for estimating IRT model parameters, assessing model fit, and locating IRT software for popular IRT models, (3) to provide details for how IRT models can be used to construct credentialing exams (e.g., item selection, new test designs), to identify differentially functioning exam items, and to equate scores across multiple forms of an exam, and (4) to address important and frequently asked questions about IRT (e.g., how big do sample sizes need to be to use various IRT models? What do practitioners need to know to apply IRT models successfully?) Follow-up reading material will be suggested.

Item Analysis Tools: An Overview of Currently Available Tools
Cliff Donath, President, The Donath Group
"Using Statistica for Classical Item Analysis"

James Olsen, Ph.D., Chief Scientist and Vice President, Alpine Media Corporation
"Using BILOG-MG for Analyzing Multiple Group and Test Form Data"

John Poggio, Ph.D., Dir., Center for Educational Testing and Evaluation, University of Kansas
"Using SIMSTAT for Classical Item Analysis"

John C. Kalohn, Director of the Placement Programs Department , ACT
"Using Excel to Graphically Represent Item Level Bilog Output"

Betty Bergstrom, Ph.D., Vice President Solutions Integration, CAT*ASI
"Using Winsteps for Analyzing JTA data"

Deb Schnipke, Ph.D., CAT Senior Program Manager and Psychometrician, Computer Adaptive Technologies, Inc.
"Using SPSS for Classical Item Analysis"

Michael Yoes, Psychometrician, NCS Pearson
"Using XCalbre for Basic IRT analysis"

Managing the News Media: Taking Control of Your Media Coverage
More and more, the testing industry is in the news on a regular basis. And that means organizations need to take an active role in managing the outcome of media coverage that affects them. In this workshop, a veteran public relations professional will provide information on how to manage news coverage, shape key messages, develop relationships with news organizations and rebuild a reputation damaged by negative news coverage.

The session will also include tactics to build a proactive media relations program that can bring positive visibility to an organization and help support its business objectives, from helping sales to recruiting employees.

Practical Concerns for Test Security
If you need to deliver secure exams, to the right candidates, at the right time, then this session is for you. Designed to identify the security concerns of both small and large testing programs, the half-day session will take you through detailed (and proven) security processes used successfully by a number of high profile, international testing organizations.

The session will present:

• Best processes, procedures, and actions for ensuring exam security
• Case studies (paper and pencil as well as CBT)
• A systematic look at the threats and vulnerabilities that can occur with "non-secure" exams
• The cost of having an exam compromised
• Key ingredients of a secure test program
• A "5 point" look at security, including: (1) security from a legal standpoint; (2) responsible care and control of information and test files; (3) maintaining exam integrity; (4) ensuring security at the test delivery site; (5) global considerations
• Sample security policy statements

You will leave with:

• A tool that allows you to benchmark the "security" of your certification program
• An understanding of your program's security strengths and weaknesses
• An understanding of key actions to undertake that will "raise the bar" on your program's security
• A clear understanding of what it will cost (time and money) to further enhance the security of your program
• A wide range of practical ideas and actions security safeguards and processes that will reduce threats to your program

The knowledge to build and execute - or enhance - a security program that's right for your organization's environment and budget

Test Development from Scratch
This session will provide a complete description and hands-on practice with the test development process from job analysis though examination production. Included will be such topics as: job analysis, test specifications, item writing, item banking, item review, test form assembly, and examination production. The morning session will a formal presentation of the principles and procedures needed to understand the test development process and its significance in developing an examination that is psychometrically sound. The afternoon session will be a hands-on training session in writing quality items. Topics will include: types of items, matching items to match the test specifications, identifying item flaws, and editing hints. Participants may attend 1/2 day sessions - each session designed as an independent module.

Testing In-House - A 'How to' Guide
This seminar is directed at test sponsors interested in exploring taking some or all of their testing in house. You will learn how to construct, build, publish, deliver, score, and report your own exams. We will also look at options for controlling the administration of examinations in-house remotely or at a public test site or via the internet.

Focus on Certification/Licensing

Practical Lessons Learned in Computer-Based Testing
We are in the midst of a paradigm shift similar to the one that occurred in the early 20th century with the move from individually administered examinations to mass testing. In the course of making the change to computer-based testing, testing professionals, examination sponsors, candidates, and computer-based testing vendors have learned some valuable lessons. In this discussion, the focus will be on the two areas: test development and test delivery/administration. We will cover the practical lessons learned in the course of the paradigm shift represented by the conversion from paper-and-pencil to computer-based testing.

Setting Standards for Web- and Computer-Based Testing
Paul Jones, Ph.D., Galton Technologies
"The Borderline Survey Standard Setting Method: Efficient and Effective"

The borderline group method uses the test performance of actual candidates to set cut scores, eliminating the need for SMEs to gather in a special meeting to review the exam. Traditionally, the borderline group method has used judges to classify candidates as being either Above Borderline, Borderline, or Below Borderline qualified, based either on the judge’s own knowledge of the candidates or on supervisor ratings. Over the past year Galton has been actively exploring the use of candidate surveys to make these classifications, thereby making the borderline group method practical for global certification programs. Here we discuss what experience has taught us about the borderline survey method, including test definition, survey development and review, and survey analysis and decision-making. Our goal is to show attendees how borderline survey standard setting can be folded into a quality test development process with relatively little incremental effort. We conclude that borderline survey standard setting is both efficient and effective.

Richard Luecht, Ph.D., University of North Carolina, Greensboro
In certification and licensure testing, standard setting is a judgmental process of establishing a policy and associated decision rules about competency. Panels of judges - usually experts in the field - are taken through a particular set of procedures which ideally lead to a specification of the requisite level(s) of knowledge, skills and abilities that characterize a competent professional. Many well-known procedures have been developed for multiple-choice and other simple-response item formats (e.g., modified Angoff, bookmark). In contrast, standard setting has been less well researched for computer-based tests that employ complex simulation or work sample tasks. In practice, standard setting for complex computer-based tests can be very costly and time-consuming. Technical complications can also range from information overload during the standard setting process (i.e., expecting the judges to efficiently review and synthesize sometimes enormous amounts of complex response or performance data) to integration of the standard setting process with requisite scoring and scaling procedures used for the test (i.e., validating the standard setting process with respect to the scoring and scaling of the test). This paper discusses some of the key issues in setting standards for complex computer-based tests and presents some possible solutions. Applied examples are taken from the Microsoft Certified Systems Engineer examination program.

Stephen Sireci, Ph.D., University of Massachusetts, Amherst
Setting passing scores is one of the most important and difficult tasks in licensure and certification testing. For valid pass/fail decisions to be made, the standard setting process must consider the qualifications required for the credential, the rigor of the exam, and expected levels of performance on the exam by qualified and not yet qualified candidates. Numerous standard setting methods exist in the literature, all of which can be used to produce reasonable and defensible passing scores, if implemented properly. However, most are time consuming to implement and are rarely supported by validity evidence. Furthermore, when a test is computerized-adaptive, many of these methods are impractical or inapplicable. In this paper, a review of traditional standard setting methods is provided, standard setting issues specific to computerized-adaptive tests (CATs) are discussed, and some newer methods for setting standards on CATs are described. To balance efficiency and validity in setting standards on CATs and other computer-based tests, newer methods such as the Wainer, item sorting, and direct consensus methods appear to be promising, and gathering validity evidence during the standard setting study is recommended. A summary of techniques for speeding up the standard setting process while maintaining the validity of the passing standard is provided.

What Types of Validity Evidence Should You Have in Your Hip Pocket When the Lawyers Come To Call?
Validity evidence is indirect proof of the quality and legal defensibility of a testing program. Training and testing managers are responsible for documenting, gathering, and maintaining several types of validity evidence throughout the test development cycle. This session will describe the critical types of validity evidence to be collected and kept on file to help protect an organization from potential legal challenges. Dr. Naylor will discuss and illustrate the critical process steps to build in validity elements using the Content-Process Model as a guide. He will highlight the role of seminal legal cases in occupational analysis and testing such as Griggs v. Duke Power and Albemarle v. Moody. Dr. Naylor will also describe a recent case in which a court trial was avoided through the presentation of sufficient validity evidence. Dr Zane will present a list of documents, records, and other validity evidence he developed to protect his company from informal test challenges and formal test litigation. Dr. Zane required each of his test developers to maintain current files containing these critical records. During this portion of the presentation, he will highlight the minimum documentation requirements for each type of validity evidence along with his preferred requirements. Dr. Naylor and Dr. Zane will provide handouts containing a bibliography of important court cases, legal documents, and laws. They will also provide copies of the validity evidence documentation list. A copy of the presentation will be available from ATP.

Focus on Clinical

Neuropsychological and Other Performance Monitoring via Networks
Dennis Reeves, Ph.D., Naval Hospital Camp, Pendleton, CA
Robert Kane, Ph.D., Baltimore VA
The computer has added new dimensions to the assessment of cognition. Advantages provided by automated assessment include accurate presentation of test stimuli and scoring, an increased range of tasks, and the ability to obtain precise measures of response speed and variability. Despite the relative newness of the personal computer, automated testing has already developed a rich history. Nevertheless, we are just beginning to tap the potential of the computer for the assessment of cognition. In addition to providing a quick review of the state of the art emphasizing DoD contributions, this presentation will focus on newer implementations of automated assessment including the incorporation of tasks assessing divided attention and resource management, telemedicine, and portable hand-held assessment devices.

David Erlanger, Ph.D., HeadMinder, Inc.
"An Internet Platform for Test Development, Administration, Scoring and Reporting"
HeadMinder, Inc. is a three-year-old company that develops testing technologies to facilitate the use of the Internet by
test publishers, clinicians, and researchers. This seminar will detail the steps required for adapting, developing and maintaining tests for use on the Internet. A discussion of statistical and scoring procedures uniquely suited to the Web environment will be included. Integration of Palm-based protocols will be addressed.

Objective Test Assessment on the Internet
David Pfenniger, Ph.D., Performance Assessment Network
Remote Clinical Assessment: An Introduction: Remote clinical assessment (RCA; a.k.a. monitoring assessment) is the use of communication technology by clinicians and patients for diagnostic interaction. RCA may play a significant role in an overall clinical care management programs, and also may be deployed for research purposes, for example, in pharmaceutical clinical trial protocols and epidemiological or disease outcome studies.

The rationale for RCA is compelling. It is consistent with: 1) Overall health care trends toward home-based clinical care and outreach to patients; 2) Increasing accessibility to clinical care in rural populations or other areas underserved by clinical specialists or facilities; 3) Leveraging technology to achieve both enhanced quality and decreased costs of service delivery; 4) Existing and emergent communication technology and increasing utilization of such platforms by both clinicians and patients.

Internet-based assessment and testing or e-testing is an exemplar of RCA. Clinical assessment inventories are being administered to patients online. In many of these sessions the patient is taking the test online from home or other location remote from the clinician. Demonstration projects are underway delivering e-testing to patients on touch-screenweb pads and reporting data to physician hand-held devices.

The test and measurement industry is fortunate to have a product type that is readily amenable to digitalization and remote communication transfer with preserved validity. It is predicted that RCA will become a primary feature of clinical assessment practice within two years.

Leigh Jerome, University of Hawaii
"Remote Clinical Assessment: New Concepts and Emergent Technologies"

Converging advancements in health care, science, and telecommunications are creating new challenges and opportunities for the testing community. The Internet and the synthesis of science and technology have yielded new solutions for diagnosis and assessment including digital imaging, hypertext, video-conferencing, biosensors, robotics and simulation.
Wireless and high-bandwidth capabilities will provide even greater sophistication and portability. As these technologies evolve they will generate new diagnostic pathways toward both expanded and integrated services. This presentation will explore innovations and emerging trends and their potential impact on remote clinical assessment.

Larry Weiss, Ph.D., The Psychological Corporation
"Equivalency analysis of paper-pencil and web-based versions of the Beck Depression Inventory"

According to the Standards for Educational and Psychological Testing (APA/AERA; 1999) paper and pencil tests adapted for computer administration should be accompanied by studies evaluating the equivalency of the two modes of administration. If equivalency cannot be demonstrated, then it is inappropriate to utilize scores and norms developed based on traditional administration with the computer administered version.

This presentation reviews the extant literature on equivalency studies and summarizes the major findings from this body of research with regard to several different classes of assessments (personality tests, clerical ability tests, etc.). We will then present an example of a web-based administration of the Beck Depression Inventory, and discuss design issues that influence equivalency. Some of the design issues to be discussed include allowing the respondent to skip items, to reverse and review previous items, and to change answers.

Training and Evaluation of Clinical Skills Using Interactive Multimedia
Technology continues to provide publishers with alternative methods to deliver content and assess individuals - some more costly than others. When does it make sense to use multimedia? This session will demonstrate an instance where interactive multimedia was the more effective method for providing training and assessment of skill acquisition.

This session will demonstrate the modules used to develop the skills needed to conduct and score a semi-structured clinical interview. The benefits of audio and video presentation and immediate feedback on performance will be illustrated. Included will be a demonstration of modeling of the target behaviors and ongoing assessment and tracking of the acquisition of those skills. The culminating item type is video based and uses real-life interviews.

Focus on Education

Educational Assessment by Computer
"Implementing adaptive testing in educational settings" -- G. Gage Kingsbury

Public schools lack the time, money, and measurement expertise necessary to implement an adaptive test. Even if they could implement CAT somehow, it would be politically unacceptable, and it would be impossible to interpret the results for teachers, students, and parents. This presentation describes the adventures of a group of school districts who don't let a small matter like impossibility stand in their way. This includes a discussion of problems concerning hardware, software, logistics, and reporting. It also includes specific solutions to these problems which have been identified and implemented in particular schools and school districts. Finally, it describes the ways that CAT is causing school districts to rethink their educational practices.

Electronic Testing in the K-12 Setting
This session will explore the critical issues among quality content, flexible technology, equivalency to paper-and-pencilversions, and immediate feedback in the delivery of electronic testing in the K-12 educational testing arena.

The Future of Educational Assessment by Computer from the Publisher's Perspective
A panel of four publishers will discussion the future of educational assessment:
Dr. Sykes will present the design, implementation, and results of an on-line validation study of a test of cognitive skills.

Dr. Mott will talk about the specific pros and cons of the high-stakes computer-based state tests in Virginia. He will present the pros and cons of the technological, societal, political, legal, and educational issues.

Margie Jorgensen will address the issues from the business perspective.

Molly Zebrowski will take a marketing perspective on future possibilities for computerized-educational assessment.

Focus on Industrial/Organizational

Computer-Based Testing Practices in the Workplace
This session will benchmark CBT practices in the workplace.

Legal Issues
This session will focus on the legal issues surrounding CBT that are specific to Industrial/Organizational division. We will discuss potential adverse impact cases due to differences in experience/familiarity within computer usage across ethnic groups for occupations where computer skills is not a BFOQ. We will also address ADA's reasonable accommodations -- what impact will play in remote web testing.

Transition to Technology: Design and Application Issues with Employment Tests
As technology continues to improve and become less expensive, well-established employment tests originally developed and validated in paper-and-pencil format are migrating to computer platforms (e.g., CD-ROM, Web). With this transition to technology, a number of practical and psychometric issues emerge which should be addressed, particularly in the case of time-sensitive assessments. This session will describe important considerations, methods, and lessons learned in the computerization and application of two widely used cognitive ability test series.

Cross Divisional

A Comparative Analysis Between IT and Non-IT Programs
When developing certification programs, regardless of the setting, a shared outcome is an assessment tool which provides a valid and reliable measurement of competency. While this outcome is similar, the means by which this outcome is achieved differs when you compare IT programs with on-IT programs. Topics will include motivation for having a program, test development issues, marketing issues, standards, and policies governing certification.

Anti-Piracy Campaign: An Update from the IT Security Council
Early in 2001, companies within the IT industry formed an IT Security Council (ITCSC) for the purpose of bringing together key individuals within the certification and testing arena to discuss procedures, processes, actions and ownership. The goal of the group is to identify and share best practices within the industry, establish new and better practices where they might be employed and provide leadership to the rest of the industry.

Assuring Excellence in Test Development, Delivery and Analysis
There is a professional need to assure excellence in developing, delivering, and reporting computer based as well as traditional tests. This session presents alternative methods that are designed to assess the quality and value of the procedures used in designing, developing, and delivering tests. Various standards, including the Standards for Educational and Psychological Testing, Computer Based Testing Guidelines,American National Standard ISO 9001:2000
Quality Management Systems--Requirements, ISO/IEC 17024 Accreditation for Personnel Certification Organizations, and the NCCA Standards for the Accreditation of Certification Programs will be compared. Advantages and disadvantages of internal and external audits will be described. An illustration of a process for conducting external audits using standards developed by the Buros Institute for Assessment Consultation and Outreach will be provided.

ATP Computer-Based Testing Guidelines: Application across Multiple Disciplines
Computers are now standard and pervasive tools that significantly impact our daily lives. In testing too, computers have changed the ways in which tests are developed and administered. In the face of the rapid growth of computer-based testing, the Association of Test Publishers (ATP) has sponsored the development of guidelines to help assure high measurement quality of computer-based tests and to provide direction for the principles and procedures used for developing and administering these tests. These guidelines are intended to supplement, extend, and elaborate on the Standards for Educational and Psychological Testing (1999, "Standards") as they apply to computer-based testing (CBT).
The Guidelines are written to reflect guidelines for best practices in computerized testing. The Guidelines are divided into two sections.

Part 1 provides some background and rationale for the different parts of test design and development for those involved in computer-based testing who have little training in measurement. Those readers are referred to the Joint Standards or a measurement textbook for elaboration of the topics presented here. Recall that these Guidelines are intended to supplement, extend, and elaborate on the Standards for Educational and Psychological Testing (1999, "Standards") as they apply to computer-based testing (CBT).

Part 2 describes the Guidelines for development and use of computer-based tests. The Guidelines cover areas of test development and uses that are the responsibilities of the test sponsors, test development organization, test delivery organization, and score users. (Note that these are not necessarily different organizations and some organizations may consist of only one or two individuals). This section is divided into six chapters, which cover planning and test design, test development, test administration, scoring and score reporting, psychometric analysis, and stakeholder communications

The purpose of this session is to provide the following: 1) Historical perspective of the Guidelines, 2) Updates to the Guidelines since February, 2001 and 3) Application of the Guidelines to various disciplines.

Audience Definition: Alternative JTA Processes
Traditional Job Task Analysis techniques often fall short of developing accurate knowledge and skills domains for certification exams. They tend to consume a large amount of time, are prone to observer bias, lack true collaboration among participants, and often do not provide the complete results needed to build valid testing vehicles. The Job Task Analysis Workshop, on the other hand, allows job incumbents and subject matter experts to validate the job role definition, identify tasks performed on the job, map “best practices” processes, identify critical tasks and skill gaps, and create full terminal and enabling objectives (constituting the knowledge and skills domain of the profession) - all in only four days! This session will provide an overview of the challenges faced in performing a job task analysis, and describe how to hold a Complete Job Task Analysis Workshop in only four days.

Best Practices in Item Banking for Computer-Based Testing
Configuring and maintaining an item bank for computer based testing is both science and art. What should be stored, classified, and tagged to make best use of items for delivery in a computer based environment? This session will draw on examples in educational testing and certification to illustrate best practices and common pitfalls.

Bias Issues with Computer-Based Testing
We will review the literature of studies conducted to insure the absence of bias (race, culture, language, SES, etc.) in computerized testing. We will also look at studies that compare performance estimates obtained by computer based testing and computerized adaptive testing versus those obtained from paper and pencil tests. In some cases computerized testing actually results in higher test performance than when individuals take their tests on paper. These findings are of particular importance in test settings which are subject to strict legal review such as school testing programs and in industrial-organizational psychology.

Case Studies: Workplace Simulations
Assessing interpersonal skills by paper-and-pencil instruments has proven difficult; such assessments have either been found to be unreliable or highly correlated with cognitive ability. As an alternative, video clips depicting simulations of interpersonal situations can be presented via computer. When a clip reaches a critical juncture, the video stops and a multiple-choice question is presented that asks "What would you do in this situation?" This presentation will focus on the design and development of computer administered video assessments of interpersonal skills such as leadership and teamwork that are important in the workplace. Topics will include scoring, research results, applications, and issues requiring special consideration.

Case Study: Computer-Based Testing For Driver Examinations
The International Computer Driving License (ICDL) is the world's largest and most accepted program for testing and certifying end-user computing skills and essential IT knowledge. The ICDL is awarded upon completion of 7 Modules ranging from essential concepts of IT, word-processing and spreadsheets to using e-mail and the Internet. The ICDL is a certificate of knowledge and proven competence based on a single agreed syllabus that is independent of any hardware or software products. Today employers, government officials and educators in more than 55 countries use ICDL as the worldwide standard in credentialing competent and skilled computing end-users. Participation levels in the ICDL program on a worldwide level are now approaching 2 million candidates. In the United States, ICDL is delivered by ICDL-US
(http://www.icdlus.com), a wholly-owned subsidiary of the European Computer Driving License Foundation (ECDL-F). The ECDL-F is the not-for-profit global governing body of ICDL.

Computer-Based Testing and Instruction in the Year 2005
Shawn Bird, Ph.D.; Exam Solutions
James Olsen, Ph.D., Alpine Media Corporation
The Transformation of Instruction & Assessment -- 10 Ideas that could Revolutionize the Industry

The desktop computer, and now the Internet, have introduced learners and examinees, as well as authors and publishers of all types of content to a level of power and flexibility previously unimagined by practitioners of chalkboard and paper-and-pencil methods. With new, increasingly powerful computers and portable devices, software, networking and Internetworking technologies emerging daily, the pace of innovation is growing at a tremendous rate. Accompanying this have been unprecedented developments in the theories, principles, and technologies of user-centric personalization and customization. While history shows us that innovation has only slowly crept into the now well-established vendor-centric training and test delivery markets, it is widely believed that the barriers to entry and innovation will soon break down. This presentation discusses the future of training, testing, and assessment technologies as well as the likely future of the training, testing and assessment business as it is buffeted by unsatisfied demand for quality, innovation, and personalization, as well as a fundamentally different economics of delivery. The topics covered in this presentation include trends and emerging technologies, the role of standards, the commoditization of training and testing services, the imperative to achieve interoperability across vendor tools and systems, and a new orientation that puts the learner and the examinee at the center of the entire process.

M. David Merrill, Utah State University
What Happened to Valid Kinds of Measurement?

The author has had an opportunity to review many recent on-line courses. In most of the courses thus reviewed the only measurement is a set of remember-what-you-were-told questions usually in multiple-choice or short answer format. To make sure the objectives are consistent with these forms of measurement they usually contain words like define, describe, list. In almost all cases, however, these are not the intended outcomes of the instruction. The real objectives, as obvious from the content, are usually to be able to solve some problem or carry out some task. In addition to remembering information solving problems almost always involves three additional types of objectives: how-to carry out the steps, what-happens as a result of carrying out the steps, and what kinds of decisions are necessary to select the correct procedure for different situations. Answering remember type questions does not constitute valid measurement for these objectives. In this short paper valid measurement for how-to, what-happens and kinds-of objectives will be described and illustrated.

Andrew Gibbons, Ph.D., Utah State University
Integrating Testing and Instruction: The Questions of Scope and Resolution

The advance of any technology beyond a certain level of precision and controllability depends on (1) accurately repeatedly measuring, (2) measuring the right things, and (3) measuring and analyzing within a time frame consistent with synchronous process correction. High-risk, high-payoff areas of training have tended to solve measurement problems by taking multiple measurements frequently in highly realistic performance settings. Such measurements often include the gathering and recording of more voluminous data than can actually be used. In such cases of situated measurement, the line between instruction and measurement often blurs or vanishes altogether.

Progress in instructional designs, and in particular in the integration of instruction and testing seem to be blocked by questions regarding the nature of the abstract measured construct and schemes for partitioning the construct and giving it form for measurement purposes. Work Models are practical performance constructs used in design to enable dynamic measurement by progressively and systematically defining growth in scope and resolution during measurement. The Work Model structure is particularly useful because it supports the integration of instruction and measurement, reversing traditional designer views about the priority and necessary ordering of the two.

Cross Pollination - Lessons Learned From Computer-Based Testing in Different Markets
In this session, we will have a panel consisting of representatives from different industries. The discussion will focus on key issues and there will be a sharing of experiences relating to computer-based testing. The panel will consist of ATP
Division Chairs and will be moderated by G Harris.

Detecting Candidate Misconduct
The benefits of CBT are many, including test security. This session will highlight many of the advantages surrounding CBT including some of the prevention steps used to deter individuals from gaining an unfair advantage, a series of detection methods used to identify potential problems and remediation efforts once a substandard situation is confirmed. Some of the high-tech devices that can be used to try and obtain an unfair advantage in the testing room will be discussed.

Evaluating Alternative Item Types in Computer-Based Testing
Cyndy Fitzgerald, Ph.D., Microsoft Corporation
Moderator

Cynthia Parshall, University of South Florida
This presentation will first address the value of innovative item types from a measurement perspective (such as reducing guessing and expanding the types of cognitive skills assessed). Next, the task complexity of innovative items will be discussed, particularly in terms of item writing and test development efforts. Then, a classification scheme for innovative item types will be introduced. This classification scheme has five dimensions of innovation. They are: item format, response action, media inclusion, level of interactivity, and scoring. Examples of innovative items will be offered, to illustrate each of these dimensions.

Deborah Kerman, Ph.D., Microsoft Corporation
The Microsoft Certification Group is about to embark on a study of the programmatic value of various item types. The goals and motivation of the study, as well as the planned methodology, will be discussed in this session.

Brian Bontempo, Ph.D. Microsoft Corporation
As the number of testing organizations planning and developing innovative items increases, the need for a framework by which to evaluate innovative items has surfaced. This session will provide a framework that focuses on the psychometric, cognitive, financial, and operational aspects of evaluating innovative items. In addition, the results of one phase of Microsoft’s item type evaluation study will be presented. These results pertain to the psychometric performance of Drag and Drop, Hot Area, and Choose N items.

Key Problems and Practical Solutions in Computer-Based Testing Programs
This session identifies a series of key problems and practical solutions for building computer-based certification programs. Using a case-study presentation, this session highlights key problems and provides practical solutions to each of these identified problems.

Making Tests Accessible to Test Takers with Disabilities: New Challenges and New Opportunities Under the ADA
Each year, hundreds of test takers with disabilities request accommodations under the Americans with Disabilities Act (ADA). This session will discuss the steps that ETS is taking towards accommodating the unique needs of individuals with learning, physical or sensory impairments in a computer based testing environment. Specific suggestions will be made regarding test item construction in light of the principles of universal design.

Managing Large Scale Computer-Based Testing Programs
This session addresses critical issues and potential solutions involved in managing large scale computer-based testing programs. Important issues that will be discussed include: global availability, multi-vendor test delivery, test standardization, test localization and adaptation, stakeholder communications, test revisions, and security issues.

Online Assessments as an Organizational Development Tool: Demonstration and Application Overview
Delivering computer-based assessments via the Internet can be a complex anda resource consuming task for publishers. This session will tell the story of what the publisher of the Myers-Briggs Type Indicator(R) tool went through in developing on online computer-based assessment site. We will golive on the Internet to show what an actual online assessment experience is like including logging on, setting up your own customized assessment account, ordering administrations over the Internet, assessment administration, interpretation guidelines, and report delivery options.

Online Item and Test Development
When resources are geographically spread out, bringing together groups of people to participate in the test development process is sometimes infeasible. New software tools make it possible to perform many test development activities over the Internet, including test blueprinting, item writing, item review, test construction, test review, beta/field testing, item statistical analysis, and final-forms construction, review, and testing. This session will look at some of the methods for performing these activities on-line, rather than in-person.

Opportunities and Challenges of Computer-Based Testing
This session will provide a basis for assisting your organization in managing the testing needs of its customers, judging the support needs of both personnel and exam review committees, in dealing with technology transitions, and preparing for work that needs to be done both before the decision can be made and after product delivery, the process of transferring from paper and pencil testing to computer-based testing and the advantages and difficulties found with each.

Items to take away with you will include: a record of problems that testing candidates have encountered at the test centers will be presented along with a review of candidate evaluations of the computer-based testing experience over a three year period and sample forms tailor the RFP instrument to gather bids and generate accurate projected costs and timelines to aid in the decision process.

Psychometric Issues When Using Multiple Test Distribution Channels
Test publishers often use more that one test distribution channel for testing. Whether this is done to gain greater market penetration to get better vendor responsiveness, or to obtain costs breaks, the result is a number of psychometric issues which must be addressed. Psychometric issues such as item handoffs, data integrity, test presentation, statistical consistency, versioning, and consistent exam environments may cause inconsistent results. This panel session will present the issues which need to be addressed under these circumstances as well as our experiences using multiple distribution vendors.

Scoring Essays via Computer-Based Testing
This session will examine automated essay scoring (AES) in the context of web-based assessments. AES can provide both a formative and summative evaluation of content and general writing ability. Four major AES engines will be discussed with a focus on similarities and differences along with a general explanation of how the technology works. The session will conclude with a discussion of how AES can be applied to a wider diversity of documents such as those contained in electronic portfolios.

The Financial Model for Computer-Based Testing
This session defines the key cost elements, including expected averages and variances, involved in the planning, development, delivery, administration, and maintenance of a successful computer-based testing program. The intent of this session is to provide practical cost analysis procedures and guidelines for organizations that are considering transitioning from paper-based testing to computer-based testing, from computer-based testing to computerized adaptive testing, or from computer-based testing to computerized performance based testing. Financial and political costs will be considered. Primary focus will be on small and medium size organizations with national and/or international delivery needs.

The Key to Developing Computer-Based Tests - Automated Test Assembly Procedures
Krista Breithaupt, Ph.D., American Institute of CPA's
This presentation will provide a high-level description of a design proposed for the computerized CPA licensing examination developed at the American Institute of Certified Public Accountants, and the implementation steps required to create and study prototypes. This computer-based test (CBT) will allow sequential (adaptive) testlets to be presented to candidates, followed by testlets that provide complex performance simulations in each of four sections of the CPA examination. This application of the sequential testlet model is one form of computerized sequential adaptive testing (Luecht & Nungester, 1998).

Sequential testlets are under consideration as a potential solution for competing psychometric, content and other constraints encountered in test assembly (Luecht, 1998). In addition, this CBT design might lead to improved precision in total scores, more efficient use of the item pool, and tailors to some extent the test experience for examinees of different ability levels. These expectations are the subject of ongoing research for the AICPA undertaken by psychometricians and graduate students at the University of Twente, University of North Carolina-Greensboro, University of Massachusetts-Amherst and University of Illinois-Urbana/Champaign.
Some practical steps required to evaluate and prepare for this mode of CBT in a paper and pencil-based testing program will be described. Topics will include accelerated targeted item writing, item coding for inventory control, thumbnail prototype testlet construction, and systems and co-ordination requirements to facilitate independent delivery of pre-test prototypes in universities and participating accounting firms.

Wim J. van der Linden, University of Twente, Netherlands
Historically, the practice of test construction in education and psychology has shown developments from: (1) the construction of standardized tests to the process of assembling tests from item banks tailored to the user’s specifications; (2) from the use of intuitive test construction rules to the application of model-based algorithms; and (3) from manual sorting of items on index cards to item selection by a computerized test assembly system.
In computerized test assembly, a combination of items is selected from an item bank that is optimal in a statistical sense and at the same time satisfies a set of content specifications for the test. Formally, the problem has the structure of a constrained combinatorial optimization problem in which an objective function is maximized subject to a set of constraints, both typically modeled using 0-1 decision variables for the inclusion of the items in the test. The foundation of this approach was laid in Birnbaum’s (1968) pioneering work on IRT-based test assembly. Currently, a large variety of test assembly problems have been modeled as constrained combinatorial optimization problems and various algorithms and heuristics are available to solve them.
In the presentation, we will focus on the role of the test assembly model that the algorithm has to solve to select an optimal test. In particular, it will be emphasized that a set of tests assembled by a computer can be given any desired feature by including appropriate constraints in the model. This basic principle will be demonstrated for a large variety of test features, such as their content composition, information structure, item-set structure, reliability, validity, observed-score distribution, level of speededness, degree of parallelness, and item-exposure rates. The empirical examples include applications to the problem of assembling a single test form, multiple test forms, tests measuring multidimensional abilities, and adaptive tests.

Richard Luecht, Ph.D., University of North Carolina, Chapel Hill
"An Automated Test Assembly Heuristic for Multistage Computer-Adaptive Tests with Complex Computer-Based Performance Tests"
Automated test assembly (ATA) has evolved tremendously over the past decade. Powerful ATA procedures and software have been developed that use linear programming item selection algorithms, network flow algorithms, genetic algorithms, or item selection heuristics that are based on the concept of a greedy algorithm. Unfortunately, specific applications of these most of these algorithms to computer-based tests are not always straightforward. Applications are especially difficult to implement when the tests employ multistage adaptive testing, or, when they include complex item types (e.g., problem-based item sets or computerized performance simulations). This paper describes a generalized ATA heuristic and some strategies for simultaneously building multistage, adaptive tests that incorporate testlets, problem-based item sets and complex performance types. The paper incorporates applications to large-scale test production - i.e., building many test forms over time. Specific examples from two large-scale computer-based testing programs are included to illustrate applications.

The Virtual Classroom: Using Web Conferencing for Certification Prep
Certification candidates are continuously seeking ways to prepare for certification exams. One way to do this is through interactive, live, computer conferencing. This session is a case study describing how Hewlett-Packard delivers certification prep online workshops as part of an integrated education and certification model. Topics covered include: Virtual Classroom - what it is, how it works; Workshop design, feedback from candidates, and other applications for the virtual classroom.

Transitioning From Computer-Based to Web-Based Testing: Development and Delivery Issues
Shawn Bird, Ph.D., Exam Solutions
"Overcoming the Obstacles in Transitioning from Computer to Internet-based Testing" The Internet is slowly making a transition from an experimental medium to a viable channel for delivering online training, testing, and assessment. A number of mature Internet-based testing and assessment solutions are now available essentially off-the-shelf. And with Internet technologies improving almost daily, the focus of testing practitioners is beginning to change from "getting online" to deploying and supporting sophisticated, high-quality online testing and assessment products. This presentation reviews and presents solutions to many of the critical development and delivery issues that drive the transition from traditional computer-based exams and assessments to online equivalents. The presenters demonstrate how the technical barriers to sophisticated online testing have all but been overcome, and why the critical challenges ahead lie in maturing the business side of testing. The topics addressed in this presentation include content representation, key Internet technologies, the role of standards, the comparability of online and offline instruments, the transition process and its management, and the changing face of the testing business.

Alison Indrisano, Prometric, a Thomson Learning business
"The Role of the Internet in the Learning Cycle Listening to the Voice of the Customer"
The Information Age has created the need to make more content and opportunities for learning and measurement available so that people of all ages can continuously develop as knowledge workers. In evaluating the content choices available today and in the near future, an analysis of opportunities and limitations of technology-based learning and measurement solutions is required. The success of the Internet as a delivery tool for measuring and validating learning will be largely dependent on its match with the needs of the "test taker", "test maker" and "test owner". Hybrid models, which utilize a blend of technologies as building blocks for a complete learning and measurement system, are evolving. New stakeholders are creating a need for new content and testing types to provide these blended solutions. Alison Indrisano, Senior Vice President of Client Services and Support at Prometric, a Thomson Learning business, will highlight the opportunities for hybrid models of learning and testing that are present today and likely to evolve in the near future. She will share observations resulting from needs assessments of several Prometric customers worldwide.
Alison will discuss the elements of the customer decision-making process as they evaluate and implement Internet-based testing models and evolve toward blended models of learning and testing. Contrasting points of view will
be explored through a discussion of the needs of other clients whose requirements cannot be met by the Internet model today.
Through this presentation, a spectrum of "test development and delivery building blocks" that are a part of the comprehensive suite of services provided by Prometric will be outlined.

Anthony Zara, Ph.D., NCS Pearson
"The Appropriateness Of Web-Based Testing: Applications And Pitfalls"
Computer-based testing in a variety of applications has exploded in the United States over the past decade. Large numbers of testing stakeholders are using the power of computers to enhance their testing programs. These enhancements (as compared to traditional paper-and-pencil testing) can include modern measurement models (e.g., CAT), technology-rich item types (e.g., including video, audio), increased availability (e.g., daily testing), and enhanced security. As the Internet evolves in sophistication and usability becoming a more ubiquitous medium of communications, it becomes increasingly appealing to harness its features for testing applications. On the surface Web-based testing would seem to be a natural evolution from computer-based testing. However, basic industry standards and measurement principles dictate that standardized administration conditions are required for many applications and the World Wide Web, by its very nature, almost precludes standardized conditions. This presentation will provide some definitional clarity to the term "internet testing" and highlight some of the issues related to standardized testing and the Internet illustrating both appropriate and inappropriate applications.

Translating Credentialing Exams: Practical Guidelines and Steps to Insure Validity
There are numerous test publishers who have created tests which must be translated and adapted. There are critical issues involving: Psychometrics, translation processes, job analyses, etc. Including international issues.

What Clients of Testing Vendors Want
Clients who provide CBT services are as varied as the programs they sponsor. This session explores how CBT providers meet diverse needs while assuring standardized delivery. Options for delivery services will be discussed.

Who is Representing the Examinee?
The success of a testing program requires input and endorsement from a diverse set of constituents. In some testing situations, the activities and goals of stakeholder groups may restrict test taker rights or may not be clearly understood and accepted by examinees. This session will explore the rights and responsibilities of test takers in high stakes testing programs. The roles and expectations of test takers, sponsors, developers and administrators will be considered in the context of professional guidelines and industry standards.

Platinum Sponsor Session

Adventures In Cyberspace: Using the Internet for Job Analysis Projects and Survey Delivery via Web Conferencing - The Chauncey Group International
The session will provide the opportunity for participants to explore the benefits and challenges associated with Internet-delivered surveys. Recommendations about Internet surveys will be provided-including deciding whether to deliver surveys by Internet or by mail and recommendations for successful Internet survey delivery. Participants will view an actual Internet-delivered job analysis survey. Dr. Montgomery will provide information about designing quality Internet surveys.

The conduct of virtual meetings will be discussed. Session participants will view Web Conferencing software. Important product features will be highlighted-viewing, creating and revising documents; application sharing; audience polling and write-in questions. Recommendations will be provided regarding the conduct of effective Web Conferences.

Automated Scoring Technologies Used for NAEP Performance Items - Educational Testing Service
Technological solutions allow schools to better prepare students for direct writing assessments utilizing web-based applications that provide students with formative writing practice, instantaneous scores and diagnostic feedback using natural language processing (NLP) scoring models. This session will present the Criterion Online Writing Evaluation Service and discuss how different schools and districts are using the automated scoring technology to compare student writing with national standards. The session will also present an overview of NAEP’s development and research activities to incorporate automated scoring technology within future NAEP assessments. Additional discussion will focus on ETS Technologies’ development of an automated scoring prototype, concept-rater (c-rater), for the evaluation of constructed responses in content areas and its effectiveness at producing "right/wrong" or "credit/no credit" ratings. This research and technology has the potential to evolve into an automated scoring application appropriate for evaluating short-answer constructed responses in online instruction and assessment applications in virtually all disciplines.

CAT At Your Fingertips - Computer Adaptive Technologies, Inc.
Computer Adaptive Technologies (CAT) can help you realize a valid, relevant, easily administered test. Our suite of services includes a best-of-breed content management system to create and bank your questions and tests, psychometricians on staff to help you formulate the right questions, program managers to assist in deployment, and a cutting edge Internet-based delivery system that makes it easy to administer tests to anyone, anywhere, anytime.

During these sessions, CAT will demonstrate our state of the art testing technology including:
Internet based content management
Create questions and publish computer based tests from anywhere in the world. Develop and bank innovative item types including multi-media and simulation items.

Internet based content delivery
Explore the variety of test delivery options available with the Internet.
Administer:
low-stakes practice tests at home via a browser
mid-stakes tests with a proctored on-line administration
high-stakes tests at a secure test center

Join us to really see all your options!

Computer Adaptive Technologies (CAT) is an Evanston, Illinois based international application service provider for computer-based testing. Wherever there is progress to be measured for advancement, achievement, proficiency, or status, CAT can assist you with an innovative, cost effective, technology rich solution.

Computerized Test Development - Galton Technologies
Throughout the credentialing industry, organizations are realizing the power of technology in any test development effort. Through state-of-the-art test development tools, test sponsors are reaping the rewards of speed and efficiency in all phases of test development.

Attend this workshop and you will witness the profound benefits technology offers to the test development process. We’ll cover it all; security, test design, item banking, psychometric reporting & analysis and test publishing. In our world of tightening budgets, shrinking development timelines and limited access to subject matter experts (SMEs), few can afford to continue using out-dated, inefficient tools. When it comes to streamlining test development, you’ll see in this session that Galton Technologies likely has some solutions you have been looking for.

Implementing a Licensure or Certification Program - An Insider's Perspective - Prometric, a Thomson Learning business.
Ultimately the success or failure of a licensure or certification program can be traced back to the very earliest decisions that shaped the course of future events. Just as with any complex system of interdependencies, a testing program is constructed of countless answers to seemingly simple questions. While each program is unique, they all share a common base of design attributes. This session will explore these characteristics with the goal of producing an implementation process that is reliable and reproducible. This session will combine a thorough discussion of a proposed project framework with real-world examples of successful program implementations.

David Meissner, Executive Director of Client and Program Implementation at Prometric, a Thomson Learning business, leads an innovative new team of Implementation Analysts. Implementation Analysts work in concert with Prometric's Account teams to translate client needs into clear systems and operational requirements. They then provide comprehensive project management from analysis and design through program launch.

Implementing Simulations in a Testing Environment: Going where few have gone before - TestOut Corporation
Come learn how to create cutting-edge simulation-based test items using existing technologies. Discover why concerns about cost, time and required expertise are myths in simulation item development. Find out what our research data says regarding open simulations. Know why leading certification programs are increasingly incorporating simulation test items in their certification programs and why you should too.

If you want to know more about simulations, if you want to implement simulations in your tests, or if you want to improve your existing simulations - this session is for you.

You know they’ve been coming for years - resistance is futile.

Managing Your Item Bank: A Success Story with Chauncey's TestExpress - The Chauncey Group International
This session will present a demonstration of Chauncey TestExpress, an item management software program designed to be flexible and dynamic to meet the needs of small paper-and-pencil testing programs as well as large computer-based testing programs. The features of the software include capabilities to store multiple-choice and free-response type items, multiple content codes for items, statistics, references, and item writer information.

Migrating to Computerized Testing - Galton Technologies
Over the past several years, Galton Technologies has grown to be a leading provider of certification and test development services. As credentialing programs continue to emerge and grow in various industries, it is clear that computerized testing is rapidly becoming the standard for test delivery in the credentialing industry.

In this session, Galton will show you how to take your paper-based credentialing program to computerized testing. We’ll cover all the critical factors related to moving your program to computerized testing, particularly the costs and benefits of making the change.

We'll introduce you to Galton's new, innovative program, ExamExpress, designed to help you gain access to computerized testing while avoiding prohibitive costs and barriers. You'll learn that computerized testing isn't just for large, mature programs. Most importantly, you'll understand the full solutions and benefits offered by computerized testing, enabling you to take your program to the next level of excellence.

Publishers' Use of XML-based Web Services in 2005 - Questionmark
XML-based web services supporting the creation, assembly, deployment, and discovery of dynamic distributed applications represent the next wave in computing. This talk will offer a futuristic look at how technology and content will come together in the next five years. It will include a discussion about how publishers can configure and maintain their content in a scalable and distributed environment.

Questionmark Perception in the Learning and Certification Process - Questionmark
How can assessments be used to the best advantage throughout the learning process? What are the most effective applications of diagnostic, formative, and summative assessments? When and how should you deploy low, medium, and high stakes exams? This presentation will explore the deployment of job analysis surveys, task analysis surveys and certification exams. It will touch on the use of assessments as tools for analyzing learning/training needs and as interactive learning experiences that enhance course-related testing. Attendees will gain an appreciation of all the styles of computerized assessments within the learning and certification process.

Technological Advances in Computer-Based Testing - NCS Pearson
Technology is revolutionizing the licensure and certification industry. On-line tools provide real-time information to efficiently manage your program. Advancements in exam item types can improve the measurement process. Digital biometric technology enhances the security and integrity of your program.

This session will demonstrate how technology can improve the development and management of your testing program. We’ll also provide you guidelines to consider as you implement these new technologies within your program.

Technological Advances in Computer-Based Testing - NCS Pearson
Technology is revolutionizing the licensure and certification industry. On-line tools provide real-time information to efficiently manage your program. Advancements in exam item types can improve the measurement process. Digital biometric technology enhances the security and integrity of your program.

This session will demonstrate how technology can improve the development and management of your testing program. We’ll also provide you guidelines to consider as you implement these new technologies within your program.

UTD: Explore the Possibilities of The Future Today - Prometric, a Thomson Learning business
This presentation will detail the new test driver technology that Prometric has deployed to service the assessment and certification industry. This technology, the Unified Test Driver (UTD) from Prometric, offers clients benefits and services
that were not available before.

Areas of focus include the technical design of UTD, the adoption of XML as a descriptive exam language for test construction, portability of intellectual property and development of custom logic to support exam logic.

Most importantly the presentation will focus on real-world examples of applications of this new technology to exam sponsors specific testing requirements. Case studies will be covered that show how clients benefited from applying UTD to their psychometric needs.

Post Conference Workshops

Information Technology Certificaiton Security Council (ITCSC) Introductory Meeting
Interested in protecting your IT Certification Exam investments? Come to this special interest group meeting to find out about ITCSC and what it can do for you.

Special Interest Group: Advancing Performance Testing
Please join us for a discussion of the performance testing.