September 2000

J. Patrick Jones
Professional Examination Service

This article describes the major elements of a communication plan for the implementation of a computer-based testing (CBT) program. The major benefits and potential drawbacks of a CBT program are reviewed, and the information needs of various stakeholder groups are identified. The article concludes with an overview of communication strategies and evaluation techniques that can facilitate the transition to CBT.

Advances in technology have significantly increased the acceptance of computer-based testing (CBT) as an effective means of delivering assessments for a wide range of test sponsors. The availability of worldwide CBT networks and the increased use of the Internet as a test delivery medium have contributed to a rapid transition from alternate testing formats to CBT. The success of this transition will depend, to a large extent, on the ability of testing professionals to communicate the benefits and limitations of CBT to stakeholder groups.

The objective of this article is to describe the major elements of a communication plan regarding the implementation of a CBT program. The major benefits and potential drawbacks of a CBT program are reviewed, and the information needs of various stakeholder groups are identified. The article concludes with an overview of communication strategies and evaluation techniques that can facilitate the transition to CBT.

While test users and test developers recognize the enormous potential of CBT, some constituencies within the testing community are less sanguine regarding the shift away from alternate test administration methods such as paper-and-pencil testing. The following poem found on a web-site hosted by an association of college test administrators addresses some of these concerns.

A computer was something on TV
From a science fiction show
A window was something you hated to clean
And RAM was the cousin of a goat

An application was for employment
A program was a TV show
A cursor used profanity
A keyboard was a piano

Compress is something you did to the garbage
Not something you did to a file
And if you'd unzipped anything in public
You'd be in jail for a while
Log on was adding wood to the fire
Hard drive was a long trip on the road
A mouse pad was where a mouse lived
And a backup happened to your commode

Cut you did with a pocketknife
Paste you did with glue
A web was a spider's home
And a virus was the flu

I guess I'll stick to my pad and paper
And the memory in my head
I hear nobody's been killed in a computer crash
But when it happens they wish they were dead!

A well-designed CBT program offers test users ample opportunities and venues for testing. Most CBT interfaces are easy to use, and they offer a standard testing environment that can be effectively adapted for candidates with special needs. Test developers can deploy assessments based on improved psychometric models that feature adaptive testing and enhanced item and test formats. Test security is enhanced due to controls built into the delivery systems that limit item and test access to authorized individuals. In some testing programs available on the Internet, testing can occur at any time and at virtually any location in the world. Most CBT programs feature the provision of immediate score feedback to candidates and test sponsors.

What are the potential drawbacks of CBT, and how can they be minimized or avoided? One common concern among stakeholders is that success on a computer-administered test requires a high level of computer proficiency. To address this concern, the user interface must be designed to minimize reliance on computer skills that are not part of the construct being measured by the assessment. Test sponsors can also reduce test taker anxiety by providing tutorials and CBT sample tests to candidates in advance of the testing session. Finally, the test delivery procedures should include adequate on-site support in the event that test takers need assistance during the testing session.

A second criticism is that the relatively high cost of CBT is not offset by the perceived benefits. Critics cite the cost advantages of alternate delivery systems (e.g., paper-and-pencil) and question the real value of CBT. In order to counter this criticism, a complete analysis of all cost elements associated with CBT and alternate delivery models should be undertaken. Most cost analyses fail to account for candidate opportunity costs associated with delays in testing and the effects of these delays on the ability of the candidate to pursue career or educational opportunities. For example, if a paper-and-pencil test is offered at a limited number of test locations on two annual test dates, candidates may have to incur high travel costs and forgo opportunities as a result of the limited access to the test event. Competitive issues need to be considered, especially in situations where a rival testing program is administered via computer. The costs and risks related to security threats associated with paper-and-pencil testing procedures also should be identified.

A third potential drawback of CBT is that access to high quality and affordable delivery channels is limited in some global markets. To address this shortcoming, the viability of delivering the assessment via the Internet should be considered. For some programs, the test may be delivered in a cross-channel (e.g., two or more CBT providers) or a mixed-channel format (e.g., paper-and-pencil and CBT) as a means to improve test access.

Another potential limitation of CBT is that the nature of the assessment may not permit immediate score feedback to candidates or test sponsors. One method of addressing this limitation is to provide preliminary results to test takers at the conclusion of the testing session and an official score report after final scoring activities have been completed. An analysis of the score reporting time lines related to CBT and alternative delivery models should be conducted. For most testing programs, the implementation of CBT significantly reduces the time required for test scoring and reporting as compared to alternate test delivery formats.

In developing a communication plan relating to a transition to CBT, the first step involves the identification of stakeholder groups and their information needs. Some of the information needs are similar among diverse stakeholder groups, whereas other aspects of a transition to CBT are relevant only to a limited set of stakeholders. The information needs for some common groups of CBT stakeholders are discussed below.

Test Sponsor: The organization responsible for the development and administration of the CBT program must have a thorough understanding of the measurement model underlying the testing program, and the resources required to implement the model in an effective and efficient manner. The major features, benefits, and drawbacks of the CBT model should be articulated for the test sponsor. Test sponsor staff must understand the impact of the CBT program on internal and external stakeholders. For example, if the test sponsor is converting a testing program that is currently administered on two common test dates on a national basis to a CBT program that offers "on-demand" testing worldwide, the effects of this transition on all program activities should be anticipated and documented. Test sponsor examination program activities affected by a conversion to CBT include examination application review and processing, fee collection, item and test development and review, test administration, customer service, scoring and score reporting. A related issue for consideration by the program sponsor is infrastructure changes required to support the more frequent testing opportunities afforded by a transition to CBT. Telecommunications and computing resources should be reviewed to ensure their continued viability with the change in test delivery format.

Test Users and Candidates: Users of tests and candidates need to understand the major implications of CBT, including testing schedules and channels, test model features, and test fees. The timeline and major milestones associated with the transition to CBT need to be clearly communicated to these stakeholders. Where possible, sample questions configured to match the anticipated CBT delivery format and model should be made available to test users and candidates well in advance of the implementation date for CBT. Information relating to the administration and scoring of the test (e.g., test adaptation and translation, data integrity, scoring algorithms, security features) should be made available to test users and candidates.

Test Publishers: Testing services and test publishers require knowledge of the proposed CBT model and its major features, along with the timeline and major milestones related to the transition to CBT. The roles and responsibilities of the testing service and test publisher relative to the channel partner and test sponsor need to be carefully articulated and documented, especially as they relate to the development of the CBT test driver, management of quality control and scoring activities, and communicating with test users and candidates.

CBT Channel Partners: Knowledge of the CBT model and its major features, along with the roles and responsibilities of other stakeholders involved in the development and management of the CBT program should be clearly conveyed to the CBT channel partners. The time line for program development and implementation and the infrastructure requirements expected by the test sponsor (e.g., site availability, hours of operation, call center requirements) should be documented for the channel partners. Program policies relating to eligibility determination, score reporting, and other test administration and scoring activities need to be communicated to the channel partners by the test sponsor.

Regulatory Community: For professional licensing and certification examination programs converting to CBT, regulatory agencies require information about their roles and responsibilities, including applicant eligibility determination, score transfers, and other administrative activities affecting these agencies. These agencies need to understand the major service and interface features of the CBT model, and the time line for implementation of the program. In some situations, regulatory agencies will need to change their infrastructure and workflow processes to support the new mode of testing. Regulatory agencies should have access to materials published for test users and candidates to support their efforts to communicate with these stakeholders. Finally, aspects of the proposed program that may require changes to existing practice acts or board rules should be discussed by the test sponsor with the regulatory agencies early in the transition process to ensure that these legal issues are adequately addressed prior to program implementation.

Training/Educational Community and the General Public: Agencies involved with candidate training and education can serve as an effective communication outlet for information on CBT programs. The service and interface features of the proposed CBT program, along with the time line for implementation should be made available to these stakeholders. The training/educational community and the general public should also have access to materials designed to prepare candidates for the planned CBT program.

A variety of media can be effectively used to provide stakeholders with information relating to the CBT program. These media include test sponsor web sites, training and candidate preparation materials, press releases, test bulletins, newsletters, and journals. For some programs, industry conferences and seminars or invitational workshops may be convened by test sponsors to ensure that key constituencies and decision makers have access to information concerning the CBT program. Documents issued directly to candidates such as testing program handbooks and score reports can be effectively tailored to provide timely information about the CBT program.

To form a strategic plan for the communication program, test sponsors typically gather input from test users, test publishers, CBT channel partners, and other stakeholders. One effective means of identifying the information needs and concerns of important constituencies is the conduct of market research studies. The results of these studies can be used to guide the development of documents, briefings, and other media targeted to particular audiences and issues related to the CBT implementation plan.

Key stakeholders and decision makers often benefit from briefing papers or fact sheets to facilitate their role as spokespersons for the CBT program. These documents should address specific segments of the CBT implementation plan and be released prior to beginning each new phase of the plan. Test sponsor staff often receive regular communications on the program that focus on workflow and infrastructure changes related to the transition plan. The publication and dissemination of documents that present "frequently asked questions" (and their answers) are effective tools to ensure that consistent information is available in response to stakeholder issues and concerns.

In most testing programs, a broad range of media will be required to reach all key constituencies. The test sponsor should publish contact information for authorized individuals who are responsible for managing the dissemination of information on the CBT program.

A variety of programs can be administered to evaluate the utility of the communication plan. Surveys assessing stakeholder reactions to the implementation activities can be administered to candidates at CBT test centers, test users during the sales or fulfillment process, trainers and educators at industry conferences, and other key program participants at other venues. An analysis of the results from these surveys can help guide the development of communication programs targeted for specific groups on particular program aspects. Another means of gathering information on the effectiveness of the communication plan is evaluation of customer service call center logs and incident reports maintained by the test sponsor or CBT channel partner. These information sources can also prove valuable during the development of CBT program enhancements after the initial implementation phase has been completed.

The development and implementation of a comprehensive communication plan significantly impact the overall success of a CBT program. Addressing the information needs of key stakeholders builds acceptance and overall endorsement of the CBT implementation plan.

¹ Paper presented at the February 2000 Computer-based Testing: Applications for the New Millenium of the Association of Test Publishers.