Proactive Planning for Testing Irregularities

This blog article from Andrew Wiley of ACS Ventures (https://acsventures.com) who authored the section in the Technology Based Assessment Guidelines on Test Disruptions shares some insight on dealing with testing disruptions.

I am going to start this post with a small confession. I have admitted this to some friends and colleagues, but I have never really stated this out loud. So here we go. There is a part of me that likes it when things go wrong. I know that sounds a little off, but it is also true.

I have been working in the assessment industry for approximately 30 years (when did that happen!). I have had a variety of roles, directly working on and scoring tests, conducting research, and now as an independent consultant. During that time, I have been involved in projects as an external evaluator for testing programs that have experienced significant testing disruptions. When I talk about testing disruptions, I am thinking about things like a DDOS (denial of service) attack that prevents tests from being delivered, loud concerts playing right outside a test administration location, and even tests being delivered perfectly in location A, while location B, just down the road has candidates getting intermittently kicked out of their session for no apparent reason.

When testing disruptions occur, many programs quickly move into an all-hands-on-deck mode and something of a mad scramble occurs as the programs work to collect as much data and information as possible. There is usually a level of energy and focus that you do not find when you are planning for the next set of item review meetings.

I will freely admit that my statement about being motivated by this energy is impacted by the fact that I am walking through the door as an external evaluator, and in some ways, I have not been directly impacted by these investigations. But the energy and focus for organizations is so strong in the moment to determine what happened and what actions are needed to fix it. Everyone is in the room trying to get the issue resolved and get the program up and running as efficiently as possible. But while organizations can snap to attention and get to work, far too many of these organizations are caught somewhat flatfooted when a disruption occurs.

Most testing organizations work tirelessly to complete as many steps as possible to try to avoid any serious testing disruptions from occurring. Those efforts are critical, and I am not arguing for a reduction of these activities. Instead, this post presents an argument that all of us – testing professionals and organizations – need to acknowledge that serious testing disruptions could occur within our program. And we need to have a detailed plan for how to respond in this event.

It's not feasible to discuss all priorities that organizations need to consider on this front in a blog post. So, I want to focus on highlighting some of the key aspects I have observed over time and call out some excellent resources that could be of assistance. For this post, I want to highlight three domains that programs need to consider in a plan: 1) People, 2) Policies, and 3) Procedures. The Technology Based Assessment Guidelines (ITC and ATP, 2022) review all of these actions and more and are highly recommended as a source for people beginning this process.

I should point that when confronted with testing irregularities, programs may need to split the response options into two broad areas. First, there is a need for an immediate response team; these are the people and policies that guide any decisions in real time. People in this capacity could be making decisions on whether test administration needs to be suspended or whether candidates should be allowed additional time or accommodations to finish their exam.

A second team could be characterized as the Investigative team; individuals who are reviewing and finalizing all activities after the fact. People in this role will make final decisions on which examinee records should be graded or considered incomplete, what type of retesting policies should be offered to impacted candidates, and determining root cause. Working through each of these domains, here are some key things to create a proactive plan.

What People Should be Involved?

• What training and guidance have the people administering the test received to address immediate problems? What is the chain of command to escalate issues during the test administration?
• Within the test sponsor’s organization, who are the immediate points of contact?

In the event the test is administered within a testing window, are the points of contact available the entire time? Any travel plans? What contact information should be used and do all appropriate people have the contact information?

• Who is responsible for making final decision on any testing decisions?
• Who is responsible for gathering information for the final decision makers?

Who are the backups for these roles?

• Are the people on the list aware of their role and have the necessary permissions/access to fulfill their roles?

What Policies will Guide Decisions?

• Do guidelines exist for the types of testing disruptions that could lead to suspending test administration – within a given center or locale?
• What are policies for scoring tests with incomplete data?

Can candidates with incomplete tests still get certified if they perform well enough? How would “well enough” be determined?

• What are the policies for candidates retesting if they experience any significant testing disruptions?

What Procedures would Investigations Follow?

• Can we identify anticipated disruptions and map out the data and information that would be needed to fully understand the issue?
• How would data be made available through the test delivery system; how quickly can it be shared across the investigation team?
• What service level specifications can be developed for test delivery vendors around expected response time to queries and what data can be made immediately available?

While some of these items may seem obvious, I have sat through too many conversations where test delivery vendors were only able to answer basic delivery facts through a manual review of complex log files. I have observed environments where no one was aware of the final decision maker and hours were lost as everyone realized the final decision maker was sitting on a plane and unavailable.

In other instances, test sponsors and their vendor partners have disagreements on roles and responsibilities and which tasks need to be completed under the current contract requirements and what things should be classified as new work and as such billable work. Negotiating contracts, and in particular, the detailed service level agreement language with all parties is critical to ensure that the possibility of testing irregularities is considered in any contract. The specificity of this language could assist all parties immediately and help to more efficiently address any issues that arise.

While admitting that testing irregularities are a fact of life may not be the most comfortable role to take on, the benefits that arise when disruptions do occur are enormous and help testing organizations quickly recover and continue to serve their customers at the highest levels.