Introducing the Web-based Delivery Section of the ITC/ATP Guidelines for Technology-based Assessment

This blog article by Elizabeth Stone, an independent consultant, is part of a series supporting the ITC/ATP Guidelines for Technology-based Assessment.

An Introduction to the Section

In late 2019 I was invited to contribute a section on web-based delivery of assessments to the Guidelines for Technology-based Assessment.  My first draft was due in February of 2020, and at that time the global pandemic was still (barely) around the corner.  With the onset of health-related distancing guidelines, the Internet became an even more important vehicle for obtaining news, maintaining social connections, and supporting educational instruction and assessment.  Web-based delivery, testing that takes place wholly or partly over the Internet, is not new.  Advantages to this type of delivery include having a central repository from which testing material or software can be distributed and updated and in which test-taker data can be stored; the ability to apply changes on demand to test administration in the field; and increased access for individuals who do not live near (or who cannot get to) testing centers.  While the widespread web-based delivery during the height of the pandemic may have helped to mitigate learning losses and keep education and career goals on track, it also brought into the spotlight some of the concerns about online assessment that this section of the Guidelines is meant to address.  This blog post summarizes the web-based delivery guidelines and some of the surrounding issues that motivate them.

Factors to consider for web-based delivery of assessments

Web-based delivery comprises the first portion of Chapter 3, Test Delivery Environments.  Other sections in this chapter include Local, Offline, and Mobile Delivery; Locked-down Browsers; and Interoperability.  Mobile delivery can occur in part or in whole via the Internet, and locked-down browsers may also make use of testing material provided over the Internet.  These modes of delivery each have their own specific concerns that are addressed in their sections of Chapter 3.  Many of the issues with web-based delivery discussed here are explored elsewhere in the Guidelines at different levels of detail or from other perspectives (e.g., test security [Chapter 8], data privacy [Chapter 9 and previous blog post], importance of accurate measurement and recording of response times for scoring [Chapter 4 and previous blog post], and equity issues surrounding technology access and test-taker privacy [Chapter 10]).

The guidelines pertaining to web-based delivery can be broadly categorized for the sake of convenience into: Test Security, System Robustness and Disruption Handling, and Testing Conditions.

Test Security

Test security in delivery over the Internet is a major issue and is rightfully introduced up front in guideline 3.1, which centers on “the secure exchange of test material and test taker data as appropriate for the testing purpose.”  Some security threats are direct, such as hacking into the testing network to steal confidential material.  Other threats constitute disruptions to testing, such as denial of service attacks.  Still other threats such as ransomware attacks can combine the two.  These attacks may involve illegally obtaining and holding test-taker records hostage for payment.  Further, a common response to ransomware attacks is to take a network offline while devices are cleaned of the malicious software, preventing connection to the Internet.  Protecting a network includes implementing firewalls and other security protocols to prevent these threats from being carried out, but security must also be maintained at the endpoints of the data transmission pathway.  This includes the physical security of the origination server (e.g., at a testing company) and proper encryption but can also include steps such as transmitting item responses across a secure network to be scored at a secure origination server rather than requiring keys to be scored locally in the field.

Individuals involved in testing (e.g., system administrators, proctors, test takers) should only be provided as much access to the network, secure material, and hardware as is required for their role.  It is also crucial to ensure that the individual taking the test is the registered test taker, and the documentation required for this purpose should be clearly delineated in advance.  Once testing begins, security monitoring commensurate with the stakes of the test should be conducted (e.g., through live feed camera or other means), and the test taker should be made aware in advance of what is to occur in that regard as well.

System Robustness and Disruption Handling

System stability is supported by appropriate computer configurations and connection capability individually and at scale, and this requires advance preparation by all stakeholders.  Not all test takers may be able to determine accurately the technological adequacy of their systems.  For example, a router or modem may have an advertised upper bound connection speed or bandwidth, but these factors may be impeded due to issues locally or upstream.  In addition to the system stress testing that should be conducted by the test provider before deploying the assessment to the field, benefits can accrue from providing tutorials or mini-tests with representatively interactive item types ahead of time to users and test proctors for individual system testing.  Technical issues may be further mitigated by ensuring that test takers cannot inadvertently trigger test changes that are not warranted or desired (e.g., turning on sticky keys through a combination of key presses if that feature is not required by the test taker).  This may be a difficult balance to achieve, as accessibility features and tools are a necessary support to many individuals, but mistakes may be difficult to diagnose and fix.

Given all that can happen during operational testing, system performance should continue to be monitored, and test takers and those in roles assisting with testing should have access to troubleshooting procedures and contact information in case technical issues arise.  For example, what should the test taker do if the proctor is unresponsive in the middle of the test, items aren’t rendering correctly, an error occurs, or the system simply hangs? These contacts must be well trained to answer the questions of frustrated or dismayed test takers or to escalate the issue quickly to someone who can help resolve the issue.  Data loss from connectivity issues and other disruptions should be minimized by capturing and storing responses as they are submitted and providing the ability to resume testing efficiently.

Testing Conditions

The third category of guidelines has to date received less attention than the previous two, perhaps because test purveyors have less control over these aspects.  However, the testing conditions and environment can have important impacts on the mindset, performance, and experience of the test taker.  An important guideline in the age of remote testing is 3.14: “Testing should be conducted under appropriate environmental conditions, and outcomes should be interpreted in light of those conditions.”  The Standards for Educational and Psychological Testing expands on this idea in a similar standard (6.4) that may have initially been focused on testing centers.  This effort will help to minimize construct-irrelevant variance by eliminating unnecessary distractions such as noise or discomfort and should also lead to a more standardized set of circumstances in which the tests are being taken.

Last, but certainly not least, guideline 3.13 emphasizes the importance of considering the impact of negative feedback on the test taker.  Individuals taking tests remotely may be dismayed or worse by harsh feedback about the correctness of their response or assessments of their personality.  In a remote testing scenario, it is also important to acknowledge that trying to create an adequate environment and technological setup and dealing with their own technical difficulties may place additional stress on the test taker.  And as opposed to the relatively impersonal or even sterile feel of a testing center, to which test takers can bring few of their own possessions, remote testing that requires a 360-degree view of the testing room at home often obliges test takers to share much more than they might otherwise want to.

Concluding Thoughts

Web-based delivery comes with a trade-off between the flexibility and access that the Internet provides and a potential for a resulting lack of control of the test session.  Although there has been justified emphasis on the security and delivery aspects, the testing process can vary in important ways that affect the test taker, and there are currently fewer guidelines and supports in this regard.  For example, test security may require an individual to display fully the room in which they are testing.  This brings up privacy issues and necessitates the consideration of what happens with any video footage, similar to considerations of how other test taker data or personally identifiable information are handled.  The need to attend to aspects of web-based delivery discussed here and in the more detailed Guidelines section is magnified when considering the use of this mode of delivery by possibly unsupervised minors and the situations in which they may be testing (e.g., at home).  In light of digital inequities, differences in technological literacy and familiarity, and adequacy of testing environment, more focus should be placed on investigating possible consequences and providing additional supports.  The variations in security, platform and network stability, and environment can all have an impact on test-taker experience, fairness, and validity of the interpretations of resulting scores.