Guidelines for Test Design: Blueprints for Blueprints

This blog article by Nathan Thompson, CEO & Co-Founder of Assessment Systems is part of a series supporting the ITC/ATP Guidelines for Technology-Based Assessment. Nathan contributed to Chapter 2 (Test Design and Assembly) of the Guidelines.

Architecture depends heavily on blueprints; you can’t even build a plastic playhouse in the backyard without some detailed instructions. Of course, there are guidelines around what should be covered by building blueprints, as well as how figures should be created, what documentation is necessary, and more. I remember hearing years ago about an architect that was proud of his massive collection of tracing elements for creating blueprints with paper-and-pencil, because the shape of the elements was standardized - though of course his prized possessions became obsolete the day that the first CAD software was released!

When it comes to assessment, having a high-quality test specification document, or blueprint, is also essential. Given that validity depends on the chain of evidence from test score usage back to the original design of the test, documenting all elements of that design is then the foundation of validity. As with architecture, there are some best practices that surround what should go into test design and specifications. For example, credentialing exams rely heavily on job or practice analysis studies, the data from which is then analyzed to provide the content blueprints of the test. But there are many other things to consider, which can be categorized into psychometric and non-psychometric aspects.

Non-Psychometric Aspects

The first thing to consider, of course, is the intended usage of the test, which is often couched in complex notions like how to define the role of a certain profession in a certain workplace, such as the role of an ophthalmic assistant in ophthalmic clinics and hospitals. It can become more complex in tiered situations, where there might be three or four successive certifications plus several add-on certificates.

This question then translates into the design of the job analysis. It is also used to drive discussions on other design aspects. Is the test to be a written test, a simulation, a live performance test, or some combination? If written, what sort of items will you use: traditional formats or technology enhanced items (TEI, see the section in Chapter 1 on Technology-Enhanced Items)? Are there certain levels of cognitive complexity that you wish to define and cover? What is the scoring model that you intend to use? The TBA Guidelines provide some recommendations to consider when asking yourself these questions.

Psychometric Aspects

In low-stakes and low-volume programs, the psychometrics can be minimal. There might be a single linear test form, and you analyze the results once per year with classical test theory, a simple paradigm based on averages, proportions, and correlations. But as the stakes and volumes increase, so does the complexity.

There are three primary test designs to consider: linear, linear-on-the-fly testing (LOFT), and computerized adaptive testing (CAT). CAT also subsumes a related approach called multistage testing (MST). Even with linear exams, your organization might produce dozens of forms per year, necessitating many statistical analyses. If you wish to establish equivalency of the forms, you will need to implement equating studies. The more complex methods like CAT necessitate sophisticated research studies to determine the design of the algorithms to be used (e.g., item selection) and also to record the results in the documentation for validity evidence and posterity. The TBA Guidelines provide a discussion on these test designs and the work necessary to validate them.

Summary

As mentioned earlier, a key consideration is that all discussion, research, and choices made on test design should be well-documented in a formal report. This serves as a reminder internally, but also the foundation of validity, especially if the test were to ever be challenged. You might also want to publish the document, or a simplified version of it, to interface with stakeholders such as candidates or parents. This might help to ward off some complaints!

You can see and download the full ITC/ATP the ITC/ATP Guidelines for Technology-Based Assessment here.  https://www.testpublishers.org/assets/TBA%20Guidelines%20final%202-23-2023%20v4.pdf.