√ Sponsorship is now open for Innovations in Testing 2017!
√ Registration for Innovations in Testing 2017 is now now open! Members can log into the ATP website and go to Member Benefits and Discounts for a registration discount code.
Did you miss the ATP Education Division Fall Webinar on September 8? If so -- you can catch up to this informative conversation on accommodations and accessibility in assessments by downloading the informational flyer and downloading the presentations: Coming to common terms with assistive technology and and standards world; and Common Language around Accessibility and Accommodations for States and Assessment Vendors.
Since the early 1920s, psychological and educational tests have come into widespread use as tools for professionals such as psychologists, counselors, teachers, and human resource managers in government, education, and industry. In North America today, there is scarcely a person over the age of ten who has not taken at least one test, whether it has been an achievement test, an IQ test, a personality test, or a measure of interest or aptitude in a particular field.
A principal reason for this growth in test use is that the results of professionally developed tests are more reliable and accurate assessments of human capabilities and behavior than those based on the observations of individuals alone, however well-informed or insightful those observations might be. Individual judgments about people tend to be subjective and vulnerable to error, no matter how hard one tries to be fair. Even skilled observers who pride themselves on their objectivity are apt to perceive people through a filter of unconsciously held biases, personal beliefs, incomplete knowledge, and temporary distractions.
By relying on personal observations alone, a teacher may view a slow learner in the classroom as being lazy; a psychologist may interpret eccentric behavior as indicative of a serious emotional disorder; or a human resources director may view a carelessly dressed person as incapable of handling a high-responsibility job and all three observers may be wrong with potentially serious consequences. Tests can help to reduce these types of misjudgments about individuals.
This does not mean that tests can serve as a replacement for the personal judgment of professionals. In fact, a test is usually only one source of information among several used to assist in making the most accurate and fair decisions possible. Professionally developed tests that are designed by experts, scored and interpreted by properly trained individuals, can help even the most experienced and knowledgeable decision-maker to construct a fairer and more accurate picture of an individual. With more accurate information, teachers can better encourage students who have different learning styles to learn more effectively, psychologists can better serve their clients by providing proper treatment based on more accurate diagnoses, and employers can increase the likelihood that they will hire the right person for any given position.
Today, there are literally hundreds of tests available to professionals. These may be used to measure a wide range of human characteristics and behavior. Some, such as mental abilities tests or personality inventories, may be quite comprehensive in scope. Others may be more narrowly focused aids used to diagnose problems such as alcohol abuse or to assess a job applicants mechanical ability.
Tests may have more than one use. For example, a neuropsychologist might use a cognitive test to track a patients rate of recovery from the effects of a stroke. A school psychologist could use the same test to assess a student for the possible presence of a learning disability. And a consulting psychologist could use the same test to assess a job applicants abstract reasoning ability when that skill is vital to effective job performance.
Despite the important and useful role that tests play in businesses, schools, clinics and hospitals, many people have questions about their origins and use. This is not surprising, since tests are not often widely understood by the public. Test questions are typically not supplied ahead of time, and their results are generally kept private. Tests can be highly sophisticated and complex instruments and are often based on many years of rigorous research. The following answers to some commonly asked questions about tests may help you better understand how tests are developed and how they are best used.
WHAT IS A TEST?
When we use the word test in this brochure, we are referring to professionally developed instruments such as those produced by ATP members. Professionally developed tests are constructed responsibly and are responsive to existing professional standards. It is important to emphasize that not every assessment that is promoted or represented as a test falls into this category.
A test begins as a set of questions. Experts in the appropriate field construct the questions to assess or measure a specific ability or characteristic. The questions are based on the developers knowledge of the ability or characteristic to be assessed by the test.
Next, the questions comprising the prospective test are administered to a large number (often thousands) of individuals in field studies, and the results are analyzed by measurement experts. The prospective test must prove to give the same accurate results time after time to be reliable, to be equally applicable to all groups who take it to be fair, and to measure usefully what it is stated to measure to be valid.
A prospective test that clears all these hurdles is still just a work-in-progress. Many tests are submitted to full scale standardization, where they are administered to a large sample of people under standardized (uniform) conditions. Samples of these kinds are selected to accurately reflect the national population or other target group and are referred to as the norm group or normative group. Additional studies using other large groups are conducted to further document the tests reliability, fairness, and validity. Results of these technical studies are collected in a test manual which guides test users in the proper use of the test. Finally, clear and easy-to-follow administration, scoring, and interpretation materials are created for test users.
A modern test represents a tremendous investment of time, talent, and money by the developers, technical experts, and the publisher. Creating or revising a comprehensive achievement, aptitude, personality or intelligence test can easily cost several million dollars.
ARE TESTS ACCURATE?
While no assessment procedures are perfect, tests developed properly and used within the limits for which they were designed are highly accurate. Professionally developed tests are constructed on a foundation of research results and previous test outcomes that have often taken many years or decades to accumulate.
Tests can be easy to criticize, sometimes for the very reasons that they are so useful. For example, professional guidelines on test development require that the tests underlying research be described in the test users manual. This manual must include scientific evidence that makes clear both the strengths and weaknesses of the test. These days, there are few products that document their true characteristics, the strengths and weaknesses, so clearly and openly. The professionals who administer and use the test are responsible for understanding and evaluating the underlying characteristics of the test. You can ask them about these issues if you have questions.
ARE TESTS ALWAYS RIGHT?
Well-made tests, when used for their intended purposes, make fewer errors than other single evaluation procedures. However, test results are not perfect. No procedure that assesses people can be. The best decisions come when test results are combined with information from interviews, expert observations, ratings of past work, and so on. Even after considering all the information that is available, however, the people who make the most effective decisions will tend to be those who best understand how to incorporate the information provided by the test results.
WHY DO I SOMETIMES READ THAT TESTS ARE NOT FAIR, OR ARE NOT DOING WHAT THEY SHOULD?
When you hear someone say that tests are not performing well, the best question to ask in return is: Compared with what? Tests are often used to make difficult decisions about people. While no procedure that has people as its focus will be without error, tests do an excellent job of minimizing error.
More than seventy years of theoretical and practical research at the finest universities and laboratories in the United States have gone into refining test theory. When developing a test, reputable publishers or their authors conduct research that rests on this foundation and is responsive to the highest current professional standards.
Unless the critic can point to an alternative procedure that rests on a similar body of evidence and has proven to be equally reliable, fair, and valid, it is best to take the criticism with appropriate skepticism.
HOW DO TEST PUBLISHERS ENSURE THAT TESTS ARE FAIR?
Naturally, the first step is to instruct the people developing a test to avoid questions or content that appear unfair, stereotypical or biased toward any group that might take the test. Another step is to have a panel of experts review the questions for potential bias. Such a panel would include members of various groups that might be affected by bias in the tests results. Then, before it is published, the test may be given to individuals from different groups to see how well they perform. The data are evaluated to verify the work of the panel of experts, checking whether the test questions unfairly favor one group or other and whether the test predicts outcomes equally well for different groups. Test materials that do not pass these reviews are removed.
WHY ARE QUESTIONS ON SOME TESTS SO PERSONAL?
This will vary quite a bit from one kind of test to another. In some educational and business applications for example, certain questions may be inappropriate for example, questions about sexual preference or political or religious beliefs. Tests used in these settings must be carefully chosen and implemented with these restrictions in mind.
In some other applications, however, personal questions are necessary. On tests that are used for psychological counseling and treatment, questions are personal because peoples problems are personal. When people are having marital difficulties, for example, a counselor must seek answers to questions about sexual behavior, money management, arguments over childrens' religious training, occurrence of aggressive or violent behavior, drinking problems, and so on. In these instances tests are particularly useful. They let each person answer the questions privately and safely; they consistently ask all the questions so that important issues are not missed; and they allow answers to be compared with a large database to see how problematic or extreme the persons concerns may be.
In these cases it is important that consent be obtained from the person being tested before the test is administered. For minors, written consent must come from a parent or other legally responsible adult.
WHY DO SOME TEST QUESTIONS APPEAR SO SIMPLE OR POINTLESS?
Take as an example some statements that might appear on a test that assesses emotional distress. What does it mean if someone answers Yes to the statement: My thoughts are often jumbled. Anybody could feel like that one day and not the next. The truth is, a feeling that ones thoughts are jumbled does not mean much by itself. But now what if the person also answered Yes to the statements, It seems like I am always tired, I often feel guilty, and I usually wish people would just leave me by myself. Like the jumbled thoughts example, these are statements that, taken singly, each of us could probably have endorsed at some point or other. On the other hand, if an individual responded Yes to all these statements, a mental health providers attention might be drawn to potentially serious concerns.
Other kinds of tests will ask very different kinds of questions, yet there will often be the same sense that individual questions do not seem important. A test does two things with questions such as these. First, it groups many of them together in sets so that the meaning becomes focused and a single response or answer does not get overemphasized. Second, it documents how large numbers of carefully chosen people responded to these sets of questions. A tests power derives from the care that goes into assembling sets of questions and then studying how groups of people answer them. Separated from that background of research, the point is never clear in a single persons answer to a single question.
WHAT RIGHTS DOES A TEST TAKER HAVE?
First of all, a person (or the parents of a child) should voluntarily agree to a test administration, just as he or she would to sitting for an interview. It is also a right of a test taker to receive some information about the purpose of the test and how the results will be used. Further, it is the test takers right to have the results of the test kept confidential except for the test taker (or legal guardians), the experts who are authorized to give and interpret the test, and the individuals who are involved in the decision making process for which the test was administered.
IS IT ALL RIGHT TO GET THE TEST BEFOREHAND AND STUDY THE QUESTIONS?
No. Even on tests where there is no incentive to cheat, it is very important that the test questions remain unknown until the test is given. That's because a test is valid only if all persons take it under the same conditions that is, on a level playing field, so to speak. Any familiarity with the questions ahead of time may distort the results. Test publishers and responsible test givers go to great lengths to guard the integrity of the tests to ensure that they will be fair to all test takers.
HOW DO YOU KNOW IF A TEST IS BEING PROPERLY USED, OR HOW DO YOU KEEP TEST RESULTS FROM FALLING INTO THE WRONG HANDS?
The American Psychological Association and most state and provincial agencies for professional licensure have standards and codes of ethics on the proper use of tests. Publishers and users of educational tests also have strong policies on test use. Any person who has a concern about the misuse of test results should promptly call the appropriate professional agency.
ONCE A TEST IS PUBLISHED, IS IT EVER UPDATED?
Yes. Publishers try to ensure that the tests they publish do not become obsolete. They do this by periodically updating a test or extending its validation. For example, norm-referenced tests, which compare an individuals results to the results of a group of his or her peers, must be frequently updated since national norms may change over time.
WHAT KIND OF TRAINING SHOULD A TEST USER HAVE?
Persons who purchase and use tests should be qualified and trained in the application of the tests they will use. It is their responsibility to understand the nature of the test they are giving, to be aware of the research that validates the test, and to know the proper conditions under which the test should be given. They are also responsible for knowing how to interpret the results appropriately.
WHO INTERPRETS THE TEST RESULTS?
A trained professional such as a psychologist, counselor, or human resource manager typically interprets the results of professionally developed tests. Test publishers make available extensive aids for interpreting tests. These aids include manuals, printed reports, or interpretive guides that help test administrators report and apply the results properly. The publisher also provides the research results on which the test report or interpretations are based.
IS IT POSSIBLE FOR A TEST TAKER TO MANIPULATE OR CHEAT ON A TEST?
Although some people believe they can outsmart a test, it is seldom the case. First of all, most people don't cheat on a test if they are seeking some kind of help by taking the test. They realize that only candid answers will help professionals give them appropriate and effective assistance.
Tests that are used in settings where there may be an incentive to manipulate or cheat are constructed with checks and balances that identify inconsistent or inappropriate answers. These alert the test administrator that someone has attempted to answer questions deceptively. Test publishers and test users are also careful to maintain the security of tests that may be used in these situations, making sure that test materials are properly kept in secure places and are not released to non-professionals.
WHO SETS THE STANDARDS FOR TESTS?
Ultimately, the test publisher is responsible for the quality of the test. However, tests used in schools, businesses or other public settings are regulated by federal and state laws. Also, test developers and publishers are generally members of and support professional groups that have published standards for test development and use.