©2002 Association of Test Publishers
Save or read as pdf file

ATP to host third annual conference: Technology in Testing: Application and Innovation

The Association of Test Publishers will present its fourth annual conference, Technology in Testing: Application and Innovation, February 24 - 26, 2003. The site of this year’s conference will be the Amelia Island Plantation located in Amelia Island, Florida (outside of Jacksonville).

“There are several new and exciting tracks at this year’s conference including: Program Management; Integration of Learning and Testing; Test Development; Alternative Testing Technologies; and CBT in the EU,” said Conference Co-Chair Bill Cramer, of Chauncey Group International. “Another added feature at this year’s conference will be poster sessions designed to allow attendees to get a glimpse of the most current technology in testing,” he added.
   
Among the keynote speakers at this year’s conference will be Senator Chris Dodd, Senior Senator from Connecticut, who will talk about the challenges and opportunities facing America's workers;  and, Randy Bennett, Ph.D., Distinguished Presidential Appointee, Educational Testing Service (“ETS”),  who will talk about advances in technology, which will force fundamental changes in the format and content of assessment.

Senator Dodd will talk about the roles of government, education, certification and licensure programs and new technologies for measurement and  career management.  And Dr. Bennett will discuss how education leaders in several states and numerous school districts are implementing technology-based tests for low and high-stakes decisions in elementary and secondary schools and across all key content areas. He will also talk about the significant challenges that include cost, measurement, technological dependability, and security issues, and how state efforts will need to go beyond the initial achievement of computerizing traditional multiple-choice tests to create assessments that facilitate learning and instruction in ways that paper measures cannot.

ATP Chair Cyndy Fitzgerald, of Microsoft Corporation announced that the 2003 Career Achievement Award in Computer-Based Testing will be given to Ronald Hambleton,  distinguished professor and chairperson of the Research and Evaluation Methods Program and co-director of the Center for Educational Assessment at the University of Massachusetts at Amherst.

“Professor Hambleton's research has focused in recent years on several topics of interest to ATP -- computer-based test designs, setting performance standards on credentialing exams, translating credentialing exams for uses in multiple languages and cultures, applications of item response theory, and test score reporting,” Fitzgerald said.

She also also noted that  Professor Hambleton is an author of several  texts including Fundamentals of Item Response Theory  and Item Response Theory: Principles and Applications,  as well  as having served as a Past-President of the National Council on Measurement in Education, Division 5 of the American Psychological Association, the International Test Commission, and Division2  of the International Association of Applied Psychology.

Fitzgerald urged conference attendees to check out the ATP General Meeting which will be held the morning of February 25th.  “This is the Association’s annual meeting and its a great place to learn about ATP, even if you’re not a member, to hear about accomplishments of the past year as well as goals for the future,” she said.

Sessions for this year’s conference have been sponsored by all divisions and include such titles as: The Reality of Testing Technology in the Classroom Today; Growing your Certification Program Internationally; Proven Strategies for Protecting Intellectual Property; and Unproctored Employee Recruitment and Selection.

“Without a doubt this conference keeps getting better,” said Conference Co-Chair, Don Wagner, of Pearson Professional Testing. “The attendance keeps going up each year, we’ve grown from 350 attendees three years ago to an anticipated attendance of 550 this year.  And we’ve sold out every year.”

Wagner urged individuals interested in the conference to register early in order to avoid being locked out.  Registration is available on line at www.testpublishers.org

No Child Left Behind: AAP/ATP Submit Comments

The Association of American Publishers (“AAP”) and  ATP submitted  comments in response to the Notice of Proposed Rulemaking in the Federal Register, 67 FR 30452 (May 6, 2002), regarding Title I -- Improving the Academic Achievement of the Disadvantaged, more popularly known as the No Child Left Behind Act.
 
After reviewing the proposed regulation, both Associations commended the Secretary for proposing them, and the negotiators who agreed to them. In their comments they  noted that “the proposed regulations do not require States to tear down everything they have done and start from scratch.”

Nevertheless, both AAP and ATP submitted a number of comments and suggestions regarding several of the proposed sections. In Section 200.1, State Responsibilities for Developing Challenging Academic Standards, AAP and ATP urged the Secretary to clarify, in the final regulation, when cut scores must be provided for the science assessments.  Also, to avoid confusion regarding the number of academic achievement levels a state must have, they urged that the preamble to the final regulation, and any subsequent subregulatory guidance, should clarify that a State may have more than the three levels described in the proposed rule. They also suggested an explanation that the achievement levels need not have the same names as are used in the proposed rule; (i.e., they do not have to be called “basic, proficient, and advanced”).

With regard to Section 200.2,  State Responsibilities for Assessment, AAP and ATP praised the section for its consistency with applicable Federal laws governing the disclosure of information, but argued that  the protection was not as extensive as they had originally recommended. The two Associations maintained that, in addition to protections under Federal law, protections provided for
under State law and contractual agreements should also be recognized.  Accordingly, they recommended that §200.2(b)(5) be amended to read as follows (new language underscored):

• (5) Be supported by evidence (which the Secretary will provide upon request, consistent with applicable federal and State laws and contractual agreements governing the disclosure of information) from test publishers or other
  relevant sources ….”

• Both Associations supported the flexibility provided for in proposed Section 200.3 Designing State Academic Assessment Systems, but not without some comments:

Both Associations supported the proposed wording in §200.3(a)(1), that a State’s assessments must address the depth and breadth of the State’s academic content standards; be valid and reliable and of high technical quality; express student results in terms of the State’s student academic achievement standards; and be designed to provide a coherent system across grades and subjects. But AAP and ATP noted in their comments that, “subregulatory guidance will be needed to operationalize these qualitative standards consistent with the provision.”

Both Associations supported the Secretary’s proposal in §200.3(a)(2) that a State may include in its academic assessment system criterion-referenced tests, norm-referenced tests, or a combination of the two.  But AAP and ATP added in their comments that they do not believe it is necessary to differentiate between types of tests because “any test— regardless of what is it called — that is used for Title I purposes must comply with applicable requirements.” 

And finally, both Associations strongly urged the Secretary to ensure that the preamble to the final regulation is consistent with the regulatory language as proposed.

• In Section 200.8, Assessment Reports, both Associations strongly supported §200.8(b)(2), which clarifies that the requirement to report itemized score analysis does not require the release of test items.  AAP and ATP noted that this was one of their earlier recommendations. However, they also noted that when negotiators added this provision, it was substituted for language which they believed was constructive and recommend that it be added to the proposed rule so that it would read as follows (new language underscored):

“(2) The requirement to report itemized score analysis in paragraph (b) of this section does not require the release of test items or information on each test item.”

In their conclusion, AAP and ATP reiterated there were no provisions in the proposal to which they objected, only that there were specific issues that should be clarified in the final regulation.

Letter from 2002-2003 ATP President and Chair
[Editor’s Note : Dr. Cyndy Fitzgerald, is Manager of Psychometrics and Research for Microsoft Corporation.]

Dear ATP Members,

The Board of Directors for the Association of Test Publishers met September 19-20th in Redmond, Washington. The first day of the meeting included an analysis of the drivers of success of the organization as well as the sharing of best practices. Past presidents shared their tales of camaraderie as well as the accomplishments that have occurred during the past ten years. The organization has moved from a small group of professionals collaborating within the organization at various conferences  during informal "breakfast club" meetings to the networking that has been fostered during the annual conferences hosted by ATP starting in  2000.

During the second day of the meeting the ATP Board of Directors and the Divisional Leadership Team established a number of objectives for the next several years.  There were four central themes that evolved from these sessions:

• Promote the value of testing

• Promote legal and legislative advocacy

• Develop strategies and procedures for protection of intellectual  property
  and test security

• Promote best practices in testing including an annual conference and dynamic
  web services

Please join the ATP Leadership Team by participating in the various initiatives being driven by the Board of Directors and the Divisional Officers. Thank you and I look forward to a successful year.

Sincerely,

Cyndy Fitzgerald, Ph.D.
President and Chairman of the Board of Directors
Association of Test Publishers

ATP FOCUS ON... ©opyright

Copyright Infringement -- two words that cost test publishers thousands, perhaps millions, of dollars annually.  So what is a test publisher to do?

That’s the question being posed by ATP’s Clinical Division whose members have chosen to make public awareness of copyright infringement a key component of their goals for the coming year. And yet copyright infringement is not a problem unique to clinical test publishers. Neither is it a new problem -- in the past year there have been a rash of high profile copyright abuses involving clinical tests. Last Spring a televised Midas Muffler advertisement flashed an actual Rorshach inkblot test, presumably for the entertainment of viewers. Test Publishers were not entertained.  Later in the year a California newspaper published yet the Rorshach inkblots complete with answers and psychological interpretation related to a convicted criminal. 

But test publishers report that not all copyright infringement incidents are as noticeable as these.  In fact most copyright violations are generally of a quieter, more insidious type -- the simple photocopying of test materials, scoring keys or textbooks in an effort to save money. Or the  dissemination of test questions for the purposes of cheating.  Or the disclosure of test questions to address parental concerns in an educational setting.  “We’ve grappled with the problem of copyright infringement for years and years,”  said Gregg Gillmar, Senior Vice President of Western Psychological Services (“WPS”). And no matter what the reasons, whether infringers believe they have a legitimate right to do what they are doing, or whether it is done insidiously,  he said the problem is so severe he estimates it costs the Los Angeles-based publisher at least 50-percent of potential earnings every year.

Gillmar said we are always on the look out for new ways to tackle copyright infringement, but admits it is difficult since infringers are constantly finding new ways and new technologies to carry out their deeds.  He said that encrption is employed as part of their copyright protection of computer programs, and for paper and pencil tests he reported that, whenever possible, WPS utilizes auto-score forms instead of scoring keys.   These special multi-part forms, with carbonized spaces, are not only difficult to copy, but increase speed and accuracy in the scoring process.  Gillmar noted that, “although they increase the price to the consumer, they add value at the same time.”

But while WPS is busy securing special auto score forms for their customers, other companies are peddling ways to get around the system. This year the ATP Clinical Division, challenged the running of an advertisement in a national newspaper for psychologists by a company that advertised a generic scoring program that lets test users automate the scoring of virtually any multiple choice or true-false test.

“Using an unauthorized or unlicensed scoring program to score published and copyrighted tests is unethical and illegal,” said Chris Gruber, Ph.D.,  Past Chair of the Clinical Division and  Director of Research and Development at WPS.  He,  along with ATP Executive Director William G. Harris, Ph.D.,  have entered into a dialogue with the publication’s editors and are hopeful they can influence them to take more care in their future advertising policies.

At WPS, Gillmar said, they also focus on notification. “For years we placed flyers in our test packages apprising customers of copyright law.” Though they had no way to measure the effects, he said they felt at least they were doing their part to educate consumers.

Other publishers have turned to other more aggressive solutions such as special “counterfeit proof” paper. One company, Document Security Systems of Rochester, NY offers paper with words of warning embedded in the fibers which surface upon photocopying. Other, more expensive options, defy being copied at all - they emerge completely black from the photocopying process.  Steven Morse, Manager of Marketing and Sales for Document Security Systems said his product is purchased by printers who in turn publish test materials, particularly in the educational market where copyright infringement is also of paramount concern.
   
Educational Test Publishers couldn’t agree more. “This (copyright infringement) is a really big deal for us, and we have a trademark/copyright specialist that works in our legal department. Every page of all of our tests have a copyright violation notice/do not duplicate warning on them, ” said ATP Board Member Amy Schmidt, Director of Higher Education and Evaluation Research for The College Board. 

She added that The College Board successfully sued the Princeton Review  for copyright violations when they had individuals take the SAT and then published memorized items on their website. 

ATP Board Member Carol Watson, Vice President and General Manager for NCS Assessments,  also noted that taking legal actions is always within the realm of possibility. “We have a person who focuses on this as part of her job.  She is an attorney and also handles a number of contracts issues for us.  In addition, all of our marketing and sales staff (at NCS) is on the look out for violators and forward on the information.  Our first step is generally to send a letter.  When necessary, we bring in outside counsel,” she said.

And at AGS Publishing (formerly American Guidance Service), they have an individual devoted exclusively to copyright issues.  “Our Contracts, Permissions, and Royalties Manager responds to possible issues of copyright infringement and handles such issues as granting permission to reproduce a table from one of our manuals into a text book. She works with our legal advisors...she also addresses the issues associated with software language,” said ATP Education Division Chair Kathleen Williams, Vice President of Product Development for AGS.

Williams is a proponent of having an individual who can work full time on copyright issues. “ I think having one person deal with these issues gives us both expertise and consistency. It is a valued position here at AGS for obvious reasons, ” she said.

The ATP Clinical Division, headed by   Marcia Andberg, Ph.D., of Marcia Andberg Associates, have taken on the challenge of educating test users and the public about copyright.  They are creating an area on the ATP website where information about copyright can be accessed. Additionally, the ATP Board of Directors has dedicated a budget to the distribution of educational public service announcements created by Western Psychological Services. To view this area members can go to www.testpublishers.org then to the Members Only section, click on Legal/Legislative and scroll down to Copyright issues.

ATP loses valued leader in the passing of Dr. James Adair
by Jamie Mulkey

This fall, the Association of Test Publishers lost a valued leader -- Dr. Jim Adair, Manager of Worldwide Certification Exam Development for Lotus Certification.

Dr. Adair served as Vice Chair of ATP’s Certification Division and also as Co-Chair of the 2002 Conference Program Committee.  His numerous past work accomplishments include research, teaching, book authorship and project management. He was most proud of his pioneer efforts in the development of performance-based testing. 

Members of the the Association of Test Publishers are saddened by his loss and greatly appreciate all of Jim's contributions as a professional in the field of a testing, as a leader in the Association, and as an individual who served as a role model, an inspiration, and a friend to many. 

Web-Based Test Security?
By Betty A. Bergstrom, Ph.D., Vice President of Solutions Integration, Promissor

The Test Publisher asked ATP member Promissor (formerly CAT*ASI) to address online testing security issues and how their products, among others in the marketplace, address these concerns...

For sponsors watching the evolution of Web-based tests, the suspense is growing intense -- how much will such tests change the landscape for high-stakes testing? Delivering exams over the Internet has appeal for sponsors of distance learning and credentialing exams, but many fear that relying on this method would cause their assessments to become suspect and lose their value.

This discussion will divide Web-based tests into two categories that both use the Web for delivery: the more secure browser-based tests, and online tests that lack any real security features. On the plus side, online tests can be delivered quickly and easily any time, any place and at a reduced cost. This approach is appropriate for low-stakes online tests where the sponsor has not made a large investment in test content and where cheating would have little impact or benefit. So for low-stakes tests, online testing enables sponsors to utilize many different types of testing venues.
        
 With secure browser-based testing, sponsors like the new security features being built into sophisticated testing systems. Sponsors also see this advanced testing technology as enabling them to administer their own secure browser-based tests, and in some cases decide to establish their own private testing centers.

Such testing software automates important security safeguards that protect the value of the sponsor’s assessment. The purpose of these features is to prevent the test taker from using the computer for any other purpose -- from the time the test session begins until it terminates. This keeps them from trying to research answers in other files or via the Internet, and from stealing test content by saving it to another application. Here’s how such test security functions:

Prevents access to test items before and after session; Prevents using operating system commands to access other applications; Prevents access to other applications by maintaining a full-screen test presentation window that can’t be moved or resized; Hides all navigation controls, menu options, and toolbars outside the test;  Closes test only by terminating the session.

While the test sponsor who is truly savvy about security sees locking down the desktop as most important for high stakes testing, these features can protect test content for low-stakes or practice tests in the home or office. Mid-stakes test centers offer greater security, are generally more or less dedicated to testing, have well-defined practices and procedures, and have proctors to identify test takers and monitor their activities.

High-stakes test centers go one step further. These centers use LAN servers to drive the tests, and their test administrators and proctors are employees of the center. The sponsors or vendors who provide high-stakes testing services may operate such centers themselves, or subcontract with community colleges, training centers, or other such third parties to administer and proctor the tests on their behalf. For high-stakes testing purposes, sponsors also want their test delivery technology to have features for effective test proctoring. For example, it’s important to control the timing of high-stakes tests. Testing systems with this security feature require the test administrator or proctor to authorize the start by entering a personal ID and password into the test delivery  system.

Such testing systems may also require staff to record any data the test taker submitted to confirm their identity, along with a digital photo and perhaps even a fingerprint scan.

Another important concern with high stakes delivery is how test content and results are sent back and forth between the test site and the testing system’s servers. Since sending test data over the Internet could risk theft through wiretapping, router snooping, or other means, this data should be encrypted to maintain security. In addition, it should also be stored in an encrypted and unreadable format. This way, if someone somehow obtains test data, it will be virtually useless.

For the highest level of security, the sponsors or vendors providing high-stakes testing services should have full control over the testing facility, its test administrators, and its proctors. Although this control helps prevent cheating and the theft of test content, by far the most important factor is the security provided by the test proctors. Highly effective test supervision, when combined with secure testing technology, will produce the highest levels of test security.

Ideally, such proctors should be directly employed by the sponsor or testing vendor, thoroughly trained and certified in proper security procedures, and closely supervised to ensure their performance meets defined security standards.

Conversely, using part-time proctors employed by an educational institution, nominated by the sponsor, or sometimes even by the test taker, throws into question the ability of such personnel to consistently enforce a uniform set of sound security procedures. Any substandard practices, suspicious results, or compromise of test content as a result of such testing will cause any grade, certification, or license the sponsor may grant to be considered suspect.

With the advent of secure browser-based testing technology, test sponsors now have many more choices in how their tests can be delivered. In setting up their program, they must first determine which delivery option and security level best fits the requirements of their testing program.

In all these venues, secure browser-based testing can do much to protect against cheating and the theft of test content. Also increasingly important is the role professional test administrators and proctors can play in bringing greater overall test security to any of these venues. Sponsors who want truly secure testing must find a way to gain access to an established force of testing professionals trained in following proper security procedures to defined standards.

How can testing services vendors help sponsors make these decisions?  One way is to promote awareness of the features and standards available for secure browser-based test delivery.  By doing this, testing services vendors help protect the value of our client's testing programs, test credentials, and investment.
 
Editor’s Note: Betty Bergstrom is a member of the ATP Computer- Based Testing Guidelines Task Force.

SCHOOLS INTEROPERABILITY  FRAMEWORK

In early November ATP Executive Director, Dr. William G. Harris, along with ATP’s Legislative Counsel, Alan Thiemann, Esq., convened a group of ATP members at Thiemann’s office in Alexandria, VA to discuss the Schools Interoperability Framework (“SIF”) and its importance to the test publishing community. 

“The SIF standards will choreograph the ways in which products communicate with each other and how the data from these products are integrated into common databases.  These standards are directed at products for the K-12 educational market,” Harris explained.

Many ATP members who provide educational or clinical  assessment tools or services or plan to provide assessment tools or services under the No Child Left Behind Act (“NCLB”)  attended the two-day meeting.

Facilitators for the meeting were Tim Magner, Director of SIF and Alan Thiemann.

“During the past several months, ATP has been involved in discussions with the Software and Information Industry Association (“SIIA”) which administers the SIF -- a project which we believe could have major ramifications for test publishers – both of educational and clinical products used in schools,” Thiemann explained. He noted that, “the SIF is an ‘open standards group’ – which means that the technical standards it is creating are available for everyone to use without charge.  It is also being viewed by the US Department of Education as a means for submitting student data in compliance with federal laws.”

Initiated in 1999, SIF now has more than 120 member companies, organizations, and governmental entities working together to develop standards using eXtensible Markup Language (“XML”) and Internet communication protocols to share data efficiently and securely among software applications so that products utilizing different languages and platforms can talk to one another.  This interoperability between different vendors’ products allows educational customers to save time and resources in managing their information needs, as well as to choose technology products with the confidence that they will integrate with existing products.  SIF is also working to develop a Certification Program that will provide third-party validation that a vendor’s products are SIF compliant and thus ensure customers that the software they purchase correctly adheres to the SIF standards.  As a result of these activities, many state and local K-12 educators are looking to SIF as the primary vehicle to address their interoperability and integration goals.

“Stakeholder input is vital to the development of the SIF standards.  Many vendors, including some test publishers, already have been participating in this effort.  However, ATP believes that it is important to provide its members with a coordinated avenue for participating in the evolution of the final SIF standards – to assure that they meet the needs of as many test publishers as possible and will work for as many products as possible,” Thiemann said

As a result, he reported that he (Thiemann) and Harris have been discussing the standards and the process for adopting them with SIF’s Director, Tim Magner. 

As part of an invitation sent out to ATP members, Thiemann provided a summary of relevant areas of the proposed SIF standards and information on how they are likely to be used. He added that it also provides input obtained from an SIF Instructional Services Working Group which met in September, and included a discussion of how the SIF standards will dovetail with the No Child Left Behind Act.

Thiemann noted that there are two main SIF documents that need to be reviewed:  SIF Implementation Specification, dated May 22, 2002; and SIF Draft Data Objects Specification, dated July 31, 2002.

He said the Implementation Specification represents the second revision of this document.  It contains a number of guides for test publishers to use in deciding how their products will fit into the SIF model for actual use.  SIF is developing a set of XML plans for organizing data (so-called “schemas”), including identification of field lengths or whether to require fixed or variable lengths. 

The Draft Data Objects is a proposed set of data objects that do not yet contain any  Document Type Definitions  (“DTDs”) for putting data together; rather they are comprised of specific data elements whose use is mandatory, optional, or conditional.  NOTE: SIF has indicated that once all data flows have been identified, then it will develop appropriate DTDs and schemas. 

Although the draft objects include (or eventually will include) many different areas such as library, transportation, food service, etc., the existing draft objects of highest interest to test publishers are found within the Instructional Services Working Group.  Each object has a unique global identifier so no two objects are alike. Thiemann also noted that there are several objects in other work areas that may be relevant to test publishers as well. 

Thiemann and Harris noted that, on a related front, “we wish to point out that SIF is in discussions with IMS Global Learning Consortium (‘IMS’) about how to integrate their standards with the SIF proposals.  IMS has developed the Questions and Testing Interoperability standards (“QTI”) for use with higher education and training applications.  These standards are available for review at www.imsglobal.org/specifications.cfm.”

Thiemann also noted that, “although the IMS standards are lengthy, they were developed by participants with knowledge about assessments in areas other than K-12 and thus contain many elements that may be necessary for ATP-member assessment products.  ATP believes it is critical that there should be a single shared standard – not two distinct sets of standards – for companies that publish tests in two or more education areas.  Trying to conform to two standards would create difficult problems and tend to negate meaningful efforts at standardization”.

In regard to the meeting in Alexandria and whatever subsequent steps follow, Harris stated that,  “Our goal is to ensure that ATP members can actually use the final SIF standards. We have been told that our input will be considered before any final standards are released.”


NOTE:  Both documents, in their entirety, can be accessed through the ATP website located at www.testpublishers.org. The documents reside on the “Members Only” page.  To reach them click on “Members Only” found on the left sidebar and then on Schools Interoperability Framework.

Association Notebook
ATP Welcomes as Members...American Printing House for the Blind, American Testing Authority, CASAS, EDAC, EMC, Gershongroup, Lamark, Marcia Andberg Associates, Measured Progress, MetaMetrics, nfer-Nelson, Pacific Metrics and PEN.

Next Board Meeting... Feb. 26th -27th, Amelia Island Plantation, Amelia Island, FL. The meeting will be held 3-6:00 p.m. on the 26th and 8:00 a.m. - 5 p.m. on the 27th. Members can have business placed on the agenda by emailing the ATP Board of Directors c/o LScheibatATP@aol.com

New Website Page for Member News...check out the ATP web site at www.testpublishers.org and click on member news for press releases and announcements from ATP members.  Also on the website check out the new Press Room and also the new page for copyright issues. Go to the Members Only  section, click on Legal/Legislative and scroll down to copyright issues .

ATP Gratefully Acknowledges its sponsors for the  2003  Conference Technology in Testing: Application and Innovation

PLATINUM
 ACT, Inc. • American Testing Authority, Inc. • The Chauncey Group International • Educational Testing Service (“ETS”) • Galton Technologies, Inc. • Integral 7, Inc. • Pearson Professional Testing • Prometric, a part of the Thomson Corporation •Promissor, Inc. • Questionmark Corporation • TestOut Corporation 

GOLD
 Applied Measurement Professionals, Inc. (“AMP”) • BrainCore.Net, LLC • Buros Center for Testing • i-asses.com • Professional Credential Services, Inc. • Vantage Learning • Castle Worldwide

SILVER
The College Board • Performance Assessment Network (“PAN”) • LaserGrade

The Test Publisher newsletter is posted with the understanding that the content of the newsletter does not constitute the rendering of legal, accounting or other professional opinions.  If legal advice or other expert assistance is required, the services of a competent professional should be sought.
 

©1997 - 2003 Association of Test Publishers