©2000 Association of Test Publishers
Save or read as pdf file

LaCosta Resort and Spa site of 2002 ATP Conference - Computer-Based Testing: Assessment at your Fingertips

The Association of Test Publishers (ATP) will host its’ third annual industry conference - Computer-Based Testing: Assessment at Your Fingertips, February 4 - 6, 2002 at the La Costa Resort and Spa in Carlsbad, CA (north of San Diego).
   
 “This year’s event includes pre-conference workshops, general sessions with informative presentations, concurrent sessions on relevant topics, and networking activities,” said ATP Chair Jamie Mulkey, Ed.D.,  of Hewlett Packard.
   
“This year’s conference promises to be an exciting, informative program which will facilitate the exchange of ideas across many segments of the test publishing community,  while at the same time offering the relaxing atmosphere of the La Costa Resort and Spa,” Mulkey added.
   
A preview of the conference agenda includes content sessions such as: Business Implications of Building and Maintaining a Successful Test Program,  New Trends and Tools, Test Development & Management,  as well as sessions with a specific focus on Certification and Licensure, Education, Industrial / Organizational and Clinical areas of testing.
   
Highlights of the conference include key note presentations: Dealing with the Media in High Stakes Testing by  David W. Smith, CEO and president, NCS Pearson and David Hakensen, vice president, public relations, NCS Pearson;  And,  Distributed Simulation: The Need to Test for Teamwork Skills Using Technology-Based Assessments by Dr. Jack Thorpe a U.S. Air Force consultant.
 
Additionally,  ATP plans to recognize Benjamin D. Wright, Ph.D. with the Career Achievement Award in Computer-Based Testing. Dr. Wright is Professor of Education and Psychology at the University of Chicago and Director of the Mesa Psychometric Laboratory. He has published 150 papers on Rasch measurement; authored 12 books, including Best Test Design and Rating Scale Analysis and developed the two most widely used Rasch measurement computer programs, WINSTEPS and FACETS.
   
Mulkey also noted that ATP will hold its yearly General Meeting on the first afternoon of the conference. She emphasized that the meeting is open to all attendees of the conference, not just ATP members. The General Meeting will focus on legal issues in test  publishing and will include an opportunity for the individual divisions of ATP to meet and plan goals for the coming months.

LEGAL UPDATE
Arizona court decision favorable for test publishers... by Alan J.  Thiemann, Esq., ATP Legislative Counsel

On November 27, the Arizona Court of Appeals issued an Order affirming entirely the decision of the Superior Court in Phoenix Newspapers, Inc. v. Lisa Graham Keegan, State Board of Education, and State Department of Education, Case No. 1 CA-CV 00-0284.  This ruling means that “anchor” items in one form of the Arizona Instrument to Measure Students (“AIMS”) do not have to be released to the newspaper under the Arizona Public Records Law.  This result ensures that Arizona’s state assessment program does not have to expend additional money to build new tests as would have occurred if reusable “anchor” items had to be released.  Therefore, this decision represents a win for Arizona and for the Association of Test Publishers, which filed a brief in support of the state as amicus curiae.

Although all three judges affirmed the lower court decision, Judge Gerber wrote a separate opinion that largely adopted the position taken by ATP – namely, that test security requires all test questions should be kept private to avoid an “Alice-in-Wonderland approach to public education” where students’ performance on disclosed questions reflects what they were able to memorize.  Judge Gerber also was troubled by the possible precedential effect the Order might have on public disclosure of other state tests, which he feared “opens a Pandora’s box …, intruding into an arena best left to the legislature and forcing the courts to use a newly-honed Ockham’s razor to sever complex from less complex test questions, a task better left to educators than judges.”  Finally, Judge Gerber offered a light-hearted conclusion to his opinion, noting that he had learned a personally useful academic lesson from the case: “The next time I face a difficult test, I can sue to discover the test questions in advance.”  We think Judge Gerber’s observations are absolutely on target and they may provide us with some useful ammunition in other similar cases down the road.

The ATP brief raised issues we felt were significant to the testing industry:

1.  Test items are not “records” under a state public records law, only the
   test results;

2. There is no valid distinction between “anchor items” and
   any other items on a secured test intended for reuse;

3. A  “trade secret” existed for all test items on Form A under the contract with the original
   test publisher and the state, which could not be ignored in favor of public disclosure under the Public Records Law; and

4. Many other state tests equally would be affected by the ultimate decision, including all other forms of the AIMS, the SAT 9, high school decree substitute exams, college entrance exams for entrance into state universities, and a host of other public/governmental tests. 

Virtually every one of these issues was ignored by the Court of Appeals, based on the very limiting facts of this case. 

The Court of Appeals affirmed the original decision based on the very specific facts of this case – the admission by the state that test items are
subject to the Public Records Law; the limited dispute to one form of the test, rather than all similar tests; the determination that the limited inspection process waived any confidentiality with the test developer; and the State’s decision to release some additional items after the superior court’s decision (while still arguing it should have been able to protect them).  All these factors influenced the three appellate judges to affirm the lower court ruling, rather than seek to deal with any of the “tough” issues raised by  ATP. 

The fact-specific nature of the decision means the case will have little precedential value.  However, ATP’s involvement in this case has well
prepared us for future state issues on test security and release of test items – and the decision contains some useful language for dealing with those issues.  

ATP Hosts First Webinar: Introduction to Item Response Theory

More than 50 ATP members and nonmembers had a virtual learning experience this fall  as attendees of ATP’s first successful webinar.  On September 19th and again on October 16th, Ronald Hambleton, Ph.D.,  from the University of Massachusetts at Amherst, taught two 2 1/2 hour classes on Item Response Theory (“IRT”) -- and none of his students had to leave their home computer to get there.

Dr. Hambleton taught his classes in cyber space and his students tapped into the lecture via the internet and using the support of the Hewlett Packard Virtual Classroom. Students were able to hear Dr. Hambleton’s lecture through a telephone bridge line and they were able to supplement their notes with down loadable slides and ask question via the telephone. Topics covered by the webinar included descriptions of  
different IRT models, the concepts of unidimensionality and item characteristic functions, approaches to estimating model parameters, the
application of IRT to computer adaptive testing and descriptions of some popular IRT software.

The Webinar had one sponsor, the -- The Donath Group -- who were entitled to a free registration and slides announcing their sponsorship and introducing their company  to webinar attendees.

Attendees paid between $150.00 and $195.00 depending on their status as members and/or early bird registrants.

The ATP Board of Directors has appointed a committee to explore future webinar opportunities.

Webinars : Frequently Asked Questions

 Q: How do I attend a Webinar?
A: You log on to a virtual classroom using a URL address.  In the case of the ATP Webinar, attendees logged into the Hewlett Packard virtual classroom located at www.hpe-learning.com. Important :  You should always log on ahead of time to make sure you can get in the classroom.

Q: Are there any specific PC requirements?
A: No, there are no PC requirements other than having Windows Internet Explorer 4.x or higher or Netscape 4.x or higher.

Q: Do you have any requirement for browser versions or sound functionality (Real Audio, Windows Media Player, etc.)?
A: No, you will also be dialing into a telephone bridgeline. So you will need two lines: one for your phone and either a LAN connection or fast phone line connection for your computer.

Q: Can I print any of the material?
A: Yes, you are able to print the slides that are shown.

Q: What kind of visibility does a sponsor receive?
A: Depending on the terms of the sponsorship agreement, sponsors can display slides or sets of slides at pre-determined times during the webinar.

ATP Gratefully Acknowledges its Webinar Sponsor THE DONATH GROUP

ATP FOCUS ON...

Probably very  few ATP members have seen as many tests roll out their delivery doors as Von Hoffmann Graphics (formerly Bawden Printing) -- and yet do not fall into the category of a Test Publisher.  For  more than 30 years Von Hoffmann has been the final stop on the way to publication for millions of tests.  Although they are not officially a “test publisher”  they have been a longtime supporter and Associate Member of the Association of Test Publishers.
 
“Obviously our number one reason for being an ATP member is to be supportive of the industry and our major customers, but we also need to learn and stay on top of where the industry is headed for our own strategic plans,” said Mark Bawden, Senior Vice President of Customer Relations for Von Hoffmann.

And knowing where the industry is headed has been a longtime interest of Von Hoffmann. Von Hoffmann began test manufacturing as a small plant in Iowa City, IA printing a daily newspaper and running a distribution warehouse.  But the company’s location in Iowa City, where standardized testing and scoring technology was developed in the '60’s, soon lead it to customers such as NCS Assessments (formerly MRC),  ACT and other large profile test publishers.  Bawden attributes Von Hoffmann’s success at expanding into the test printing business as one part business savvy and the other part good luck.  “We were in the right place at the right time,” Bawden said. “But we also knew an exciting opportunity when we saw it.”

At present, Bawden  reports that Von Hoffmann Graphics utilizes twelve heat-set eleven inch presses, each capable of producing over a million two-color pages per hour.  And he noted that the company has recently expanded its test printing capability at its Owensville, Missouri location.  This plant now boasts upgraded technology in every area of test production  from printing to binding to distribution.  "Now all of the same press bindery, distribution and security services formerly offered at the Eldridge (IA) facility are available in the Owensville location. Our goal is to double our capacity in the test production and distribution area,” said Bawden.

But printing tests is not all there is to the business, as any  test publisher will attest --  security is a number one concern.  Bawden points to
Von Hoffmann’s obscure location in Eldridge and the company’s recent expansion of its testing capability into out-of-the-way Owensville as
beneficial to the creation of secure printing environments.  And aside from location, Von Hoffmann has an impressive array of security capabilities -- from bar coding and automated tracking to scrap shredding and bindery  scanners to 24-hour surveillance -- capabilities that most other printers never dream of needing in their facilities.  “High stakes testing means that more and more we need to account for every single copy of every single test -- which means bar coding and tracking every book, every package, every carton and every skid,” Bawden said.    

And while Von Hoffmann is understandably  concerned about the trend toward computer based testing, they do not foresee any significant decline in the educational paper and pencil  testing market for at least the next decade. Educational testing in particular is on the rise, Bawden said, due in part to the push from federal  and state government for accountability in the schools.  Von Hoffmann is closely watching legislation that President Bush would like to have passed by year’s end.

At the same time, Bawden said, computer technology has served to enhance the printing business.  He pointed to Von Hoffmann’s ability to offer computer-to-plate prepress capabilities in all of their plants to meet the growing demand for producing plates directly  from electronic files.  This process eliminates steps, saves time and enhances accuracy and quality.  It also facilitates the ability to move work easily between the Eldridge and Owensville plants as customer demand dictates.

“Our test publishing customers have unique needs -- but it’s Von Hoffmann’s goal to continue to meet those needs, as we have for the past 30 years, into the next century,” Bawden concluded.

Editor’s Note: For more information on Von Hoffmann Graphics go to www.vonhoffmann.com

ATP continues dialogue with Federal Department of Labor’s Employment and Training Administration (“ETA”) on the subject of O*NET

On November 29th ATP Executive Director William G. Harris attended an O*NET Managing Partners meeting.  This was his report on that meeting...

My time at the meeting was confined to a working lunch.  It gave me, however, adequate time to discuss the three key points that emerged from our November 16th meeting in Chicago.  The three points were:  (1) the O*NET managers needed to improve their effort to communicate and engage test publishers about O*NET business opportunities, (2) the managers needed to devise a clear plan of prevention and outreach to avoid the misuse of the ability profiler and the self-assessment tools as “de facto” selection devices, and (3) the managers needed to provide assurance that there is a commitment over the long term to the maintenance of high quality DOL/ETA employment databases.

On communication, there was no disagreement that the O*NET managers previously acted in a manner that made, at best, cooperation with test
publishers ineffective.  We agreed in principle to explore ways to improve communication and to engage test publishers in future O*NET activities when possible.  As part of the communication effort, we discussed website links, articles in the ATP newsletter and online journal, and presentations at ATP and O*NET meetings and conferences.  The more O*NET officials can invite test publishers to be active participants in O*NET’s ongoing development, the more likely we will recognize and incorporate the O*NET features into our products and development activities.  Some O*NET officials have an appreciation of the potential value of working with us and they appear to be directing O*NET’s approchement with us. 

On preventing “de facto selection creep” from occurring with the ability profiler and the self-assessments, it was clear that this issue received
little, if any, attention during the planning and development phase of the assessment tools.  This “oversight” was an embarrassment, especially to the North Carolina group which has the operational responsibility for O*NET.   Also, this oversight has highlighted why it was a poorly formed decision to exclude ATP from early DOL/ETA planning meetings that discussed attaching assessment tools to the O*NET.

My concern about selection creep centered largely on the ability profiler.  As noted at the Chicago meeting, the ability profiler has not been released.  It has a scheduled release date for the first half of 2002 calendar year. 

At the November 29th meeting some of the attendees readily recognized that the ability profiler and the other self-assessment tools might be misused.  Training and outreach are planned for One-Stop personnel, but apparently the issues that concern ATP are not well covered in the O*NET Academy Training program.  ATP is considering a proposal from the Office of Policy and Research to develop separate training  modules on the proper use of the ability profiler and the other tools, the difference between self-assessment and selection tools, and on the legal importance of using assessment tools for their intended purpose.  Further discussion of ATP involvement in the development of the training modules is slated with O*NET officials in January.

The final point pertains to the importance of DOL/ETA ensuring that the O*NET databases will receive ongoing improvement.  The commitment to high quality O*NET databases is paramount to O*NET’s effort to heighten test publishers’ interest in O*NET’s commercial value.  Put differently, O*NET’s importance is directly tied to the quality of its databases and to O*NET’s long-term plans to refine and enhance its databases.  The attendees fully accept this point and they recognize that the ability to create and maintain world-class databases depend on making them valuable to test publishing and other relevant sectors of the market.

In the coming months ATP will meet with O*NET officials to discuss in detail the proposed O*NET training modules.  O*NET officials are still interested in ATP reviewing the ability profiler and providing written feedback.  I anticipate scheduling a second meeting in Chicago during the March-April timeframe.  I  ATP members to visit the O*NET website and to become familiar with the its content.  A careful survey of the website will facilitate our future meetings and activities.

Editor’s Note: O*Net can be accessed at www.doleta.gov/programs/ONET

Hewlett Packard Certification Program Manager leads ATP as 2001-2002 Chair

Benjamin D. Wright, Ph.D. honored with the Career Achievement Award in Computer-Based Testing

Benjamin D. Wright, Ph.D. will be recognized at the ATP  Conference as this year’s recipient of the Career Achievement Award in Computer-Based Testing. Dr. Wright is Professor of Education and Psychology at the University of Chicago and Director of the Mesa Psychometric Laboratory. He has published 150 papers on Rasch measurement; authored 12 books, including Best Test Design and Rating Scale Analysis  and developed the two most widely used Rasch measurement computer programs, WINSTEPS and FACETS. Dr. Wright has taught annual workshops and courses on Rasch measurement since 1969 and chaired 70 doctoral dissertations on that topic. Many of his students are contemporary leaders in psychometrics.

Editor’s Note: At this writing Dr. Wright is recovering from serious illness. Our thoughts are with him and his family at this time with best wishes for a full  recovery.

Association Notebook

ATP Welcomes its newest members...J.D. Edwards & Company, Minds In Action, PEN, Psychometrics Canada, Ltd., Vantage Learning and VIP Tone, Inc.

The next board meeting of the Association of Test Publishers...will be held Thursday, February 7, 2002 at the La Costa Resort in Carlsbad, CA.  Members can have business placed on the agenda by emailing the ATP Board of Directors: laurens@testpublishers.org or wgharris@testpublishers.org.

The next General Meeting... will  be held Monday, February 4, 2002 at the La Costa Resort in Carlsbad, CA(just north of San Diego). For registration information go to the ATP website at www.testpublishers.org.

ATP Congratulates... those individuals elected (or re-elected) to the Board of Directors: Cyndy Fitzgerald, Microsoft Corp, James Olsen, Alpine Media, Amy Elizabeth Schmidt, The College Board, Carol Watson, NCS Assessments and Charlie Wonderlic, Wonderlic, Inc. 

ATP’s Public Relations Committee needs material...Do you or your company have Public Relations materials that convey the value of testing to society or give information about proper test use, development or delivery that you would be willing to share?  The newly formed ATP Public Relations Committee is seeking material for the new “press room” located on the ATP Website. If you have any great papers, brochures, presentation materials, books or booklets that you would be willing to have posted on the website to help ATP in getting a positive picture of testing out to the media and the public, please let us know. Email laurens@testpublishers.org.

DIVISION NEWS... as reported by division leaders at the October Board Meeting

Education -- The Education Division held a successful conference in the spring at the annual meeting of the American Educational Research Association (“AERA”) held in Seattle.  At that time Kathleen Williams of the American Guidance Service (“AGS”) was elected as Vice Chair of the Division.  However, in June, ATP Education Division Chair Amy Schmidt, Ph.D., of the College Board was then elected to the ATP Board of
Directors.  Therefore Williams became  Education Division Chair.  At the ATP Board Meeting in October Williams outlined division goals for the coming months: Goal I was to increase communication through email and division meetings; Goal II was to recruit new members; Goal III was division involvement at the 2002 Computer-Based Testing conference.

Industrial -- The Industrial Division held a successful conference in the Spring at the Society for Industrial Organizational Psychology (“SIOP”)
convention.  Industrial/Organizational Division Chair Michael Segovia of Consulting Psychologists Press reported that members of the I/O division had participated in the review of the Computer-Based Testing Guidelines and been involved in conference planning for the 2002 CBT conference. Segovia reported that division goals for the coming months included: increasing the division’s attendance at the 2002 CBT conference, addressing division members’ public relations needs and encouraging the exploration of copyright protection issues.

Certification -- Certification Chair Linda Waters of the Chauncey Group International reported on division goals for  2002. She said  Goal I would be to promote and develop ATP's CBT conference and to increase attendance, as well as presentation topics. She reported that Goal II would be to increase Certification division membership; Goal III would be to assist with review, publication and dissemination of the CBT Guidelines;  Goal IV would be to provide effective information / communication flow among the Certification Division members; and Goal V would be to respond to the changing economic climate by  identifying the impact of economic downturn on certification business, provide position statements to employers and prospective candidates regarding value of certification and survey the membership regarding changes
in certification services.

Clinical -- The Clinical Division held a successful meeting at the annual convention of the American Psychological Association (“APA”) in August.  Due to a change in career path, Clinical Division Chair Pamela Becker Dean, resigned from her position.  Vice Chair Chris Gruber, Ph.D., of Western Psychological Services is now Chair of the Division.

Gruber reported that last year there had been dissatisfaction among Clinical ATP members over the the APA convention.  Division members were surveyed for their views and their suggestions and Division leaders met with APA leaders and staff.  This year, Gruber reported many of the suggested changes were implemented.    He reported that division goals for the coming months included periodic member updates and increased involvement by clinical division members in the February conference.  He added that he would like to see ATP implement some
copyright programs and that he would be soliciting input from division members on the release of a new APA policy on “Release of Test Data”.

Editor’s Note:  ATP’s White Paper-- Test Disclosure Guidelines  is available on-line under the Inside ATP section  of the ATP website at www.testpublishers.org