Keynote speakers, from a legislator to an academician, to a business owner, share a variety of messages...

Senator Dodd: Improving the Competitiveness of America’s Workforce

Calling the United States a leader in the development of technology, Senator Christopher Dodd (D-CT) cautioned attendees at the fourth annual ATP Conference that the U.S. will not remain so unless it ensures that the next generation of Americans has the education and the skills needed to compete not only in the U.S. workplace, but in the global  economy as well.  And he called upon test publishers to insure that their assessments work to assess accurately  these skills.

In his address, Dodd noted, "many public schools are failing and many individuals are unemployed because they lack the skills for the new economy." He pointed out that jobs once open to an individual with less than a high school diploma have become technologically advanced.  "The UPS delivery man shows up at the front door carrying a clipboard equipped with more technology
than we used to put a man on the moon," he quipped. On a serious note he added, "we (policy makers and citizens)  bear a collective responsibility to insure that everyone has an opportunity for success."

Dodd praised the No Child Left Behind Act  signed into law by President Bush last year, but at the same time cautioned that "testing, in and of itself, is not an educational policy." He likened excessive testing in schools to "taking the temperature of a sick person over and over again" which, he said is not going to make them better.  "I wish I could say that the federal government is poised to provide a better education -- but millions of children face the possibility of being left behind.  That is the reality." Dodd noted.

Dodd urged Americans to reflect on the federal government’s strong history of support for public education dating back to the Civil War. He noted that it was during that time of national strife, that legislation was passed putting land aside for the purpose of building public schools. "During some of the most critical moments in history,  leaders recognized that education was crucial," Dodd said.  He went on to liken those days of past strife with today’s world situation. "We are now in the midst of a war on terrorism, our
economy is in a turn down, state budget deficits exceed millions of dollars, and there is a great temptation is to cut back on public schools," he stated.
   
Dodd was critical of private and parochial education which he said, is not the answer, "The thought that we can walk away from our public education obligation is ludicrous. We can’t shut down failing schools on the assumption that there is a nice little parochial school down the road that is going to fill in the gap."

At the same time Dodd admitted that public  education in the U.S. isn’t equal. "There are," he said, "tremendous disparities." He noted that in his home state of Connecticut, Hartford is the poorest city in America, while the state of Connecticut is one of the most affluent.  And the difference in educational opportunity is a stark 15 minutes apart.  He reported that 36-percent of the inner city teachers in Hartford are not certified. And yet quality education is available in outlying areas.

Dodd was also critical of the state of  adult education in the U.S. and of the administration’s budget which he said cut $622 million for job training programs. "The most successful businesses are those that understand that education is a lifelong endeavor," he said.
 
Among his solutions to these problems Dodd called for increased emphasis on a public private partnership with corporate America. "I can’t see that we can continue to depend on property taxes and think that equal opportunity is going to be afforded.  We need more participation from the corporate sector," he stated.  And as for test publishers, Dodd said the challenge would be to create assessments that accurately report the state of affairs of education in the United States today.

"We’re all in this together -- this is a challenging time for all of us.  Events overseas are adding additional pressures. But if we need to make the proper investments in our educational system -- not just elementary and secondary -- but all the way up to job training," he concluded.

Randy Bennett:  Inexorable and Inevitable: The Continuing Story of Technology and Assessment

"The advance of technology is inexorable and therefore the use of technology in assessment is inevitable... and we (test publishers) must keep pace."  -- That is the message Randy Bennett, Ph.D., distinguished presidential appointee, Educational Testing Service ("ETS") brought to attendees of the ATP conference.  

Bennett, whose keynote address opened the second day of the conference,  outlined the state of technology education taking root across America. He reported that 35 states now have virtual universities to deliver or promote distance learning and that student enrollments at some online institutions have already reached traditional on-campus levels. He reported that as of Sept. 2001,  86-percent  of students between 18 and 24 years of age are using a computer in school and that most college students ( 73-percent) say they use the internet more than the library for  research. Bennett noted that at the kindergarten through grade12 level, at least 14 states have established virtual high schools and 90-percent of Kindergarten through grade 12 teachers report using the internet as a teaching resource.  He said that 75-percent of students ages five through nine years of age, and 85-percent of students 10 through 17 years of age use a computer in school. He also reported that 26 states require technology training for teacher licensure.

And in the most recent proof of the advancing status of technology in education, Bennett noted that the No Child Left Behind Act seeks to explicitly improve student achievement through the use of technology.

But advancements, Bennett cautioned, do not come without issues and challenges. "e-learning has (to-date) no strong research base to support its effectiveness," he said.  And he pointed out that many distance learning companies have failed. " The dot com crisis," he reported, "hit education too."  But perhaps the greatest barrier to innovative teaching, Bennett said, is that today’s assessments continue to measure yesterday’s goals. And, he added, "they continue to use yesterday’s paper and pencil technology to do it."

Bennett reported that several states are actively working to switch to computer technology in assessments. "But no matter how the issue is looked at, technology assessments are higher cost than paper and pencil," Bennett said.  He added that the high cost of implementing the No Child Left Behind Act  at the same time that states are dealing with large deficits makes technology assessments prohibitive for most. 

Another challenge, noted Bennett, is that time-lines for the No Child Left Behind Act  are extremely ambitious. "By 2005-06 states must be able to assess students in reading and math on an annual  basis.  This is ambitious just for pencil and paper," he said.
In addition to cost, Bennett listed a number of other challenges including measurement and fairness issues, comparability across delivery modes, comparability across platforms, comparability across students with respect to standards and issues related to security.

Another area in which schools are finding challenge is in equipment software and network dependability. "Ongoing maintenance and troubleshooting are an issue for schools," Bennett reported.

But despite these seemingly daunting issues, Bennett predicted that the forward progression of technology in education will, in fact, be spurred by the same catalysts that offer the most challenge. "The No Child Left Behind Act demands yearly progress or else," Bennett said.  Thus, he conjectured, school officials are going to need to know how students are doing at the start of the school year and throughout the year. "Computer assessment can give immediate results as well as give benchmark and formative information," he said.

Bennett noted what he felt were "trends to watch."  He included on this list:  inexpensive, wireless devices, more powerful delivery of technologies, more powerful automated scoring technologies, more informative reporting technologies, as well as the emergence of a science of assessment design.

On a conclusionary note,  Bennett cautioned those in the testing field that delivering traditional tests electronically must only be a beginning. "If we’re not careful we might just put into place the ability to make bad tests faster and better," he said.  In the pursuit of good tests and good testing practices Bennett concluded, "we (test developers) must be inexorable."  

Charlie Wonderlic: So What About Technology?
 
"We ARE test publishers,"--  that was the final emphatic statement made by Charlie Wonderlic, president of Wonderlic Inc., as he wrapped up both his keynote address and the 2003 ATP conference with a humorous poem he authored for the occasion. In it he summarized the struggle of his, and other test publishing companies,  as they attempt to keep pace in the fast lane of technology, while at the same time retaining their identity as quality test publishers. 

Prior to reciting his poem, Wonderlic reflected on his own journey into the industry of test publishing as well as that of his family’s business.  Wonderlic completed his formal education in the mid-80s, graduating with what he said was one clear conviction, "I didn’t want to go into the family business." Testing, he explained wasn’t really "hot" at the time and sales were low. The company, founded by his grandfather, brought in about  400k annually.  Nevertheless, Wonderlic gave the business a try and within 3 years was able to triple sales to 1.2 million.  "And I didn’t know what I was doing!" he quipped.  Wonderlic attributed much of the success to the introduction of new marketing techniques and the growing popularity of testing.  Ten years later, he said, Wonderlic, Inc. was bringing in nine million in annual sales.  Reflecting on his past success and the success of the industry, Wonderlic advised both those entering the testing industry for the first time, as well as those with experience, to make sure they remain open to change. "Be careful not to build a box around what you do. Explore new ways - (utilize) customer surveys, experiment, diversify offerings," he said. 

Wonderlic noted that the industry had weathered change in the past.  He said the big issues of the ‘60s and ‘70s were civil rights, and that the 80s, were characterized by recession. "We have a long legacy that we should be very proud of," he said.

At present, Wonderlic said, as a company and as an industry, they are being presented with challenges - some new, some that have been faced before.  "We are in the middle of a recession.  Our sales have gone down and in some cases significantly enough that we’ve had to take severe actions that we’ve never had to take before such as laying off people."  He noted that as a company they are retrenching. "We’re focusing on operating expenses and on making sure we survive."

But Wonderlic also noted that retrenchment is not a substitute for strategic planning. "Retrenchment will buy us time, but not a future." And the future he admitted is daunting. " All I can count on today is that everything is changing.  I’m not sure I have the time, the intellect and the resources to keep up with it... any consumer can get on the internet and find out in two clicks what (products) are out there, what (they) cost and what their options are. Anyone who prices their product too high for the value is going to be gone," he said.

But at the same time Wonderlic echoed the optimism that has brought his company as far as it has, "A wonderful opportunity has happened!" he exclaimed.  And the task for test publishers in the new millennium will be, "to figure out how to leverage the internet." 

 Wonderlic said that as a business owner the only decision left was whether they would insource their technology or outsource it.  But either way, they would seek to leverage it, "Technology and innovation are not merely an option anymore, they’re a prerequisite to our future."

ATP FOCUS ON...

This year’s winner of the ATP Career Achievement Award:  Dr. Ronald Hambleton

Research, and lots of it, is what Ronald Hambleton, this year’s recipient of the ATP Career Achievement Award in Computer-Based Testing, prescribed for the testing industry. "Lets do the research and then move forward with assessment innovations based on research findings - to do anything other would be a denial of all that we have learned about constructing valid assessments," he said.
And if anyone knows what it takes to construct valid assessment , it is Dr. Hambleton who, as distinguished professor and chairperson of the Research and Evaluation Methods Program and co-director of the Center for Educational Assessment at the University of Massachusetts at Amherst, has spent 37 years in the area of psychometrics.

Hambleton, upon accepting this year’s Career Achievement award, outlined his concerns and cautions for the testing industry.  He noted that although his little lap top has more (technological) power than the entire University of Ohio had with 56,000 students in 1966 -- test developers still don’t have all the answers when it comes to accurate and valid assessment. "I am suspicious of any company that says ‘We’re the one who measure what people know with absolute certainty’ or use slogans such as ‘The company with all the skills to get it done brilliantly’," he said.

Hambleton went on to describe several areas where testing still struggles with major misunderstandings.  He said that despite the burgeoning area of international testing, test developers are still struggling with the best methods to properly translate tests into numerous languages without losing validity in the process "There are major misunderstandings about the difficulties of translating and adapting exams from one language to another," he noted as he gave several examples.  He said the American phrase "out of sight, out mind" translated into French comes out as "invisible, insane."  In another example he said American and Finnish students were both asked if the words pessimistic and sanguine were similar in meaning.  But because the re was no equivalent for sanguine in Finland, Finnish students were given an obvious opposite alternative and wound up outscoring American students 2 to 1.

Such problems occur, said Hambleton, when the assumption is made that any translator is qualified to translate exams and that only one type of translation process is required.  "(Test developers) need to use forward and backward designs to review tests and compile empirical evidence to address construct, method and item bias...that and hire qualified translators," Hambleton said. 

Hambleton said that despite advances in modeling of test data, questions remain regarding model choice (both in fit and practicality) and calibration of items.

In the area of item format, Hambleton noted that today there is perhaps lots of hype, but insufficient research with regard to cost/benefit analysis and other concerns.   " (There is) lots of sizzle here with simulations, virtual reality, push back features, and performance tests... being able to put a medical student in a pair of gloves and feel organs , some hard, some squishy...  But if you can only get a few questions and answers... within 15 minutes -- it’ll never hold up in court," Hambleton noted.

Hambleton noted that test security, always an issue within the testing industry, is now a major concern, especially in light of the sheer numbers of items that are required for valid computer based testing, "It only takes one person to put that item on the Internet and share it with the world," he said.

And finally Score Reporting. "How can reports be made more understandable and more informative to candidates and users?" Hambleton gave the example that  a large percentage of the population does not understand the scoring of a simple IQ tests. "You’d be shocked to know how many people don’t know the difference between a percent or a percentile and how may people actually believe that 70 on an IQ test is passing and 100 is perfect!" Hambleton said. We have a real job here to communicate the meaning of the numbers.   In conclusion, Hambleton gave a pitch for one of his favorite topics the full employment of psychometricians. And he also noted that computers and measurement technology hold great promise, but cost analysis and research must not be taken lightly.

Editor’s Note:  Professor Hambleton's research has focused in recent years on several topics of interest to ATP -- computer-based test designs, setting performance standards on credentialing exams, translating credentialing exams for uses in multiple languages and cultures, applications of item response theory, and test score reporting.  Professor Hambleton is an author of several  texts including Fundamentals of Item Response Theory  and Item Response Theory: Principles and Applications,  as well  as having served as a Past-President of the National Council on Measurement in Education, Division 5 of the American Psychological Association, the International Test Commission, and Division 2 of the International Association of Applied Psychology.

Association Notebook

ATP Welcomes as Members...BrainCore.net, LLC, Institute of Social Psychology, Linux Professional Institute,  and Symantec.

Next Board Meeting... October 2 & 3, at The Sagamore, Bolton Landing, New York.  Members can have business placed on the agenda by emailing the ATP Board of Directors c/o LScheibatATP@aol.com

Presentation Papers Available on the ATP Website...conference attendees can download presentation papers from the ATP conference held February 24-26, 2003 at www.testpublishers.org. Non-attendees can purchase a password to download the papers. For details, go to www.testpublishers.org.

Next ATP Conference... The 2004 ATP Conference will be held February 2-4, at the Renaissance Esmeralda Resort in Indian Wells, CA (near Palm Springs.)

ATP hosts successful conference for more than 500 attendees, representing 20 countries

Walter Lin Wang (second from right), COO of the American Testing Authority (ATA) and his staff traveled from Beijing, China to exhibit as a platinum sponsor and to attend the ATP conference in Amelia Island, FL

ATP Gratefully Acknowledges its sponsors for the 2003  Conference Technology in Testing: Application and Innovation

PLATINUM: ACT, Inc. • American Testing Authority, Inc. • The Chauncey Group International • Educational Testing Service ("ETS") • Galton Technologies, Inc. • Integral 7, Inc. • Pearson Professional Testing • Prometric, a part of the Thomson Corporation •Promissor, Inc. • Questionmark Corporation • TestOut Corporation 
 
GOLD:  Applied Measurement Professionals, Inc. ("AMP") • BrainCore.Net, LLC • Buros Center for Testing • Castle Worldwide  • Citigate Global Intelligence & Security • i-asses.com • Performance Assessment Network ("PAN") • Professional Credential Services, Inc. • Vantage Learning 

SILVER: The College Board • The Donath  Group • Examsoft • LaserGrade

ATP Also thanks Hewlett Packard for its donation of computers.