©2004 Association of Test Publishers
Save or read as pdf file

Even Ronald McDonald Takes Tests McDonalds’ COO tells conference attendees testing makes good business sense

McDonald’s USA oversees 30,000 restaurants, employs one and a half million people worldwide and has served up burgers and fries to over 40 billion individuals -- but as McDonald’s
COO Ralph Alvarez told attendees of the fifth annual ATP conference - when it comes to tests they are just one more satisfied customer.

“Keep doing what you’re doing (developing tests) and we’ll continue to be a satisfied end user-because we are seeing the results,” Alvarez told attendees.

Alvarez detailed how McDonalds’ top management grew concerned several years ago that their leadership in the burger market was slipping.  Their closest competitors, Burger King and Wendy’s, appeared to making inroads in a market they had successfully dominated for decades. So, according to Alvarez,  they took a closer look at what had made them competitive in the past and what was needed to “get them back on track.”

The results of their survey showed that the key to their past success had been their ability to successfully replicate good business practices, as well as have their franchises consistent provide a quality product.  And although that formula still made for a sound business plan, they found that new hurdles were being thrust in their way.

“We found we were employing many more individuals for whom English was a second language. We were also making an effort to employ the physically challenged,” said Alvarez. That, added to the fact that over half of the 600,000 employees hired by McDonalds every year are working their very first job, was creating new challenges for the decades old franchise, he said.   And those factors, coupled with the organization’s burgeoning growth was putting a heavy strain on out-moded methods for hiring, training and supervision.

“It cost us $400 on average to train a crew member, and $8,000 to train a manager, so when we lost them, it was a hefty price to pay,” Alvarez said.  It was this high cost of turnover that lead the company to look at the costs of deploying predictive tests and to review their training methods.

“Eighty percent of our managers move up from the crew level and yet we had no verification procedure that a person who was a good crew member would be a good manager...it’s a whole new set of skills,” Alvarez noted.  By using tests, he reported,  “employees were more successful and happier.”

And happier employees lead to happier management as McDonald’s USA watched turnover decrease by 25 percent. 

“From an industry point of view - I looked at testing as something that happened in the school system.  It wasn’t an integral part of how to run an organization... but we found that the verification of knowledge is critical  for a successful organization,” Alvarez said.

Spurred on my their success, McDonalds USA turned to a review of its training programs. They had prided themselves on being among the first to utilize video training - but now they found that innovations in the assessment industry could greatly enhance their previous methods.

“The video learning was a great competitive advantage, but there was no follow up,” Alvarez said, laying out the problem.  The company turned to a program of ebased learning with simulations.
With the enhanced training program, Alvarez reported, employees get to practice taking orders and greeting customers long before they need to do the real thing. “We removed a level of fear (for the new employee) and the result was better retention and more satisfied customers,” Alvarez said.

The other plus, he added, was food safety. “We have to calibrate the food and make sure it’s safe.  Every six months our field staff must be verified that they are up to speed on all times, temperatures and procedures involving safe handling of food.  There’s now a system and a verification process in place to assure us that this is being done consistently and well.”

McDonalds USA took its new success at the employee level to the supervisory level.  He likened the McDonald’s franchise to “30,000 little factories that need to be run as smoothly as possible,” and the supervision involved was costing more money every year. Through the use of etraining, testing and verification they were able to bring the number of their supervisory regional offices down to 21 from a high of 38 – and at the same time significantly improve performance.

Overall Alvarez reported that McDonalds USA was pleased with their progress and would continue to look to test publishers and outside vendors to develop and deploy their tests. “Test development is not our core competency,” Alvarez stated.  But testing, he said, has allowed them to return their concentration to what they do best  -- “selling burgers.”

Megatrends: Technology in Testing
CITO CEO tells conference attendees technology in testing subject to supply and demand

Test developers can dream about future innovations all they want – but if test takers and users aren’t convinced of the benefits, the

 old fashioned rules of supply and demand will prevail to stymie progress. That was part of the message which Marten Roorda, CEO of the Netherlands-based CITO brought to the ATP conference in his keynote address, Mega-trends:Technology in Testing.
    
“It is a tough lesson from economic history that innovation, economic renewal or growth work best based on a technology pull, when the market itself demands something,” he told conference attendees.
    
Roorda listed what he saw as some of the primary achievements to-date in computer based testing including automated item construction and adaptive tests to the “enormous increase in the number of possibilities for automatic processing, analysis and reporting of computer tests.”

But he then went on to point out that most of the achievements relate to production management and processing and fall within the domain of providers. 

“If I were to approach the matter negatively, I could say that the suppliers and administrators of tests have primarily considered their own interests and have forgotten about the users and candidates,” Roorda noted.

He challenged test developers to view test-technology innovation from the end-user’s point of view. Employability, he noted, is still limited. “It may be true that by now we can let the computer correct essays...but...the correction does not reveal what the candidate’s real insight is,” Roorda noted.

He added that the computer is still viewed by many as an impersonal instrument and the Internet is still fraught with concerns from security to system failure.

And until many of those concerns and others are addressed, demand will not be enough to spur innovation. “In any change in test technology the user’s advantage should come first,” Roorda said.

Roorda went on to list eight mega trends in test technology, which, he said, were based on the advantage of the innovation to the candidate. These included the integration of testing and learning;
the development of accurate competence assessments; the development of

monitoring systems that will be able to track the development of students or employees; the development of authentic and realistic tests; stealth testing, (which Roorda explained was the ability to test without the subject noticing -- though he cautioned that this type of testing must not be invasive of people’s personal integrity);  the evolution of the computer as a companion, (with keyboards retreating into the background and the emergence of spoken contact between human and computer); the development of just in time tests – making tests as close as possible to the moment of use and as near as possible to the location of use; And finally, globalization.

“In the future, testing and assessing will be part of the international domain much more and...subjected much more to world-wide standards,” concluded Roorda.

ATP FOCUS ON...David J. Weiss, Ph.D.
Recipient of the 2004 ATP Career Achievement Award

This year’s recipient of the Association of Test Publisher’s 2004  Career Achievement award, David J. Weiss, Ph.D., Professor of Psychology at the University of Minnesota, claimed his interest in metrics came from humble beginnings.

“When I was 17  I worked in a deli weighing the fish, which was to be served over the Jewish holidays,” Weiss said.  He learned quickly to meet the demands of Jewish housewives who had very specific requirements for their carp in terms of weight and size. “ Estimating the weight of a live, squirming fish is very tricky...so I learned to modify priors and interpolate data.”  And this early experience, he claimed, lead to a career in psychometrics.

Whether Weiss’ amusing background really was the catalyst for his interest in assessment, there is no question that his 30 year career in psychometrics and computer adaptive testing is a testament to his ability in the field of measurement. 

Dr. Weiss began his higher education at the the University of Minnesota – and never left.   Again to the amusement of conference attendees listening to Weiss’ acceptance speech,  Weiss claimed his decision to pursue graduate work at the University of Minnesota was purely due to the fact that the Graduate Record Exam (GRE) was not  required and Weiss did not want to “waste another Saturday afternoon”  taking a standardized test. 

“So I started in 1959 as a graduate student... and stayed,” Weiss said.

Eventually Weiss worked his way into a full time position in the Psychology Department where he first started researching the concept of using computers for test administration.  In 1968 he gave a talk in which he outlined some of his ideas related to the use of computers to deliver tests, “not just to score or interpret tests,” he said.

After his talk concluded, Weiss recalled that “ an older fellow came up afterwards and said to me, ‘ I don’t  know where you got the idea - but the computer will never replace the paper and pencil test’.”   That individual was Wayne Coffey, head of the Iowa Testing Program.  And those words, according to Weiss, were “fightn’ words.”

Weiss applied to the University for a grant and in 1970 delivered the first computerized adaptive test. 

According to Weiss’ website located at www.psych.umn.edu/psylabs/CATCentral, Computerized adaptive testing (CAT) is defined as the redesign of psychological and educational measuring instruments for delivery by interactive computers. 

Today, thanks in large part to the work of early pioneers such as Weiss, CAT can be used for tests of ability or achievement and for measures of personality and attitudinal variables. 

Again, according to Weiss’ website, the objective of CAT is to select, for each examinee, the set of test questions from a pre-calibrated item bank that simultaneously most effectively and efficiently measures that person on the trait.

In 1971, Weiss was visited by a representative of the Personnel and Training Research Programs of the Office of Naval Research (ONR) who was interested in some of the research he had done prior to 1970.  During the conversation he asked what Weiss was planning to do for his future research and Weiss told him about the rudimentary CAT system he had operational. The representatives were delighted to hear about it, since they apparently were looking for someone interested in CAT  to operationalize theoretical work on IRT and CAT  that they were already funding (by Frederic Lord of the Educational Testing Service) . Weiss then submitted the first of many applied research proposals on CAT to ONR and his research program began in 1972. This program continued through 1985, with support from other units of the U.S. Department of Defense including the Army Research Institute, the Air Force, the Marine Corps, and the Defense Advanced Research Projects Agency.
     
Weiss has also been closely involved in the design and implementation of seven or eight CAT software systems, beginning with his 1970 system, several versions of CAT research software for an ONR-supported research, three versions of the MicroCAT Testing System, and most recently the FastTEST Professional Testing System.

In 1968 Weiss launched his own journal which was later purchased by Sage Publishing, and he was integral in the hosting of  International CAT conferences in 1977, 1979, and 1982.

Despite his pioneering work in computer based testing, Weiss issued some cautions when it came to Internet testing.
 
“ I would be the last one to say that the Internet is not worthy of investigation - but I believe we are jumping in a lot more quickly than is  warranted.  It should not be used until and unless we have data that the tests we are delivering on the internet are the same as we are delivering on the PC,”  Weiss warned. His fear is that test critics will leap on any perceived weakness in a testing concept and the negativity will have a trickle down effect “until the entire testing community is in a state of misery,” Weiss said.

“ In the early days we went through great pains to compare paper and pencil tests with the CBT’s -- we were able to show that the scores were not significantly different.  I have not seen this body of literature being produced for Internet delivery,” Weiss cautioned.

He said the solution is the same solution that was used in converting paper and pencil tests to computer based tests – Standardization.  Just like the standardized weights used in the fish deli -- “Any test is only as good as its standardization,” Weiss concluded.

Association Notebook

ATP Welcomes as Members....Performance Testing Council (PTC) and the National Council of State Boards of Nursing.

Next Board Meeting... Oct. 13-14, at The Coeur d’Alene, Coeur d’ Alene, Idaho..  Members can have   business placed on the agenda by emailing the ATP Board of Directors c/o LScheibatATP@aol.com

Next ATP Conference... The 2004 ATP Conference will be held February 27 - March  2, 2005, at the Westin Kierland Resort in Scottsdale, AZ.
 

---

2004 Conference Pictures
Conference attendees, such as this group from CITO in the Netherlands, represented more than 14 countries	Conference Chair Bill Cramer (right) relaxes at a pre-conference Superbowl Sunday party hosted by ATP.  He is joined (l. to r.)  by Julia Leahy of Capstar, John Krucenski and Dave Meisner of Prometric and Linda Waters of Capstar.	ATP Executive Director, Dr. William G. Harris (center) enjoys a reception with fellow attendees Cliff Donath (left) and David Morris (right).
Sandra Winborne, Chair of the Department. of Veteran Affairs Professional Certification and Licensure Advisory Committee,  gave a stirring rendition of the Star Spangled Banner at the opening session.	Marilyn Monroe, pictured here with Bob Whelan, General Manager of Pearson/VUE,  made a guest appearance at a reception hosted by Pearson VUE.	Ever walk into a room and discover your wearing the same outfit as someone else? This conference attendee, Tracey Sheehan,  found she was dressed the same as the elevator!!
Click on picture for larger image

ATP  acknowledges and thanks

Hewlett Packard
for the donation of computers for the ATP conference and

Scantron Corporation
for its work in post conference evaluation

ATP Gratefully acknowledges and thanks its
2004 Conference Sponsors
(To  visit the websites of any of these sponsors go to
www.testpublishers.org and click on “Meet ATP’s Sponsors”)
PLATINUM
ACT, Inc. • AMP-Applied Measurement Professionals • ATA, Inc. the Provider of DST (Dynamic Stimulation Technology)• Caveon  •  Capstar • Educational Testing Service (ETS) • Pearson VUE  •  Prometric, a part of The Thomson Corporation • Promissor, Inc.
GOLD
I-assess •  Buros Center for Testing  • PAN - Performance Assessment Network • Thinking Pattern Profile • CITO •Vantage Learning  • Ordinate
SILVER
American National Standards Institute • The College Board • GMAC • The Donath Group • The Performance Testing Council (PTC)• QualityMetric, Inc. • Integral 7 
 

The Test Publisher newsletter is posted with the understanding that the content of the newsletter does not constitute the rendering of legal, accounting or other professional opinions.  If legal advice or other expert assistance is required, the services of a competent professional should be sought.
 

©1997 - 2005 Association of Test Publishers