William Trochim, left, professor of human service studies, discusses Web site evaluations with his undergraduate research assistants, Dominic Cirillo and Dawn Bonanno. Charles Harrington/University Photography
Surfing the Web or spinning your own Web site? Millions are, but you're probably not evaluating your site to see what effects it has -- or are you using your Web site to evaluate your Web site. However, a Cornell evaluation expert not only is using his elaborate Web site as an integral part of his teaching and research but also as a "field laboratory" to evaluate how well the site is accomplishing its goals.
In addition, William Trochim, professor of human service studies in the College of Human Ecology, is using his Web site to collect data on a wide variety of evaluation research projects, from whether students do better when their course resources are online to what traits the Ithaca community wants in its next chief of police.
In his first online semester, Trochim developed a Web-based resource center complete with a hyperlinked text, downloadable lectures, sample exams, simulation workbooks, a sample term paper and resource lists, class bulletin boards, an online extra-credit scavenger hunt game and more. Working with a team of graduate and undergraduate research assistants, Trochim found that students liked the flexibility and worldwide access to resources the online course offered them, but they did not do any better on class exams than his former students.
"This finding is important because government agencies, businesses and nonprofit organizations are making tremendous investments to adopt the new Web technologies but have no idea whether their investments are having an impact. In their haste to use this pervasive and revolutionary technology 'on the fly,' they have no idea what its effectiveness actually is or what unexpected consequences there might be," said Trochim, an evaluator and social science research methodologist. He is using the Web as a tool in conceptualizing and evaluating all kinds of projects and problems, including instructional technology delivery, community issues and social science research questions.
"The bottom line is that we need to evaluate and then evaluate again. Just because clients or students have more access to information doesn't necessarily translate into performance improvements. Making such assumptions might prove very dangerous -- and expensive," Trochim said.
At the moment, there are no established rules or theories to guide Web construction and use, Trochim pointed out. "We are weaving the Web out of thin air -- making it up as we go along. There has been a remarkable absence of research into how Web sites are developed and implemented and their effects and impacts. My work helps fill that gap."
Trochim is taking the tools of his trade -- the latest methodologies and technologies in evaluation and social science research methods developed over the past half-century -- and applying them to all phases of Web site development and assessment.
In studying his own Web site, for example, Trochim and Agriculture and Life Sciences junior Dawn Bonanno of Lake Huntington, N.Y., are analyzing the content of every single electronic communication between Trochim and his students over the course of a semester -- more than 500 separate messages. The analysis, which Bonanno is incorporating into an honor's thesis, shows how students use the Web in their coursework and what problems they struggle with in accessing and making use of this new technology.
"This experience, which has been wonderful, has been giving me a first-hand look at evaluation techniques that I hopefully can apply to my training and future profession as a counseling psychologist," Bonanno said. "It's also exposing me to the 'nuts and bolts' of social science research and, on a more informal note, to the great opportunities that exist even at a large university like Cornell to work closely with a professor."
With Human Ecology junior, Dominic Cirillo of Butler, Pa., Trochim is examining the different ways to analyze every "hit" to a Web page and working out the logistics of linking that data to course performance information.
Both students wrote papers describing their research that were presented at the American Evaluation Association Conference in Atlanta in November. (The full papers, of course, are posted on Trochim's Web site, listed below).
Much of Trochim's evaluation work uses a technique he has developed over the past 14 years called concept mapping. This method, which can now be implemented over the Web, uses state-of-the-art statistical analyses on input from many people -- such as the staff of an agency, corporation managers, boards of directors or members of a community -- to "map" their ideas, prioritize what's most important to whom and explore the degree of consensus across subgroups. Concept mapping involves brainstorming ideas relevant to the problem or issue, grouping and ranking the ideas and feeding them into a specially designed multivariate and cluster analysis program. The end result is a picture or map that visually conveys what the groups involved think is most important and the degree of consensus across different groups.
Concept mapping has been used in a wide variety of applications, including strategic planning, community-based decision-making, training and technology evaluation, program and product development, and even in hiring decisions for important positions.
For instance, Trochim has used concept mapping to evaluate staff goals for an employment program for the severely mentally ill; to help assess a multicultural awareness youth camp; to construct a volunteer training program for working with deinstitutionalized patients; to develop standards for accrediting graduate-level evaluation programs; to evaluate the effectiveness of corporate training programs; to determine what's most important to the Ithaca community in choosing a police chief; to explore the factors that enhance a congregation's spirituality; and to determine what's of concern to Cornell students. All of these projects, most of which were developed in the Ithaca area, can be reviewed on his Web site, and several of them were conducted over the Web.
Trochim's Web site is a valuable resource not only for his students and "eavesdroppers" but for anyone seeking help with social science research and methods. It includes open access to his lectures, textbook and course materials on evaluation methods as well as research papers and help in choosing which statistical analysis would be appropriate for a set of data. It also has links to many resources for help in creating a Web page, including clip art and graphic packages, and there is even an "O.J. Page" that argues that the legal system is a good metaphor for how applied social researchers collect and weigh evidence.
Trochim's Cornell Web site can be viewed at http://trochim.human.cornell.edu. It was developed entirely with $5,000 seed money from the College of Human Ecology. His concept mapping project to help the Ithaca community select its next chief of police is funded by a Hatch grant from the U.S. Department of Agriculture.