July 1998 // Letters to the Editor
Outcomes Measurement is Not Enough
by Stephen C. Ehrmann
Note: This article was originally published in The Technology Source (http://ts.mivu.org/) as: Stephen C. Ehrmann "Outcomes Measurement is Not Enough" The Technology Source, July 1998. Available online at http://ts.mivu.org/default.asp?show=article&id=1034. The article is reprinted here with permission of the publisher.

I've just read Ed Neal's piece on the evaluation of technology use. I'm grateful for the mention of Flashlight, but I think Ed has misinterpreted its purpose. Flashlight is actually a cure for the problem that Ed identifies in criticizing the study by Gerald Schutte. (I have to add, by the way, that I like Schutte's study better than Ed does, but that's another matter.)

Most faculty face a real problem in studying whether the use of technology has improved quality of learning. Even when faculty can collect outcomes data from an experimental group and a comparison group (without the same technology), they still can't prove why outcomes change (or fail to change). Suppose a faculty member hopes that the use of e-mail will lead to more collaborative learning and thence to better grades on an exam. Suppose that this faculty member observes that the grades have indeed improved after e-mail was made available. She still doesn't know whether the e-mail played a role. It would help to know if students collaborated more in the experimental group, used e-mail to do so, and thought the e-mail was helpful for the purpose. Without such backup, the faculty member is left to wonder if the grades changed for other reasons.

Flashlight, as a method and as an item bank, is designed to help faculty answer such questions about what students are actually doing with the technology. In the example above, use of Flashlight might have revealed that students were indeed collaborating more. That wouldn't conclusively prove that technology helped improve achievement, but it would be a more persuasive argument than that provided by outcomes data alone. Gary Brown's studies at Washington State University provide another example of Flashlight being used to interpret outcomes changes. Gary's study made a persuasive case that the ways at-risk students used technology in a set of experimental seminars was probably responsible for the improvement in their GPAs.

Of course it is not always possible to make such a direct comparison of outcomes. One common barrier is that the use of technology often makes it possible for faculty to change the goals of a course, and thus they also change the tests. When learning outcomes can't be compared directly, one of the few ways to say anything about quality improvement is to give evidence that technology students do things (like collaborate, or interact with faculty, or spend time more efficiently on studying) that research shows usually produce better learning. The Flashlight Current Student Inventory can be used for that purpose, too.

word gamesdownloadable gameshidden object gamesbrain teaser gamesplatform gamesmatch 3 gamessimulation games
View Related Articles >