The Learning MarketSpace, April 1, 2001

****************************************************************************************************************
Written monthly by Bob Heterick and Carol Twigg, The Learning MarketSpace provides leading-edge assessment of and future-oriented thinking about issues and developments concerning the nexus of higher education and information technology.

****************************************************************************************************************
QUALITY.COM CONTINUED

In our last issue, we suggested that three popular dot-coms (Amazon, eBay and Zagat) can point the way to constructing Web-based systems aimed at meeting the quality assurance needs of higher education’s students--in particular, their need to differentiate the quality of one online course from another.

Evaluating the quality of a college-level course is clearly a more demanding task than reviewing a book or commenting on an eBay transaction. Rather than asking students whether or not they "liked" the course, we should ask students specific, pre-structured questions designed to take into account those factors that experts believe are necessary to ensure high quality. Responses to these questions would generate an overall "satisfaction index" similar to the star rating systems used on the dot-com sites.

A large amount of research has proved consistently that student ratings and expert ratings coincide. Furthermore, the literature on the validity of student self-ratings says that self-reports are at their best when the questions are a) very clear, b) concern behaviors (like "I spent x hours doing this") and attitudes (like "I was treated well and got my complaints resolved"), and c) avoid much inference on the part of the respondent (like "this course would be good for a person like X").

For purposes of illustration, let’s use the framework established in a recent study conducted by the Institute for Higher Education Policy (IHEP). The IHEP study, entitled "Quality on the Line: Benchmarks for Success in Internet-Based Distance Education", is particularly useful because it demonstrates that there is indeed a high degree of consensus in higher education regarding principles of good practice in distance learning. By asking students a series of questions based on that consensus, we would, in essence, "operationalize" agreed-upon principles of good practice.

The IHEP researchers first reviewed all of the existing principles, guidelines, and benchmarks that address best practices in distributed learning and combined them into a single list of 45 "benchmarks." They then tested the efficacy of that list by interviewing leading practitioners in the field, asking them three questions: how important are the benchmarks to the institution, to what extent do they incorporate them into existing practice, and are there additional benchmarks that should be considered? In that process, the researchers dropped 13 benchmarks, added three, and combined those that overlapped. The result is a list of 24 benchmarks that are "essential to ensure quality in Internet-based distance education."

By using the IHEP benchmarks as a framework and adding two additional questions, we can construct an example of the kinds of questions that could be posed to students to create a satisfaction index for each individual course. Like visitors to the dot-com sites, students would respond to each question using a 1-to-5 scale.

* Was the technology used in the course easy to use?
* How reliable was the technology?
* Was the course content relevant to your educational and professional goals?
* Was the course up-to-date?
* How challenging was the course? Were expectations for performance set high and within reason?
* Did you receive sufficient help when you needed it?
* Was there sufficient feedback to help you achieve your learning goals?
* Was there sufficient interaction with other students to meet your needs?
* Was there sufficient interaction with the instructor to meet your needs?
* Did course activities contribute to your learning goals (vs. being a "waste of time")?
* Was the information you received before enrolling in the course accurate and adequate?
* Did you have sufficient access to learning resources--e.g., libraries, databases?
* Were course expectations clear?
* Did the course experience match the expectations?
* Were assignments and learning activities clear?
* Were evaluations (interim and final) fair?
* Did you receive information about policies, procedures, and support services (registration, payment procedures, financial aid, etc.) that you needed?
* Were your questions answered accurately and in a timely fashion?
* Were complaints addressed adequately?
* Did you receive course materials in a timely fashion?
* Did you receive adequate technical assistance?
* Did you know how to access online resources?
* Was the course flexible enough to meet your needs?
* Was the course worth its cost?

In keeping with the goal of establishing a "satisfaction index," there are several possible ways to display the output. The first would be a star system like that used by Amazon and eBay where x number of points equal y number of stars, The second would be a weighted star system where factors would be weighted according to their relative importance to calculate a "score." A third approach, similar to the Zagat subcategories of food, decor, and service, would display scores for important subcategories--perhaps course structure, course delivery, and student support.

Rather than relying on experts to assess capacity--which would be unnecessary in this system, since students would be testifying to actual results--we also need expert input to focus on providing evidence of effectiveness. Input from two kinds of experts would be desirable: those external to the institution offering the course and those from inside the institution. External experts would include employers, who can supply data about student success on the job; graduate and professional schools, which can supply data about student success in future study; formal quality assurance organizations (regional accreditors, specialized accreditors, state agencies), which can collect data on learning outcomes; and consumer protection organizations, which can conduct independent studies regarding the quality of student experiences in online courses.

Data provided from the institution would include the following: completion rates, grade distributions, class size, data from follow-on courses (success rates in subsequent related courses), pass rates on standardized examinations, studies of course quality by external teams, support ratios, delivery times, longitudinal studies, and explanations for poor performance (e.g., we replaced our old server with a new one that works)

The just-in-time, embedded evaluation methodology used by the dot-com sites is a potentially powerful device for higher education because the primary reviewers are consumers. We in higher education need to find ways to stimulate these bottom-up models in order to bring in new perspectives. If we do not play a strong role in developing a consumer- based system, you can be sure of one thing: someone else--like U.S. News & World Report--will.

--CAT

****************************************************************************************************************
SOME CANT, SOME KANT

There is every appearance that the National Governor's Association is beginning to shift some of its attention from primary and secondary to higher education. Depending upon your level of cynicism, this is either good or bad. Whatever your viewpoint of gubernatorial attention on higher education, it is clearly a signal of some discomfort on their part with our results. Given that a number of leaders of business and industry have expressed similar interest in higher education, we might take this as an opportunity to consider their concerns.

Listed among these recent concerns are increased access to higher education, better measurement of student learning. more opportunities for lifelong learning; and better alignment of academic programs with economic development. Most of these seem to be issues in which governors ought to have an interest. To the extent that tax revenues partially fund the operations of public institutions, some of these are issues that governors cannot and should not avoid.

Immanuel Kant may be best known in educational circles for his quip that lecture is the process by which the faculties' notes become the students' notes without passing through the heads of either. Devotees of Kant know that he had a number of ideas about higher education that, while perhaps not as quotable, were every bit as astute. Two hundred years ago in The Conflict of the Faculties he observed,

"Whoever it was that first hit on the notion of a university and proposed that a public institution of this kind be established, it was not a bad idea to handle the entire content of learning (really, the thinkers devoted to it) by mass production, so to speak--by a division of labor, so that for every branch of the sciences there would be a public teacher or professor appointed as its trustee, and all of those together would form a kind of learned community called a university (or higher school). The university would have a certain autonomy (since only scholars can pass judgment on scholars as such) and accordingly it would be authorized to perform certain functions through its faculties."

The university envisioned by Kant has pretty much been the model for the past 200 years since he first wrote about it. The notion of a public institution is 100 years older, and the mass production model is hundreds of years older than that.

Kant might have found better measurement of student learning strangely out of place in the governors’ list. I suppose this is a carryover from the Standards of Learning that seem to have so obsessed governors and legislators in the primary and secondary arena. To be sure, better assessment of the quality of learning is a laudable goal and one to which most faculty would readily subscribe. By better understanding learning outcomes, we have the feedback opportunity to modify instructional techniques and engage in continuous quality improvement. However, as in the public school arena, it is hard to assess just what legislators and governors would do with data measuring student learning or lack thereof.

The other three concerns seem well suited to gubernatorial scrutiny. Increasing access to higher education has long been a favorite of politicians who have an uncanny knack for reflecting public sentiment. It was just 40 years ago that many states began, reorganized or beefed up their community college system. A number of states went on a construction binge to ensure that there would be a community college no more than an hour's drive from any citizen. Access was then, as it is now, a two-edged sword: reasonable cost and reasonable connectivity. Forty years ago reasonable connectivity meant driving time or some similar geographic measure--operating as we did under the mass production model of bringing the student to the faculty. The computer and communications revolution is changing that.

The cost issue looms even larger now than 40 years ago. It is more than a little ironic to find politicians decrying the high cost of a college or university education while they continue to whittle away at the level of state support for public higher education. They may just be smart enough, however, to have figured out that there is little they can do to change the cost parameters of our historic institutions of higher learning and that their efforts might be more rewarded by focusing on new learning strategies--hence their interest in distance learning. If the Western Governors University is a fair example, their fiscal commitment to new learning strategies is no stronger than their dwindling commitment to traditional higher education.

Creating more opportunities for lifelong learning is, in part, an access issue. It is also an issue of curriculum. If governors really want to increase access to distance learning opportunities, they might spend some time figuring out how to accelerate the deployment of broadband connectivity throughout their states. Just as a reasonable public transportation system (roads, bridges, etc.) were necessary to create learner connectivity to learning centers in the 20th century, a reasonable broadband communications system will be necessary for the 21st century.

Designing curricula to support lifelong learners is something with which most of the professional colleges--engineering, medicine, and business administration, for instance--have had some success. Beyond the limits of the professional schools, most colleges and universities have shown little interest in creating continuing learning opportunities with the faculty perceiving them as "training" rather than "education".

This last issue is tightly related to better alignment of academic programs with economic development. Economic development issues have historically failed to track the higher education curriculum. This failure was, in large measure, responsible for the land grant movement in the middle of the 19th century. With the passing of 150 years, some of higher education's critics and friends may be signaling the need for a similar shift in it's role.

New strategies of asynchronous learning, whether on campus or at a distance, may be just what is needed to address the governors’ concerns--not as a replacement for our historic institutions, but as an additional weapon in the higher education arsenal.

--RCH

****************************************************************************************************************
SUBSCRIPTIONS, ARCHIVES, RE-POSTING

To subscribe to The Learning MarketSpace, click here.

Archives of The Learning MarketSpace, written by Bob Heterick and Carol Twigg and published from July 1999 – February 2003, are available here.

You are welcome to re-post The Learning MarketSpace on your intranet without charge. Material contained in The Learning MarketSpace may be reprinted with attribution for non-commercial purposes.

Copyright 2001 by Bob Heterick and Carol Twigg.