Improving the Quality of Student Learning

Virginia Tech

Based on available data about learning outcomes from the course pilot, what were the impacts of re-design on learning and student development?

The redesigned course ran during the fall 1999 and spring 2000 semesters. The enrollments at the end of the course were 1406 and 639, respectively, and the final grade averages were 2.22 on a scale of 4 in both cases. Enrollment in the fall semester of 2000 was about 1360; outcome data are not yet available.

These classes were taught in one section each in the redesigned format. The last year for the traditional course was 1996 - 1997; a preliminary form of the redesigned format was introduced to all sections in 1997 - 1998, the year the Math Emporium was opened. The first major revision of the on-line presentations was 80% complete for fall 2000, and the remainder will be in place for spring, 2001. The goals of these revisions have been (i) to improve clarity in areas where test results show students have difficulty, and (ii) to make the presentations substantially more interactive.

The main assessment data come from the final exam and course grades. In addition, opinion surveys, student focus groups and interviews with staff are used. (Note: Honors sections of Linear Algebra are not part of the course redesign. They accounted for approximately 5.6% of the students in each semester. Unless otherwise stated, statistics here and below exclude the honors sections.)

All students take a common final exam in multiple-choice format. Questions are coded according to a list of course goals and difficulty levels, and a two-person team reviews the results each semester. The team decides whether each goal covered on the exam has been "well met" (W), "met" (M), or "not met" (N), and the report is used by the course designers as a guide for changes in the syllabus or materials. For fall 1999, the questions yielded 15 W, 6 M and 4 N outcomes. For spring 2000, the results were 13 W, 6 M, and 6 N. In both cases, the reviewers concluded that the overall scores (67% and 69% correct for Fall and Spring, respectively) were reasonable for these tests.

These and earlier final exams have consistently found lower levels of success in the topic of eigenvalues and eigenvectors than in the course as a whole. This is one of the more complex topics of the course, and it is covered near the end of the term. This area is one focus of attention for the revised presentations. At this point, there is a consensus that the system works best with simpler, skills-oriented topics than with more conceptual ones.

Course enrollment and grade statistics have been steady from year to year and consistent with those of the last traditional year. The table below shows data for fall semesters:

(These numbers include the honors sections.) The general conclusion is that the revised course has yielded roughly stable performance, with some increase in retention.

There is a "blip" in grades for fall 1998. Various faculty members who have been involved with the course feel that they were perhaps more accommodating to the students during the first two years of redesign. For example, early software problems led to numerous disputes over quiz grades, and these were generally resolved in favor of the student.

For the spring 2000 semester, the exam reviewers looked closely at test results for students who were repeating the course. They did considerably worse than the group at large, accounting for 11% of the spring enrollment and 40% of the exam scores below 50. This indicates, perhaps, that the various learning paths offered in the revised course still do not reach everyone.

We have not identified particular advantages or disadvantages for minorities or women in the new format. The class, made up primarily of engineering students, is fairly homogeneous socially.

Assessment also includes surveys of student opinion. Web-based surveys were given in Linear Algebra at the beginning and end of the fall semesters, 1999 and 2000. The surveys are long (e.g., there are 80 questions on the late fall survey), and participation rates are over 90%. A number of interesting points have emerged so far, and the latest results allow us to begin looking at year-to-year progress. [In these examples, wording such as "agree" is shorthand for survey categories Agree or Strongly Agree, Frequently or Always, etc. Percentages here and below are relative to all respondents to the survey, including those who did not respond to the particular question.]

• Only 14.8% of students in 1999 and 11.8% in 2000 indicated they had studied with other students for math quizzes and tests during high school, while just 29 to 30% said they had used active learning strategies.
• Very few (3-5%) had used computers for math in high school, but 45 to 55% had used hand-held calculators. (Oddly, both these numbers decreased from early fall, 1999 to early fall, 2000.)
• Since useful study time is a good indicator of success for any course, the course designers are seeking ways in which technology can be used to increase it. In 1999, 65.2% of students surveyed near the end of the course said they spent one to two hours per week on the course, and 32% reported 3-5 hours per week. This seems to be a very small amount of study time, but it is apparently consistent with what was known about the traditional course. It is hoped that the increased interactivity in the new version of presentation software, as well as improved software for quizzes and testing, will help. The numbers for fall are consistent with this hope: 53.8% indicated 1-2 hours and 43.4% reported 3-5 hours per week.
• 70.7% in 1999 and 76.4% in 2000 said that the presentation software explained concepts well. On the other hand just under half the students said that "this course helps me to understand the relevance of the material to future studies," and 56% agreed (vs. 37% who disagreed) that they would have learned more if the course had been conducted in a regular classroom. (Lectures and a textbook are still available as options but are used infrequently.)
• 66.6% (1999) and 68.8% (2000) thought they did their best learning while working alone.
• Some questions probe the general attitude of students towards doing things in this new way. Just over half (51.3% in 1999 and 57.2% in 2000) would recommend this course to others. Fewer than 40% found the assignments "stimulating," while 36.4% in 1999 and 43.5% in 2000 felt that "instructors and tutors for this course are interested in what I have to say."
• Emporium helpers consistently score in the 70-85% favorable range on questions concerning their knowledge, politeness, responsiveness, and effectiveness in explaining concepts.
• As an indicator that technical problems are causing less trouble, the number who felt that "I spend too much time trying to log on" decreased from 32.2% to 14.4% from 1999 to 2000.

Back

Program in Course Redesign Quick Links:

Program In Course Redesign Main Page...

Lessons Learned:
Round 1...
Round II...
Round III...

Savings:
Round I...
Round II...
Round III...

Project Descriptions:
Sorted by Discipline...
Sorted by Model...
Sorted by Success...
Sorted by Grant Rounds...