HOW TO ORGANIZE A CAMPUS-WIDE COURSE REDESIGN PROGRAM USING NCAT'S METHODOLOGY

VIII. Selecting Proposals That Will Succeed

A vital element in the success of a course redesign program is to require very specific plans as part of the proposal process. Such an approach ensures that planning will be accomplished and that the redesign teams are clear about what they are going to do before grant awards are made. Teams can then focus on implementing plans that are roadmaps to success.

Program leaders should review the proposed plans carefully. NCAT recommends using the following review process. Each proposal should be read independently by program leaders and rated 1 (strong proposal: no outstanding issues or only minor issues), 2 (potentially acceptable proposal pending resolution of outstanding issues), or 3 (weak proposal: does not meet program guidelines.) Program leaders should then collaboratively create a list of the strengths and weaknesses of each proposal and follow up with each team to clarify any outstanding issues or to help project teams strengthen weak points in their proposals. For proposals rated 1, you need to clarify and resolve any minor issues. For those rated 2, you need to ask for more information, a more complete narrative, revisions of supporting forms, and so on. For those rated 3, you need to focus on the issue that does not meet the program guidelines (e.g., if the proposal lacks a cost reduction strategy, you need to determine whether the team is, in fact, interested in reducing costs) in addition to asking for more information, more narrative, revisions of supporting forms, and so on.

Characteristics of a Proposal Rated 1 (Should Be Accepted)

Appropriate Course Choice. The rationale for the redesign is clear. The proposal includes a clear statement of the problem the redesign seeks to solve such as high DFW rates, space issues, consumes too much faculty time or resources, course drift, or increased demand for the course.The course choice meets the selection criteria in the Application Guidelines, and the proposal includes data that support the choice.

Examples

  • An average of 48% of students fail the developmental English course. Assignments and assessments are not consistent among instructors. There is a lack of student engagement in the forms of high absenteeism, frequent failure to complete assignments, low quiz scores, and little involvement in class discussion. The course pedagogy is outmoded: the traditional model of teaching consists entirely of group instruction, yet writing is a highly individualized activity. In addition, the traditional approach to paper grading, though labor-intensive, has not been shown by research to have any positive effect on student writing, and it limits the number of students assigned to each section.
  • Incoming students have extremely different backgrounds in chemistry. Typically, at least 10% of the students never had chemistry before, whereas 20% were enrolled in Advanced Placement high school courses or even college-level introductory chemistry. Students often lack successful learning strategies and resist adjusting their study skills as they transition from high school to college. Student success relies too much on rote memorization rather than developing conceptual thinking and problem-solving skills. Student engagement in recitation classes is inconsistent and often inefficient. Despite weekly meetings to adjust efforts and timelines, considerable duplications of effort ensue when instructors individually compile lecture notes, PowerPoint slides, and clicker questions. The chemistry department lost several faculty positions due to budget cuts and hiring-freeze policies. As a consequence, 200- and 300-level courses are currently taught combined as one course. This practice sacrifices the quality of upper-level education and prevents students from taking 300-level courses as electives if they were previously enrolled in the 200-level course.

Appropriate Model Choice. The redesign model choice is an appropriate one for the discipline, and the proposal includes a rationale for the choice.

Examples

  • In choosing the Replacement Model, the university will follow the lead of all prior successful NCAT redesigns in Spanish. The redesigned five-credit course will more than double the number of regular sections and limit enrollment to 20 students per section. Sections will meet physically three times a week, with class time devoted to communicative exercises emphasizing oral skill development. Workbook, grammar, and writing components will be moved online. Students will spend two hours of online practice in grammar—with automated immediate diagnostic feedback—and will write weekly compositions that will be graded semi-automatically with diagnostic feedback. Students will also participate in one hour of language lab weekly.
  • The psychology redesign, using the Supplemental Model, is based on the NCAT’s Essential Elements of Course Redesign. An active, learner-centered approach will incorporate technology to facilitate a more individualized course experience while simultaneously reducing costs. A student response system will be incorporated into the classroom. Required web activities and practice quizzes will complement course lectures. An early intervention system will target students who are struggling as indicated by attendance, in-class responses, web activities and online practice quizzes. Finally, a team-teaching model will be implemented.

Clear Comparison of Traditional and Redesigned Course Structure. From reading the proposal the and subsequent follow-up, reviewers should be able to state clearly and succinctly how the course operates in its traditional format and how it will operate in its redesigned format with enrollment numbers. This is not as easy as it sounds!

Examples

  • The traditional course comprises 18 lecture-based sections of 153 students. Each faculty member chooses the content and method of delivery of course material. In the redesigned course, nine sections of about 300 will be offered. Lecture time will be reduced by 50%, replaced by online activities. Large groups will be divided into 15 smaller, online groups of 20 led by undergraduate learning assistants. During face-to-face classes, groups of 15 will be engaged in active discussion led by undergraduate learning assistants. Students will also be required to engage in low-stakes quizzing with immediate feedback.
  • The college’s redesign plan collapses the 27 sections of European and US History historically needed to serve general education students into a single European history section and a single US History section, each serving 300 students per semester. In-class time will be reduced from three hours to one hour per week. Students will spend a minimum of two hours per week in a computerized history learning center dedicated to the two courses. At least one hour will be spent on completing online publisher-provided quizzes, map exercises, and chronology work sheets. One hour each week will be spent in online discussion groups of 20 students each, moderated by virtual preceptors. Students will be able to test their own skills in historical argument and interpretation.

Sound Pedagogical Strategy. The redesign plan is concrete and has a good chance of improving student learning. The plan specifically describes ways the team will foster greater student engagement such as through online quizzes, immediate feedback in software, engaging use of clickers in class, a plan for monitoring student progress, and intervening when necessary. The plan includes assignments that are described clearly and that require critical thinking. The learning materials to be used are well identified.

Examples

  • The redesign of the introductory women’s studies course will use the Replacement Model. The quality of teaching and learning will improve significantly due to a variety of factors, including adaptation and implementation of materials and ideas already used successfully by prior NCAT projects in the humanities and social sciences. Part of the lecture time will be replaced with required online student activities and discussion. Students will work in small groups, participating in discussions around course topics. They will complete individual and group activities such as virtual field trips or examination of real data on women’s issues. To increase student feedback, Blackboard will be used to deliver a series of required, low-stakes quizzes, and personal response systems in the remaining lectures will be used to provide more in-class feedback.
  • The planned redesign will enhance the quality of the college algebra experience by motivating students to take an active role in learning and to spend time working on rather than watching mathematics. Faculty members are constantly frustrated that students in the traditional course are so passive in the classroom and want a cookbook approach to mathematics. Furthermore, with such a large population each semester, hand grading of homework is just not feasible. Therefore, very few students do the work required to master the skills and concepts the course teaches. Online assessment software will provide a tool for continual assessment and immediate feedback. The Emporium Model will also enable students with varied backgrounds to receive individualized assistance at their own pace in a learning center staffed with instructors and tutors. All students will be required to attend the learning center at least two hours per week; thus, many students are likely to spend more time doing mathematics than they are spending in the current model.

Valid Assessment Strategy. From reading the proposal and reviewing the assessment forms and subsequent follow-up, reviewers should be able to state clearly and succinctly how the team intends to compare student learning outcomes of the traditional version with the redesigned version of the course.

Examples

  • The plan has a fully described, valid assessment strategy that compares student performance on a common final examination.
  • The plan has a fully described, valid assessment strategy that compares two problems for four separate topics on the final exam (representing 80% of the exam) by using a common scoring rubric.

Valid Cost Reduction Strategy. From reading the proposal and reviewing the CPT and subsequent follow-up, reviewers should be able to state clearly and succinctly how the team will reduce instructional costs.

Examples

  • The plan has a cost reduction strategy, which is that the full-time faculty will serve more students (N=38), and one adjunct faculty member will no longer teach the course. The cost-per-student will be reduced from $213 to $174, an 18% reduction.
  • The plan has a cost reduction strategy, which is to serve more students (N=190) on the same resource base by increasing section size from 77 to 150. The cost-per-student will be reduced from $285 to $218, a 24% reduction.

Cost Savings Plan. The plan describes what will happen to the savings.

Examples

  • Cost savings will remain in the psychology department and be used both to support the redesigned course in the future and to support faculty wishing to undergo additional course redesign projects.
  • Cost savings will enable faculty to teach other communications courses and reduce the need for adjuncts in times of tight budgets.

Other characteristics of a plan rated 1 include consistent numbers (the numbers correspond throughout the narrative, the assessment forms, and the CPT), a doable plan (the redesign plan can be implemented within the project time frame), and a clear and reasonable project budget.

Characteristics of a Proposal Rated 2 (Needs Further Clarification before Accepting)

Proposals that are rated 2 are those that can be improved by asking the project team to provide more specificity or clarification. The proposal has a lot to recommend it, but it is insufficiently developed or documented. In such cases, the program leaders should discuss the issues with the project teams. If the outstanding issues can be clarified so as to meet the program requirements, the proposal should be accepted. If queries from the program leaders reveal genuine weaknesses that turn the rating into a 3, the proposal should be rejected.

Following are examples of problems we have often found in course redesign plan proposals.

Too General

  • The plan for the redesigned course structure is not well-thought-out. Red flag terms such as “accelerated” and “self-paced” are used. Lab time is not required; students may decide when to go to lab.
  • The description of what is planned to happen needs greater clarity and more specificity. For example, can 120 students meet in the computer lab at the same time? Is the lab big enough? Is this a good idea?
  • It is not clear how the team plans to use technology; the comments are very general such as, “We plan to use Blackboard.”
  • It is not clear that the redesign plan will lead to greater course consistency, because individual faculty members can still use their own materials if they wish rather than using commonly developed assignments and tests.
  • The learning materials seem to be designed primarily by the faculty when commercial software and other materials are readily available.
  • The prospective learning materials have been narrowed down, but no specific materials have been selected.
  • The plan includes the expectation of the monitoring of student progress, but no clear intervention strategy is described.
  • Plans for building consensus are weak. The project team plans to build consensus by using a survey rather than engaging with the campus community.
  • The timeline is sketchy; it is not clear whether the team can accomplish in the time allotted all that is needed.

Assessment

  • The choice of assessment approach is not clear.
  • The assessment forms are not completed correctly.
  • The number of students enrolled in the pilot sections is too small and will thus invalidate the assessment.
  • The proposal simply states, “There will be a ‘common assessment task’ that all students must complete.” What is that task?

Cost

  • It is not clear how the project will reduce cost.
  • An enrollment growth scenario is presented but is not supported with historical data or other institutional changes (e.g., curricular changes) that would lead to enrollment growth.
  • The cost reduction plan is hypothetical. The savings in faculty time from reducing the number of face-to-face teaching hours, eliminating duplication of labor to prepare materials, and using automated feedback can be allocated to “other courses, thereby reducing the need for additional casual staff employment” or “research projects in the field, thereby strengthening the research capacity of the staff.” The saved time can also be used to go to the beach or plant a garden. Because these reallocations are hypothetical and do not represent a concrete plan, the proposal should not be accepted unless specific decisions have been made about how the saved time will be reallocated to benefit the institution.
  • The CPT is not completed correctly.

Project Budget

  • The project budget may not cover all that is required based on the activities described in the plan.
  • The project budget includes ongoing, operational costs such as training of undergraduate learning assistants or graduate teaching assistants. Those costs should be calculated on the CPT as part of the continuing cost of the course because the costs are ongoing.
  • The project budget includes software licenses that should be calculated on the CPT as part of the continuing cost of the course because the licenses are ongoing.

Characteristics of a Proposal Rated 3 (Should Not Be Accepted)

The two most common reasons for rejecting a proposal that NCAT has encountered are (1) the proposal does not meet the program’s application guidelines with regard to cost reduction and (2) the proposal is a “plan to plan”—meaning, it is too general and lacks detailed planning.

The Proposal Lacks a Valid Cost Reduction Strategy

The proposal has no cost-reduction strategy that is a product of course redesign and/or it demonstrates a lack of seriousness about cost reduction.

Examples

  • The plan is to reduce the number of hours spent by each instructor from 225 to 180 hours, —a reduction of 45 hours, which is 0.028% of an instructor’s required workload of 1,595 hours.
  • The plan is to reduce staffing by one graduate student, a savings of $7,250 out of a total course cost of $164,638—a 4% savings.
  • The plan is to reduce costs by making garden-variety academic decisions. For instance, the institution offered too many sections in the past that didn’t fill, making per-student costs high and merely plans to reduce the number of sections offered to fill the sections. This is not course redesign but instead good academic management.
  • The plan relies on increased retention as its sole method of reducing costs. Although many course redesigns have produced significant gains in completion rates, especially in mathematics, many redesigns produce relatively modest gain (less than 10%). Even if the course enrollment is large, often you cannot reduce sections (i.e., reduce costs) because the retention improvement is not sufficient to eliminate a section. Even when completion rates improve more than 10%, you must have the ability to actually reduce sections. Sometimes the numbers just won’t work. Finally, even if the number of students enrolled in the course is large and you take the necessary steps to reduce the number of sections offered, the impact on cost reduction may be quite small. Retention cannot be relied on as a cost reduction strategy. (See http://www.theNCAT.org/Newsletters/Jul10.html#1 for an expanded discussion of this issue.)
  • The plan advances a bogus definition of cost reduction: “cost per successful student.” Even though the redesign plan may be more expensive than the traditional model, the argument goes, it is not if you redefine cost reduction. The accepted way of calculating a course’s cost per student is to divide the total cost of offering the course by the number of students enrolled in the course. For example, if the traditional course cost is $100,000 per 500 students, the cost per student is $200. If a redesigned course cost were $75,000 per 500 students, the cost per student would be $150. The cost per successful student is derived by dividing the total cost of offering the course by the number of students who pass the course. For example, if the traditional course cost is $100,000 per 300 students passing the course, the cost per student is $333. If the redesigned course cost were $130,000 per 500 students passing the course, the cost per student would be $260. The problem with that argument is that institutions are not funded based on successful students; they are funded based on enrolled students. Students do not pay tuition based on whether they succeed; they pay tuition based on whether they enroll.

The Proposal Is a “Plan to Plan”

Some proposals are so general or so brief that it is not clear what the team plans to do or whether it is possible to actually implement the redesign plan.

Examples

  • The proposal fails to address adequately many of the important categories listed in the Final Project Plan Format—specifically the Essential Elements of Course Redesign, a description of the learning materials to be used and a plan to build and maintain ongoing consensus about the redesign. The proposal itself is a plan to plan. It lacks the specificity needed to be fully implemented within the project time frame.
  • A lot of decisions are yet to be made. “Existing content across all iterations will be filtered and consolidated, and when appropriate, new content and new learning objects introduced. The syllabus, assessment, and student interaction and engagement activity will be reconstructed in the learning management system. In-class meetings will supplement this; however, careful consideration will be given to reducing the number and frequency and to changing the nature of these face-to-face meetings.”
  • A lot of decisions are yet to be made. The team plans to retain lectures in a writing course (including vodcasts), even though the team knows that students dislike them. This is a skills-based course requiring as much practice as possible, and a good redesign would focus on that. NCAT knows from years of experience that skills-based courses are best taught in a face-to-face lab setting with lots of personalized support available to students—not in lectures or online.
  • At-risk students (not clearly defined) will be identified and sent to do something (what that something is, again, is not described) with learning advisors, who are neither clearly described nor included in any of the cost calculations.
  • The changes anticipated in the redesign are minimal: “Classes will meet slightly less: lectures are reduced from 12 to 10, and workshops from 11 to 9. Class sizes are unchanged.” The anticipated changes are not clearly described and appear to be a laundry list of possibilities (e.g., “In-class lectures will utilize student response systems, vodcasts, MyCompLab quizzes, and textual materials.”)This is not course redesign as we know it. This is only a minor change in the course structure, with the addition of some technology to the existing format.

Q: Do you have examples of good proposals?

A: Examples of good proposals with high levels of planning detail can be found at http://www.theNCAT.org/PlanRes/Proposal%20Examples.htm.

Q: Shouldn’t a lot of these issues have been dealt with earlier on in the process?

A: Yes. NCAT’s process is designed to develop strong course redesign plans and weed out or discourage weak ones before they reach the final proposal stage. Usually, the final proposals we receive are quite good. But the level of detail we require in a final proposal is much higher than most faculty members are accustomed to producing in writing grant proposals, particularly in instructional reform. Inevitably, some pieces will be missing or get glossed over. Unfortunately, some faculty teams fail to take the program guidelines seriously—especially in regard to the cost reduction part of the program—until learning their proposals have not been accepted. The program leaders should make every attempt to resolve outstanding issues during the proposal development process but should also be prepared to say no when warranted.

Q: Can a weak proposal produce a strong result?

A: We have never had a weak proposal that was accepted into a program produce positive results—never! In its national programs, NCAT simply does not accept weak proposals. We spend a lot of time working with teams to strengthen weak proposals and turn them into strong redesign plans. We have also conducted programs in partnership with other entities, when we did not have the final decision to accept or reject a proposal. In those programs, we encountered a number of instances when the partner wanted to accept a weak proposal for political reasons (e.g., a state or system that wanted a project at every member institution, an institution that wanted a project in a certain discipline, and so on). In other instances, the partner had announced it would award, for example, 10 grants, but only 8 of the proposals were strong. The decision was made, against NCAT’s advice, to accept two additional, weak proposals in order to reach the predetermined number. We repeat: Never has a weak proposal produced a positive result. In just about every case, the project fails to reach full implementation of the redesign plan.

Q: Who should review the full proposals?

A: The project leaders (the full team) should read and rank all of the proposals individually and then meet as a team to discuss the rankings and the proposals. The team should make recommendations to the provost about which proposals should be accepted after working through any outstanding issues. The provost should make the final decision.

Q: How can we make sure certain departments we want represented do get represented even if their proposals are weak?

A: Remember that there are two kinds of weak proposals: the 2s that may need significant work but can be improved within a reasonable time frame, and the 3s that cannot be. If the target department submits a 2, then you should work with the department to strengthen the proposal. But if the department submits a 3, it is not worth the time to work further in the context of the redesign initiative. The result will be poor and will undermine the entire program. You should consider working with the department throughout the year to get it ready for a subsequent round.

Q: What if increased retention is a campus priority, and a proposal’s cost reduction strategy relies on retention and cites that priority?

A: Increased retention is a very important quality improvement goal at many institutions. It is not a way to reduce costs in the vast majority of cases.

To consider accepting increased retention as a cost reduction strategy, you need to make some demands on the team. You cannot let the team assert that it will reduce costs by meeting the campus priority of increased retention. First, you need to require the course redesign team to project a plausible increase in course completion (e.g., 10% is reasonable; 50% is not). Second, you need the team to show you how it can drop at least one section, given the structure of the course. For example, a course that enrolls 1,800 students in six sections of 300 each and that improves pass rates by 4.6%, which equals 83 students, cannot eliminate a section. Course completion rates would have to improve by 17% in order to eliminate a section. Third, the changes have to be costed out to decide whether the percentage of reduction is acceptable. For example, an early NCAT redesign enrolled 2,200 students in 25 to 30 sections of 80 to 100 students each. At the time, the total cost of the traditional course was $247,170. University data collected in earlier redesigned courses showed a 7% increase in retention. Applying that increased retention rate to 25 redesigned sections resulted in a one-section reduction (7% of 2,200 students, which is 154 students), amounting to a cost savings of only $8,239 (the total cost of a traditional section, including personnel and classroom space rental)—a 3% reduction. If you’re happy with 3% (we are not) and the reduction can be documented, then you should consider accepting the proposal. But remember: in most cases, it’s very difficult to meet those requirements.

 

 

Table of Contents

Introduction
I. The Critical Components of a Successful Course Redesign Program
II. Assessing Institutional Readiness to Redesign
III. Making Key Decisions before the Program's Launch
IV. Developing a Plan of Work
V. Building Awareness and Capacity
VI. Assessing Course Readiness
VII. Preparing Teams to Submit Strong Proposals
VIII. Selecting Proposals That Will Succeed
IX. Monitoring the Redesign Implementations
X. Maintaining Consensus and Ensuring Sustainability
XI. Building Capacity and Scaling Initial Success

Appendices:

A. Plan of Work
B. Publicity Plan
C. Call to Participate
D. Application Guidelines
E. Workshop I Agenda
F. Workshop II Homework
G. Workshop Logistics
H. Workshop II Agenda
I. Workshop III Invitation
J. Workshop III Agenda
K. Workshop IV Invitation
L. Workshop IV Agenda
M. Final Report Format
N. Program Evaluation