 |
 |
|||||||||||
 |
||||||||||||
 |
The Roadmap to Redesign (R2R)
Texas Tech University Course Title: Comprehensive Spanish Review First Year Project Abstract
Comprehensive Spanish Review First Year is designed for students who have had two years of high school Spanish. The course has a history of not being able to meet demand. In 2000-2001 only 791 students were accommodated and an estimated 1100 were turned away each semester. An increase in support for graduate part-time instructors (GPTIs) and graduate teaching assistants (GTAs) accompanied by intensive recruitment of graduate students has allowed Texas Tech to increase capacity to 1285 in 2003-2004 and to 1595 in 2004-2005. The increase in course capacity was accomplished by raising section size in traditional sections to about 30 students, which exceeds Association of Departments of Foreign Languages (ADFL) maximum class size recommendations by 50%. In addition, two non-traditional versions of the course—two large (110 students) lecture/discussion and one online/TV version (90 students)—are offered. The goal of the redesign is to meet current and anticipated growth in demand, to reduce maximum sizes of classes to within ADFL's recommended 20, and to improve language mastery—without increasing costs. In choosing the Replacement Model, Texas Tech will follow the lead of all three core Spanish practice associates. The redesigned five-credit course will more than double the number of regular sections and limit enrollment to 20 students per section. Sections will meet physically three times a week with class time devoted to communicative exercises emphasizing oral skill development. Workbook, grammar, and writing components will be moved online. Students will spend two hours of online practice in grammar with automated immediate diagnostic feedback and will write weekly compositions that will be semi-automatically graded with diagnostic feedback. Students will also participate in one hour of language lab weekly. Studio sessions taught by GTAs will be held for students scoring less than 80% on the first exam. Online teaching of grammar and writing with rapid diagnostic feedback will be more effective than traditional in-class workbook exercise time. The amount of time spent on oral skills will increase. The one-third reduction in class size to within ADFL standards will also facilitate learning, and studio sessions will provide systematic help for weaker students. During the pilot phase, parallel traditional and redesign sections will be taught using the same GPTIs for both versions. Comparisons of exam and other performance will be made, controlling both for teaching effectiveness and instructor personality. During the full implementation phase, students will be compared on performance indicators against earlier groups of traditional course students and those in the online course. In both phases, the team will use student-based evaluation measures, response-based evaluation measures, and learning-based evaluation measures with the data subjected to appropriate statistical analysis. Texas Tech plans to accommodate growth while reducing class size using the same personnel and fiscal resources. The more expensive lecture/discussion and online/TV versions of the course will be eliminated. GPTIs will teach 40 students in two sections rather than 32 students in one per semester. Since in-class time and the time spent on grading will be significantly reduced, their workload will decline despite the increase in students. The mix of personnel teaching the course will change by reducing the number of faculty involved to just one faculty coordinator, having GPTIs teach all sections, and redeploying TAs from the large lecture/discussion section to teach studio sessions. Course capacity will increase by 17% while the cost-per-student will decrease by 13%. Most importantly, Texas Tech will achieve a 34.6% decrease in regular section class size with no additional budget resources required and the overall cost of the course essentially unchanged. Interim Progress Report (as of 5/15/05) In spring 2005, Texas Tech University (TTU) implemented a pilot of its redesign with 8 traditional sections (N = 255 students) and 8 redesigned sections (N = 159 students.) Instructors of varying teaching abilities were assigned in an attempt to try the redesign with a broad array of instructors, thus eliminating a possible bias if only the best instructors were part of the redesign. A controlled enrollment procedure was used to obtain matched student samples in the redesigned and the traditional sections. This procedure was only partially successful: significantly more females took the redesigned version, and students in the traditional sections were significantly further along in their undergraduate careers. Withdrawals were low (10.56% for redesign and 13.06% for traditional) with about half occurring during the twelve-day drop/add period. There was an extremely high level (>90%) of on-time compliance in completing online assignments by students in the redesign sections. Learning outcomes for reading, writing, speaking, listening, and basics (vocabulary and grammar) were assessed via end-of-semester evaluations. Evaluation methods were typical for first-year language courses at TTU and at other institutions. There were no significant differences in learning on any of the five parameters or overall between traditional and redesigned sections. Where there were small average differences, they favored the redesign. TTU’s data analysis suggests that the number of minutes per student per week in class (as a surrogate measure of individual student time-on-task) is more critical than how many minutes a week students are in class. The minutes per week/number of students ratios are essentially the same for the traditional and redesign sections despite the former meeting 250 minutes a week and the latter 150. Students in the redesigned sections generally rated the instructors and the effectiveness of the course lower than did traditional students. This may be due in part to the fact that there was considerable resistance on the part of the redesign instructors related to the use of technology and curricular innovation. That negativity appears to have been conveyed to redesign students. (That attitude seems not to be present in a second pilot during summer 2005.) In addition to performance data, the team collected data on demographics, technology competence, motivation towards the course, and mid-semester and final exam results from 411 students. In addition, TTU has response and time-on-task data for every item of every try by every student for the online interactive drill for the redesign students, multiple standardized observations of traditional and redesigned class sessions, and copies of all compositions, exams, etc. for all students involved in the pilot. Data analysis has been in progress during summer 2005. A particularly surprising result was that quality of teaching (judged on the basis of supervisor assessment and student course evaluations) had no effect on student learning: students learned the same regardless how good or bad the teaching was. ANOVA and correlational analyses indicate this is not an artifact of instructor grading biases (e.g., poorer instructors grading more leniently to get better evaluations). Further investigation suggests that this is an artifact of a deeper problem: unreliable outcomes assessments. Factor analysis indicated that TTU is measuring 19 different things, not the four skills and underlying basics. Only the basics assessments enjoy acceptable Kuder-Richardson reliability measures. As a result, the team is completely redesigning the learning outcomes evaluation and performance measures for the fall 2005 full implementation of the redesigned course. The primary problems during the pilot included 1) instructor hostility to technology and curricular innovation, 2) a course coordination structure that proved inadequate to insure reasonable similarity across traditional and redesigned sections, and 3) some problems associated with changing the textbook that resulted in reducing the actual differences between the traditional and redesigned versions of the course (roughly going to an online workbook and lab manual in the traditional sections rather than the previously standard use of paper workbook and language lab). TTU also believes that far more extensive training and preparation for teaching the redesigned course is needed; consequently, a four-day workshop before classes begin in the fall plus a mentored supplemental-model version for training new instructors not yet ready to teach on their own will be put in place. In summer 2005, TTU conducted a second pilot to try some of the changes made in response to the first pilot. These included going back to using the language lab and paper lab manual in the traditional sections, improving testing and grading, and determining whether in the second go-around there would be less hostility towards the technology and curricular innovations. (Preliminary information suggests this is the case.) TTU also needed to find out if the redesign would work in the compressed schedule of summer sessions. For the summer pilot, TTU used the supplemental model for two reasons. In order to analyze data and be ready for fall 2005 redesign full implementation, the team did not have enough time to train all instructors adequately to teach the redesigned version but were able to train them to use the online component as a supplement. Also, in the fall, TTU plans to have both replacement and supplemental model versions of the course. Most students will be in the former, but about 60 will be in a supplemental model version that will be used to mentor and train instructors who are not yet ready to teach a course on their own. The latter will meet five days a week (one day in large lecture, four days in small sections with the mentor co-teaching once per week) and will include the online components of the replacement model redesign. Full implementation of the redesign (36 sections) will be implemented in fall 2005 using a much changed team coordination structure with separate coordinators for different components of the course under the oversight of the PI as Executive Coordinator. Testing and evaluation procedures are being significantly altered in an attempt to obtain reliable and valid performance outcomes measures. Much more extensive initial training and on-going mentoring and training will be in place. In fall 2005 TTU will begin planning a similar replacement model redesign of SPAN 2301 for spring 2006 pilot or full implementation and plan to do so for SPAN 2302 for fall 2006 pilot or full implementation. As part of the development of the redesign, the Department of Classical and Modern Languages and Literatures (CMLL) is providing about $136,000 in special salaries for project graduate research assistants and for technology enhancement. In addition to this there is significant involvement of the five regular faculty, all of whom receive some sort of released time for curricular involvement, as well as CMLL technology support and Language Lab personnel. Interim Progress Report (as of 4/15/06) Texas Tech completed full implementation of the redesign for SPAN 1507 in fall 2005 with 720 students in 36 replacement model redesign sections and 60 in a supplementary model Lecture-Discussion version used to train new TAs under close monitoring and mentoring. The team adopted a new coordination structure with an Executive Coordinator and five others broken into teams for coordinating in-class, language lab, online, and testing/assessment components of the course. A four-day workshop was held to train teaching personnel before classes began as well as weekly meetings with the teachers and a weekly coordination team meeting. The project has three graduate research assistants assigned full-time to the project as well as faculty members from Spanish and Applied Linguistics and the Department Chair who is the project PI and Executive Coordinator. Thus far, the full implementation is working very smoothly with no significant problems. The team is collecting survey and performance data similar to what they did in the pilot studies. They are also developing new testing protocols for Listening Comprehension, Speaking, Reading , and Writing that are derived from ACFL standards and operationalized as subjective protocols for Speaking and Writing and objective protocols for Listening and Reading . These will be validated as to reliability and construct validity during the semester. The team has not yet analyzed the survey and performance data (over 100,000 data points) from the fall 2005 semester, but anecdotal and impressionistic information indicates it was very successful. In spring 2006, Texas Tech piloted a similar redesign of the first-semester second year course (SPAN 2301) with 12 redesigned sections (240 students) and 12 traditional sections (360 students), with planned full implementation scheduled for fall 2006. The development and validation of ACTFL -derived assessment /grading protocols for each of the four skills continues. In November 2005, the team reported on the redesign project at a session of the ACTFL meeting in Baltimore. They are also sharing their redesign approach with the University of Missouri at St. Louis. In the redesign, did students learn more, less or the same compared with the traditional format? Improved Learning Texas Tech fully implemented its redesign of Comprehensive Spanish Review First Year (SPAN 1507) in fall 2005 and piloted a similar redesign of the first-semester second-year course (SPAN 2301) in spring 2006. Since the team has better comparison data for the latter, they include it in their assessment. Students were assessed using a post-test that addressed reading comprehension, writing, listening comprehension, speaking, grammar, and vocabulary. Overall, student learning of Spanish either improved under the redesign or showed no significant difference. In the SPAN 1507 pilot, there were no significant differences in student performance on any of these dimensions. Because defects were found in the standard testing and assessment methods (based on publisher-provided test banks) being used, however, vocabulary and grammar were the only two learning dimensions able to be assessed comparatively. The team developed new testing protocols and methods as part of the SPAN 1507 full implementation. Redesign students performed significantly better than traditional students in vocabulary (a mean score of 81.04 vs. 74.18 for traditional students), and there were no significant differences in performance between the two groups in grammar (70.37 redesign vs. 71.35 traditional). Oral skills could not be comparatively assessed, but the redesign students performed at the national average. The SPAN 2301 pilot allowed more comprehensive comparisons:
Improved Retention
Were costs reduced as planned? Texas Tech carried out its plan to increase the number of students enrolled in the redesigned course from 1590 to 1860 annually. The cost reduction techniques described below enabled Texas Tech to redefine a normal teaching load from one 32-student section to two 20-student sections per instructor in the five-credit-hour version of the course and from two 30-student sections to three 20-student sections in the three-credit-hour version of the course accompanied by an overall decrease in instructor workload. As a result, class size was reduced from 32 to 20 while course capacity was increased without increases in classroom demand or instructional cost. The overall cost of the course declined 13%, from $326 per student to $283 as planned. Pedagogical Improvement Techniques What techniques contributed most to improving the quality of student learning? Careful division of learning activities between in-class and online. Texas Tech’s basic strategy was to use technology where prior research indicated it was most effective and to use class time where it was most effective. The result was carefully scripted class sessions that focused on oral skills development, with reading, writing, grammar, and vocabulary taught online. Training. Of special importance was having a four-day workshop before each semester began followed by weekly training sessions each semester for all teaching personnel. Team approach. The team coordination approach adopted for the implementation stage enabled effective monitoring and coordination of each aspect of the redesigned course. Automated grading. Improved consistency in grading and feedback by automated or machine grading where feasible and comparison re-grading of work graded by humans improved the consistency of feedback across as many as 36 sections. Assessment methods. Improved testing and assessment methods improved student understanding of what they did and did not know. Cost Savings Techniques What techniques contributed most to reducing costs? Automated grading. Online grammar and vocabulary drills with automated grading and semi-automated grading of writing exercises reduced overall instructor out-of-class time. Standardized class plans. Standardized class plans with supplied teaching materials virtually eliminated class preparation time other than weekly training and practice sessions. Reduced in-class hours. The hybrid format reduced in-class instructional hours from five to three per section. Implementation Issues What implementation issues were most important? Instructor training. The full implementation went very smoothly, and there were no substantial implementation issues. In particular, instructor resistance to change experienced during the pilot did not occur during implementation. The extensive pre-semester and on-going in-service training helped instructors be more comfortable in what they were doing. The smooth full implementation was facilitated in large part by changing from the traditional coordination structure to a team one. Student satisfaction. Students were generally satisfied with most elements of the redesigned course including the mix of different activities targeting the development of the four skills plus vocabulary and grammar. Students were most strongly polarized regarding the online drill component of the redesign, with three-fifths feeling they helped them learn grammar and vocabulary and two-fifths believing it did not. Yet the performance evidence is that on average they learned better. During the pilot stage student satisfaction with instruction and the course dropped substantially compared to the old course. But by the second semester of full implementation, student evaluations had rebounded to a level exceeding the pre-redesign evaluations. Will the redesign be sustained now that the R2R program has concluded? The course redesign is intended to be long-term. As long as the department retains its present chair, the redesign will not be abandoned or substantially altered. The biggest threat to sustainability is ongoing instructor compliance. The team coordination structure combined with the pre-semester training workshop and weekly in-service training are critical to sustaining instructor compliance and cooperation. Another threat is the fact that textbooks have new editions every four years, and the team has to redo the online components substantially and the in-class components to a lesser degree. Texas Tech has created a solution to this problem where the cost of adaptation to a new textbook is less than $6,000. Will you apply the redesign methodology to other courses and programs on campus? The team has already piloted the extension of the redesign to second-year Spanish (SPAN 2301) in spring 2006. In fall 2006 they will fully implement the redesign in SPAN 2301 followed by SPAN 2302 and 2607 in spring 2007. The primary change between the second-year pilot and the full implementation is replacing one credit of in-class hours with two hours of online rather than the one hour used in the pilot. When the second year implementation is completed, about 5,000 student enrollments will be impacted annually. The redesign has been developed in a manner that is largely textbook and language independent. The cost of adapting the on-line components to another language is about $6,000 per year of language course. In languages other than Spanish, the team does not have the need to reduce class size to meet demand, so the redesigns will typically follow the supplemental model. During the summer 2005 second pilot and the fall 2005 implementation, the team also developed and evaluated lecture-discussion versions of SPAN 1507 that followed the supplemental model. It was comparably effective. During summer 2006, the German department converted their workbooks to the online drills and will adopt a supplemental model version in the fall 2006 semester. Texas Tech is encouraging other languages to follow suit. During fall 2005, the team also began using versions of the online components of the redesign in its intensive immersion language programs in Spain and will begin doing so in Germany this fall.
|
 |
Quick Links: |
 |
||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|