 |
 |
|||||||||||
 |
||||||||||||
 |
Implementation Issues Carnegie Mellon University Looking back on the course pilot itself, what worked best? One strategy that worked well was to provide some time in the lecture-part of the course to introduce students to the structure of the StatTutor interface. This time was not just an exercise in learning a new software tool, however. Instead, the instructor used this time to discuss the general approach to exploratory data analysis problems and how the major problem-solving goals are the same across problems. That said, one surprise was the difference between lecture attendance and lab session attendance. The students who had been to lecture before the first StatTutor lab session were well prepared (e.g., they jumped right into the exercises before any TAs arrived), whereas the other students (more than 25%) were somewhat surprised because they had missed the relevant lecture. Nevertheless, these latter students managed to get the system going without any trouble and did not appear to be slowed down at all. December 2001 Update: The summer experiments described above worked well. They provided valuable information about the impact of this pedagogical strategy. This kind of experiment is a natural strategy for a psychologist such as Lovett, especially given its connection to her research. It is a less natural thought for most faculty engaged in course redesign. But running this kind of small-scale experiment on a new teaching strategy would seem to be an aspect this project that is worth recommending as a more general strategy for those engaged in course redesign. It is comparatively low-cost, but can provide critical data about aspects of any redesign if the experiment is well constructed. Another strategy that seems to be working well is addressing computer application interface problems by working with an expert in human-computer interaction. Using money from the grant, we have now employed a recent graduate from Carnegie Mellon's Human-Computer Interaction (HCI) program to improve the StatTutor interface. She is deploying standard HCI methodologies, centered on user testing, to change features of the interface to improve the effectiveness software. For example, she has already determined that the feature of the current interface that leaves previously answered question open on the screen is problematic. Students reported this feature produced a distraction that may have impacted their ability to concentrate on the next steps in the lab. What are the biggest challenges you face in moving from the course pilot to the project's next phase? Our goals have remained primarily the same. One new emphasis is on using the StatTutor system for students’ homework assignments. Given the result of our course pilot, it is clear that this is a way to further reduce TA labor while improving students’ level of hands-on engagement with the material. CMU’s 2001 summer activities included design of new homework assignments supported by StatTutor for fall 2001, replacing old homework assignments graded by hand; expanding StatTutor to cover the curriculum in the second course in statistics; and improving the connection between the StatTutor interfaces and the underlying intelligent-tutoring system software so that the full power of the cognitive tutor emerges. The product will be a fully implemented intelligent tutoring system intended for almost all introductory statistics classes scheduled for fall 2001. This will test whether StatTutor can be an effective substitute for some of the teaching assistant labor in statistics labs. In addition, the use of StatTutor is being extended to supporting students in homework exercises. The longer term vision is to have StatTutor as a regular resource for helping students in doing their homework. In the coming year, we plan to run the StatTutor system off a server that will collect student data over the network. This may pose some technological challenges, but we feel they will be well balanced by the greater ability to provide feedback and scaffolding support based on a student’s history of problem-solving performance rather than just their performance on the given StatTutor exercise. A major challenge that we faced until recently was finding programmers to work on moving from pilot to full implementation. Even at Carnegie Mellon, hiring programming expertise, especially in Java (the front end of StatTutor) and LISP (the intelligent part of StatTutor) is a constant challenge. We recognized this as a necessary part of the cutting-edge nature of this project where using commercial solutions is not really possible. Although not unexpected, it is worth noting that this project has had to deal with of those "platform changes" that are so disruptive to educational technology in general. In most cases, the "platform change" that causes disruption is something like an operating system upgrade or an upheaval in markets that leads an institution to change from supporting one kind of computer to another. The change in this case is from a research to a production environment for cognitive tutors. Many faculty have done their research on a particular cognitive model that will form the basis for a tutor in our "ACT-R" development environment. This literally refers to a set of LISP programs that support the creation of a particular tutor. To turn a tutor into a production system for delivery to students, it has to be moved to our "TDK" development environment. Again, the TDK (which stands for "Tutor Development Kit") is a set of LISP programs that are tools for writing a tutor. We are in the process of moving the "backend" of StatTutor from the research environment (which supported the prototype work) to the more production-oriented TDK environment for full implementation. December 2001 Update: The challenges of recreating the "intelligent" backend of StatTutor in the newer Tutor Development Kit (TDK) environment proved much more challenging that we imagined when reporting in July 2001. The challenge was quite simply one of lack of personnel. We finally managed to hire a programmer, Ross Strader, capable of learning to program in the TDK and doing the necessary work. This process reveals many challenges that are important for larger policy considerations. First, there is simply a paucity of students who both know about intelligent tutoring systems and who are interested in creating working versions of such systems. (Students are typically more interested in the cognitive research related to these systems than in actual production of tutors.) Ken Koedinger, one of the cognitive scientists who works on Cognitive Tutors at Carnegie Mellon, often says that intelligent tutoring systems will not have the impact that they could until we find a way of developing a workforce of "cognitive engineers" who will develop the necessary expert knowledge representations and program it into working tutors. In fact, Ken himself devoted many, many (gratis) hours of work with Ross Strader to teach him enough both about the logic of the TDK and programming in that environment before Strader could begin developing the StatTutor exercises in the TDK. Without Ken's help on this aspect of the project, our timeline would have slipped even more. (This is another indication of the importance of having a general institutional commitment that goes beyond the principle investigators to any such redesign project.) Because of these delays, Lovett and Meyer were not able to develop StatTutor versions of labs that involve only single variables. (The three labs we have been able to deploy address statistical techniques with two variables.) For this reason, StatTutor versions of assignments could not be introduced in the early labs, all of which involve single variables. The cascade effect was unfortunate. It meant that not enough labs were ready to reduce the numbers of TAs, even experimentally, in the fall of 2001 as hoped. So, the students did not encounter StatTutor from the beginning. This also meant that the project of creating homework assignments with it had to be delayed by a semester. We found this an interesting lesson. It was one of those cases where we had plenty of monetary resources to address the technical problems and, indeed, knew exactly what needed to be done, but simply could not find the appropriate human resources to meet the need. In the end, the solution was to train our own.
|
 |
Program in Course Redesign Quick Links: Program In Course Redesign Main Page... Lessons Learned: Savings: Project Descriptions: |
 |
||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|