We have no comparisons for the data gathered this semester. Some faculty might argue that we should wait until these same tasks are reassessed in 2012. Others might argue that all students who passed should be able to do these tasks. Still others might argue that at least some minimum percentage (50%?, 70%?, 80%?) should be able to do these tasks.
In any case, for almost every task question less than half of the passing students could answer it. I think instructors of the downstream courses (CS146, CS151, and CS160) would rightly complain that this is far too few.
Many of the CS46A/B task questions asked students to write methods that processed arrays, strings, files, and generic collections. In some cases students had to first devise simple algorithms, use recursion, or deal with interfaces.
In my view the questions seemed clear and fair. They were reasonable interpretations of their corresponding tasks. In turn, the tasks were consistent with OC8-relevant course outcomes listed in the syllabi and they were pre-approved by the instructors.
So what went wrong? Are the tasks and related course outcomes out of synch with what really happens in these courses? If so should the syllabi be changed? Is it reasonable to ask students to be able to do these tasks? Are we doing a bad job teaching students to do these tasks? Are we reading too much into the data?
Kim, Horstmann, and Taylor are among our most conscientious instructors. Moreover, Horstmann wrote the book used in CS46A. If the problem really is with our teaching, then we should all change our majors.
I speculate that the problem is that students simply need more practice than they are allowed by the syllabi. In part, this is because the syllabi are inefficient and don't prioritize the course outcomes.
As the author of the CS151 report I found it difficult to isolate specific tasks. For example, I asked a question on the final that required students to combine the Strategy design pattern and multi-threading. I used this problem to assess task 2: multi-threading and synchronization. Only 38% got the problem correct, but some of those who missed points did so because they missed the strategy pattern part of the question and not the multi-threading part. Another case was a question I asked that required students to customize a framework. To do this they had to create listeners for menu items. The listeners were implemented as inner classes of the GUI, but this was only a very small part of the overall problem. In particular, inner classes are too general to belong on the task list. Also, perhaps some of these problems could have been avoided if the tasks were known further in advance.
Another problem was that I didn't have the time to cover all of the material mentioned in the tasks. There are six tasks relating to object-oriented programming in the rubric for an object-oriented design course. To me that indicates that the OOP material isn't being sufficiently covered in the prerequisite courses.