BSCS Assessment Report Fall 2009

Assessment Activities

During the Fall 2009 semester the program outcomes were reworded and renumbered:

Program Outcomes

The assessment process was also radically altered. Under the new process a rubric is associated with each entry in the courses/outcomes matrix. Each rubric asks for the percentages of passing students able to perform certain pre-defined tasks.

The Assessment of BSCS.OC8

To test the new process standard tasks were selected for program outcome eight (BSCS.OC8):

The ability to write programs that demonstrate comprehensive knowledge of object-oriented programming.

In addition to BSCS.OC8, three other BSCS outcomes are related to programming ability. Roughly, they are:

BSCS.OC9: Ability to implement data structures
BSCS.OC10: Ability to implement GUIs
BSCS.OC11: Ability to write programs in a functional language

These outcomes are addressed at various levels (B, I, and A) by the following courses:

	CS46A	CS46B	CS146	CS149	CS151	CS152	CS160
BSCS.OC8	B	I			A
BSCS.OC9		B	A	I	I
BSCS.OC10	B				I		A
BSCS.OC11						I

Next semester (Fall 2010) outcomes 9 and 10 will be assessed. This should give us a pretty clear picture of how well our students are learning to program.

(My real motive for mentioning these other outcomes is to forestall critics who complain that the rubrics don't get a full picture of this or that course. They aren't supposed to. Instead, their aim is to get a full picture of BSCS.OC8.)

The Raw Data

Each rubric specifies a list of tasks. For each task instructors describe an exam question or homework problem representative of the task. I will refer to these as task questions. For each task question the instructor records the percentage of passing students (i.e., students who received a C- or higher in the course) who received full or near-full credit.

Beginning Level

BSCS.OC8 is covered at the beginning level by CS46A.

This semester there were three sections of CS46A. Here are the raw reports and referenced data:

 

Intermediate Level

BSCS.OC8 is covered at the intermediate level by CS46B. 

This semester there was one section of CS46B. 

Advanced Level

BSCS.OC8 is covered at the advanced level by CS151. 

This semester there was one section of CS151. 

Summary

The entries indicate the percentages of passing students who received full or near-full credit on the task question representing the corresponding task:

Notes:

1. Kim combined the statistics from two sections. I simply reported the same results in two columns.
2. Kim reported two questions that addressed CS46B.Task3. I averaged the results.
3. NT = Covered, but not tested.
4. NC = Not covered.
5. NA = Not applicable (because the corresponding section didn't exist).

The Analysis

We have no comparisons for the data gathered this semester. Some faculty might argue that we should wait until these same tasks are reassessed in 2012. Others might argue that all students who passed should be able to do these tasks. Still others might argue that at least some minimum percentage (50%?, 70%?, 80%?) should be able to do these tasks.

In any case, for almost every task question less than half of the passing students could answer it. I think instructors of the downstream courses (CS146, CS151, and CS160) would rightly complain that this is far too few.

CS46A/B

Many of the CS46A/B task questions asked students to write methods that processed arrays, strings, files, and generic collections. In some cases students had to first devise simple algorithms, use recursion, or deal with interfaces.

In my view the questions seemed clear and fair. They were reasonable interpretations of their corresponding tasks. In turn, the tasks were consistent with OC8-relevant course outcomes listed in the syllabi and they were pre-approved by the instructors.

So what went wrong? Are the tasks and related course outcomes out of synch with what really happens in these courses? If so should the syllabi be changed? Is it reasonable to ask students to be able to do these tasks? Are we doing a bad job teaching students to do these tasks? Are we reading too much into the data?

Kim, Horstmann, and Taylor are among our most conscientious instructors. Moreover, Horstmann wrote the book used in CS46A. If the problem really is with our teaching, then we should all change our majors.

I speculate that the problem is that students simply need more practice than they are allowed by the syllabi. In part, this is because the syllabi are inefficient and don't prioritize the course outcomes.

CS151

As the author of the CS151 report I found it difficult to isolate specific tasks. For example, I asked a question on the final that required students to combine the Strategy design pattern and multi-threading. I used this problem to assess task 2: multi-threading and synchronization. Only 38% got the problem correct, but some of those who missed points did so because they missed the strategy pattern part of the question and not the multi-threading part. Another case was a question I asked that required students to customize a framework. To do this they had to create listeners for menu items. The listeners were implemented as inner classes of the GUI, but this was only a very small part of the overall problem. In particular, inner classes are too general to belong on the task list. Also, perhaps some of these problems could have been avoided if the tasks were known further in advance.

Another problem was that I didn't have the time to cover all of the material mentioned in the tasks. There are six tasks relating to object-oriented programming in the rubric for an object-oriented design course. To me that indicates that the OOP material isn't being sufficiently covered in the prerequisite courses.

The Recommendations

We are losing too many students in CS46A, both physically and mentally. The students who barely pass CS46A become casualties in CS46B or CS151. Let's slow the pace down. There are two ways we might do this.

Recommendation 1

There's too much material in CS46A and CS151 and not enough of the right material in CS46B and CS146. Rewrite the syllabi and rubrics for CS46A, CS46B, CS146, and CS151. These courses form the backbone of our efforts to teach students how to program in Java. It seems strange that two of the four are devoted to data structures. It also seems (to me) that CS146 is a bit anachronistic in its devotion to sorting, searching, and NP.

As a suggestion, CS46A could be devoted to problem solving in Java. Students would learn to work solutions out on paper, and then translate them into static Java methods. CS46B could be an introduction to object-oriented programming in Java, where students are drilled in interfaces, classes, objects, inheritance, and polymorphism. CS146 could be an advanced course in object-oriented programming covering topics like GUIs, file I/O, multi-threading, reflection, and generic collections. As a nod to the past, students could be required to implement a few generic collections using heaps, linked lists, trees, etc. Some comparison of the efficiencies of these different implementations could also be mentioned. The traditional CS146 topics could be moved back to CS155. Finally, CS151 could be devoted to UML and design patterns.

Recommendation 2

As an alternative, require students to take a placement exam before entering CS46A. The exam would test problem solving skills. Students who fall below a certain score on the exam would be required to take a preliminary course similar to the modified CS46A described above.

Recommendation 3

The rubrics need to be given to the instructors months, not days in advance. Hopefully this problem will disappear once all of the rubrics are written. I anticipate that the rubrics for Spring assessment will be finished by mid-March.

Recommendation 4

Some of the tasks need to be re-written. Specifically, we should reconsider the inner class task for the CS151 rubric. There are probably too many similar tasks for the CS46B rubric.