BSCS Assessment Report Spring 2009

 

Assessment Activities

Here is a copy of the Outcomes/Courses matrix that was in use during the Spring 2009 semester (some irrelevant columns have been eliminated):

Entries:

B/I/A = corresponding course supports corresponding outcome at the beginning/intermediate/advanced level of competency.

Relevant outcomes:

SLO3 = The ability to select, design, and implement appropriate data structures and algorithms.

SLO4 = The ability to design computing systems that are appropriate for commonly used hardware.

SLO5 = The ability to operate commonly used tools for software development, testing, and management.

SLO6 = The ability to design and implement graphical user interfaces.

SLO7 = The ability to design computing systems that are appropriate for commonly used operating system.

SLO8 = The ability to carry out object oriented design and to apply design patterns.

SLO10 = The ability to apply theoretical foundations of computer science to reason about performance and limitations of computing systems.

SLO13 = The ability to solve computing problems as part of a team.

 

1. Modifying SLO4 and SLO7

 

The statements of these outcomes are unclear. In [2] Pearce interprets the first as having something to do with digital logic and the second as having something to do with interrupt handlers:

I wasn't sure how to interpret either outcome. I suppose the second outcome might have been covered while discussing the interrupt structure of a CPU, which I did. It could have been measured by asking students to write a few interrupt handlers in assembly language. However, my students learned inline assembly and JVM assembly. Neither allows system calls.

I interpreted the first outcome as the ability to design common computing components out of logic gates. As it happened, I did cover digital logic design, even though this is not part of the official course description.

In [1] Araya simply equates SLO4 with overall course performance:

Achievement for outcome 4I was determined by the overall grade the students received in the course.

 

1.1. Actions

At the request of the Assessment Committee and the Systems and Architecture Committee, the Department voted to change the wording of these outcomes:

SLO4 = The ability to explain the structure and function of computer hardware components and their influence on the design of software systems.

SLO7 = The ability to explain the structure and function of operating systems, their interactions with computer hardware components, and their influence on the design of software systems.

2. Modifying the Assessment Process

 

The Assessment Committee is like the blind man who deduces that elephants are shaped like ropes after feeling just the tail. Each semester the assessment process generates reports on a near-random selection of entries in the Outcomes/Courses matrix and the Committee is supposed to determine if the Department is meeting its objectives or not. In addition, the Committee is asked to compare the elephant to the hippo whose tail it felt two years earlier, and is asked: which animal is better?

As a case in point, consider Araya's report on SLO10B. He says that 50% of his students achieved the desired level of competency (25% partially achieved it.) In Spring 07, the last time this outcome was assessed in CS149, Teng Moh reported that 75% of his students achieved the desired level of competency (19% partially achieved it.)

How does one interpret this huge drop and is it a cause for alarm? According to the matrix, this outcome is supported by six courses. We only have current data on one. So is the elephant like a rope or a tree? Moreover, Araya measured this competency by asking students to come up with a mathematical equation modeling the effect of page faults on memory access time (which sounds sort of advanced). We don't know how T. Moh tested this competency, so which is better: elephants or hippos?

It's the same story in all of the other regularly scheduled reports the Committee received.

2.1. Actions

 

In order to correct this state of affairs, the Committee decided that each entry in the Outcomes/Courses matrix should be associated with a fixed list of tasks. This has already been done for SLO5 and SLO2. Establishing these tasks should be done by the appropriate committees during the Spring 2010 semester. For example, tasks for the SLO 10B/CS 149 entry might ask about resource management models.

The assessment schedule was also changed so that each semester entire rows in the Outcomes/Courses matrix are assessed. This will give us a complete picture of how the department is doing on a particular outcome.

It is envisioned that all of the data gathered from the assessment process will be entered into a spread sheet. This will make it easy to spot trends spanning several years. For example, this is what the page for the assessment of SLO1 might look like:

The only question that remains is to determine acceptable minimum success levels. In the example above, 77% of the CS151 students successfully performed task 2. Is that too few? It's likely that we won't be able to answer these questions until several semesters worth of data are gathered.

3. The SLO5 Success Story

 

In response to a faculty complaint about his students' inability to manage large code libraries, Horstmann (who was the acting chair of the Assessment Committee) decided to initiate a special assessment of SLO5 using specific tasks (see [5]). He identified four tasks:

Task 1: Compiling and running multi-class programs

Task 2: Running the debugger

Task 3: Generating and browsing javadocs

Task 4: Testing a class

Horstmann designed two tests, one that tested students' abilities to perform these tasks at the beginning level (test 1) and another that tested students' abilities to perform these tasks at the intermediate level (test 2). He gave test 1 to CS151 students at the beginning of the Spring 2009 semester, and to CS46B students at the end of the semester. (CS46B is a prerequisite for CS151, although many students don't take CS151 immediately after CS46B.) He gave test 2 to CS160 students at the beginning of the semester and to CS151 students at the end of the semester. (CS151 is a prerequisite for CS160.) He then observed that, among other things, CS151 students did well on test 1 but poorly on test 2, indicating that CS151 was doing an inadequate job teaching students how to use tools at an intermediate level.

3.1. Analysis

 

The biggest lesson from Horstmann's experiment is that assessment needs to be task-based. Indeed, this was the motivation for creating the task-based rubric for SLO2 that was used in Fall 09 and going forward, all rubrics will be task-based. The only question that remains is how specific the tasks should be. To preserve some flexibility for instructors it is envisioned that tasks will be somewhat abstract. For example:

Write a Java program that modifies an array, for example, a program that sorts an array.

In the future we might want to turn this into a specific array-sorting problem and request faculty to put it on their final exams.

By giving pre-semester tests and post-semester tests Horstmann was able to pinpoint CS151 as the problem class. However, under the modified assessment schedule using task-based rubrics, the problems in CS151 would still be noticed if, for example, students emerging from CS46B did well on SLO5-B tasks, but students emerging from CS151 did poorly on SLO5-I tasks.

So how should the department address the issue of tool usage? Horstmann's report [6] suggests that the CS46A/B labs should be re-designed. This is in the process of being done. He also suggests that the CS151 syllabus should be re-designed. This has not been done yet. The Assessment Committee will request that the Programming and Software Engineering course committee (the committee responsible for the CS151 syllabus) begin this task in the near future.

Another solution to the problem might be for the department to fix a standard set of tools in a manner similar to the way a standard language is fixed. For example, fixing Eclipse as the department IDE, Trac as a standard issue tracking tool, and Subversion as a standard version control tool means that CS151 instructors could assume entering students were familiar with these particular tools at a beginning level. Support for such tools would have to be ensured by the Department as well.

3.2. Mining the CS151 Report

 

Kim submitted a thorough report on how many of her CS151 students mastered outcomes SLO6-I, SLO8-I, and SLO13-I. Apparently the corresponding report from two years earlier was "unusable" according to the Assessment Committee's notes. It would therefore be nice if some of this conscientiously written report could serve as a baseline for future task-based assessments of these outcomes.

Kim associates a list of tasks with each outcome. Most of these tasks seemed reasonable and the Assessment Committee might want to consider adopting some of them for future task-based assessments of these outcomes. Kim then cites several problems (homework assignments, exam questions, projects) that tested her students' abilities to perform these tasks.

For example, for SLO8-I (The ability to carry out object oriented design and to apply design patterns) Kim defines three tasks:

T1: Represent the design of a system or component with a UML class diagram.

T2: Translate a UML class diagram into Java class declarations.

T3: Employ at least three different design patterns. For example, three instances where students were required to recognize the applicability of a particular design pattern, then correctly instantiate that pattern either in the form of a UML class diagram or code or both.

She then defines three problems, although the second problem has three parts, so really there are five problems. Each problem tests a student's ability to perform one or more of the tasks. The following table shows her results:

While the results for tasks T1 and T2 appear to be encouraging, it is difficult to say if students mastered task T3 or not. How should this data be interpreted?

She defines a single task for SLO6-I (The ability to design and implement graphical user interfaces)

T1: Design and implement a GUI application that involves with use of multiple GUI components, layout management, and event handling.

She then cites four problems where this task was tested and reports mixed results:

Again, it's difficult to interpret these results. The problems seem very easy, almost at the CS46A level, so these numbers could be too low.

Finally, Kim defines two tasks for SLO13-I (The ability to solve computing problems as part of a team):

T1: Conduct a team-based project to design and implement an application that involves with about 10 classes, GUI programming, and design patterns.

T2: Employ a mechanism that allows the instructor to judge load balancing and degree of participation among team members.

Kim only cites the team project as the basis for assessing presumably both of these tasks. She reports that 58.3% of her students mastered these tasks. (25% partially mastered it.)

Task T2 seems a little out of place. It sounds more like a task for instructors (and a difficult one at that). If only 58.3% of her students mastered task T1, then his would be worrisome. Of course we don't have anything to compare this data with, the figures do include students who presumably didn't pass the class (the new rubrics will only take into account percentages of students who received a C- or better in the class), and CS151 is the first place in the curriculum where team projects are required.