S98-4 Revision of the SOTE and SOLATE Interpretation Guide

Legislative History:

At its meeting of March 2, 1998, the Academic Senate passed the following Policy Recommendation presented by Nancy Stork for the Instruction and Student Affairs Committee.
Rescinds: S89-6
Action by the University President:
"Accepted as University Policy." Signed by Robert Caret, 3-28-98

Policy Recommendation Revision of the SOTE and SOLATE Interpretation Guide

Whereas,      ISA-F95-2, "Revision of the SOTE and SOLATE Interpretation Guide" first came
                   before the Academic Senate in 1995 in response to many changes that had been
                   made to the SOTE and SOLATE forms and,

Whereas, It has, since then, been read and reviewed by both the full Senate and the
Professional Standards Committee, and

Whereas,      The Instruction and Student Affairs Committee has consulted with Professor Seth
                   Bates, Chair of the Student Evaluation Review Board and completed a review of
                   the changes made at the suggestion of the Senate and Professional Standards
                   Committee; therefore, be it

Resolved, That the following version of the "SOTE and SOLATE Interpretation Guide" be
adopted as University policy.

INTERPRETATION GUIDE FOR SJSU SOTE AND SOLATE SURVEYS Prepared by The SJSU Student Evaluation Review Board (September 1997) Revised by the Instruction and Student Affairs Committee (January 1998) This guide should be used by all SJSU faculty, all Department/School, College, and University Retention, Tenure, and Promotion (RTP) committee members, and all others who make judgments based on data from Student Opinion of Teaching Effectiveness (SOTE) and Student Opinion of Laboratory and Activity Teaching Effectiveness (SOLATE) surveys. Information from SOTE and SOLATE surveys are but one source of information for assessing teaching effectiveness (see Senate Policies S91-9 and S94-6). Other sources of information about teaching effectiveness should be employed before reaching an RTP decision. Responsibilities of Candidate Faculty

It is the responsibility of individual faculty members and their colleagues to ensure that other sources of teaching effectiveness, e.g. peer evaluations, departmental or individual instructor's course evaluations, and student letters, are collected and included in personnel dossiers (in accordance with University policy).

If an item mean is consistently above or below the norm range, the faculty member should provide further information about that rating.

It is the responsibility of the faculty member to ensure that information about factors that influence student opinion ratings be included in the dossier, along with the ratings. For example, the faculty member should note whether the class was composed primarily of people required to take the course. (See discussion below: Factors to be Considered When Interpreting the Ratings.)

Statistical Display

The upper half of the first page of SOTE and SOLATE reports provide shortened versions of each of the 14 questions. To the right are four double columns of numerical data. The first double column lists means and standard deviations for the class in which opinions were gathered. The second, third, and fourth double columns list the norming data collected in F89/S90 aggregated across the Department/School, the College, and the University. The lower half of page 1 displays frequencies of responses for the individual items from which the above means and standard deviations are calculated. Also included are self-reported students' expected grades, self-reported class level, and actual average class GPA for the class at the time the SOTE/SOLATE report was generated. Page 2 is a graphical display of class means superimposed upon College and University rating "norms."

The Statistics

The mean is the arithmetic average of student responses in which a score of 5 is assigned to the rating of "excellent," 4 to "above average," 3 to "average," 2 to "below average," and 1 to the rating of "far below average." It is important to remember these descriptors when interpreting ratings. Means are reported to the first decimal place.

Interpretation. The extent of agreement or disagreement on an item can be seen directly from the frequency distribution for that item displayed at the bottom of the page. A less sensitive gauge of agreement is provided by the standard deviation. Most standard deviations are very close to 1.0. A large standard deviation (e.g., 1.3) indicates that students often do not agree about what rating should be assigned. A small standard deviation (e.g., 0.7) indicates that students generally agree about what rating should be assigned.

Ranges of Typical Values ("Norm Data")

"Norms" for each item are provided at the Department/School, College, and University levels. At each level, responses are aggregated over a specified norming period (most recently, F89/S90 for SOTE) to compute means and standard deviations which serve as reference points for making comparisons. Comparisons between the class data and norm data are best made using the graphic display shown on page 2 of the report.

Ranges of typical values ("norm ranges") for the College and University levels are graphically displayed on page 2. For each item, the middle 60% of ratings, from the 20th to the 80th percentiles, was determined for all classes surveyed during the norming period. This range is displayed as a line of dashes. The class mean is printed as an asterisk on the same line.

Interpretation. If the asterisk is printed within the line of dashes, the class mean should be interpreted as no different from the norm group. If the class mean clearly falls outside the line of dashes, it can be concluded that the rating was below (to the left of the dashes) or above (to the right of dashes) that of typical scores. The usefulness and validity of a rating will be degraded if ratings within the norm area are interpreted as anything other than typical. It is also important to remember the initial response descriptors when interpreting ratings (e.g., a score of "4" indicates "above average"). The mean score of most items is approximately "4."

SOTE and SOLATE interpretation should use data across classes and semesters.

Factors to be Considered When Interpreting the Ratings

Many factors are known, through statistical research, to influence student opinion ratings. Therefore, ratings should always be interpreted with caution. Several, but by no means all, of the factors which have been shown to be consistently related to ratings are listed below.

1. On the whole, research suggests that ratings are highly correlated to expected grades. Therefore two items are provided; both a distribution of the students' self-reported expected grades and the actual class GPA given by the instructor are to be found in SOTE and SOLATE report printouts on the bottom of page 1. A distribution of the actual class grades given can also be routinely added to the printout by candidate faculty.

2. Ratings in small classes tend to be higher than in large classes.

3. Ratings in graduate classes tend to be higher than in undergraduate classes, and ratings in upper division classes tend to be higher than in lower-division classes. Self-reported class level is reported on the bottom of page 1.

4. Ratings given by students who are required to take a class are often lower than ratings by students for whom the class is an elective.

5. When a significant number of students in class leave an item blank or mark it "not applicable," that rating should be interpreted with caution. The number of students indicating these responses is reported in the frequency distribution on the bottom of page 1.

6. Ratings from team-taught courses should be cautiously interpreted as students may be unable to separate their experiences from one instructor to the next.