The Writing Skills Test, 1993-97: 30,000 First Time Test Takers
A Summary Report to the SJSU Community

Lee Dorosz, Acting Associate Vice President
Office of Undergraduate Studies, July, 1999

          In Summer, 1998, SJSU’s Writing Requirements Committee under the leadership of Lela Noble, Chair of the Committee and Dean of Social Sciences, asked the Provost to commission a study of the Writing Skills Test (WST). The authors of that study, who provided all the data on which this summary is based, are the recently retired Director of Testing, Dr. Mara Southern, and a consultant, Dr. Roulette Wm. Smith. Their complete report is available in the Office of Undergraduate Studies.

Across the nation faculty, employers, and the general public express concern about the communication skills of college students, especially their writing. In fact this concern may not have a sound statistical foundation if one takes into account the increased percent of high school graduates who now attend college. From a practical rather than scholarly perspective, however, the very perception that the problem is severe and getting worse means that corrective attempts have to be made. And there is no question that anecdotal evidence of poor writing is both overwhelming and sad.

          Many of the solutions put forth to address this problem require changes in elementary and high schools that could take years, even decades, before significant impact is felt - if then. Even solutions that would affect only the colleges themselves often include admission and advancement criteria whose debate, adoption and implementation could also take years, especially in a multi-campus system such as the California State University.

          Meanwhile, thousands of students each year enroll, progress, and graduate (or fail to do so) while long term solutions are explored. Throughout California, the groups that are having the most severe difficulties are increasing in size. It seems that no one is satisfied with the average writing ability of today’s baccalaureate graduate. Dissatisfaction with the poorest upper-division students and graduates is intense. This summary report is only one snapshot across a continuum. It is intended to be of interest and use to those trying to teach today’s students, as well as those planning for the future. The intent is problem definition as one step towards resolution.

          SJSU faculty often ask for more stringent screening measures to prevent students from enrolling in Advanced General Education and other upper-division courses until they have mastered the fundamentals of writing. It seems but a simple step to raise the bar and make the passing index on the screening test higher. Yet the data in this paper show that a simple increase in the passing index would have powerfully differential effects on ethnic and language groups and thus on the retention and graduation rates of programs with large Asian student enrollments, especially engineering, science and business. This summary provides background against which this and other possible courses of action can be considered.

Background

          Every California State University (CSU) must enforce a Graduation Writing Assessment Requirement (GWAR) that includes at least one common test taken by all undergraduates. CSU campuses vary markedly in how or even if course performance is tied to the GWAR - on some campuses there is only a test while on others a course (e.g., SJSU’s 100W) is required.

          CSU campuses also differ widely in how they decide that freshmen who are required to take remedial writing are judged ready to advance to baccalaureate-level composition. At SJSU, passage of a holistically-graded common essay exam is required; on many CSU campuses, instructor judgment, as reflected in course grade, either at the CSU or at a Community College, is the sole index that the need for remediation has been satisfied. The issue of instructor judgment vs. performance on a common examination is one of the problematic themes that pervades the debate over how student writing should be evaluated at all levels. In general, SJSU composition faculty are and have been strongly in favor of holistically graded common examinations as the basis for progress at all levels, although the final graduation-level assessment of student writing, the 100W courses, are evaluated only by individual instructors. There was at one time a common holistically graded essay as part of the 100W final grade, but it was discontinued, largely because faculty across Colleges could not agree on what constituted an appropriate essay topic.

          At SJSU, once students satisfy lower-division composition (Engl 1A/1B or equivalents) they must pass a Writing Skills Test (WST) to enroll in a junior-level writing course, 100W, passage of which is required of all students to satisfy the GWAR. (Departments may allow a high score on the WST to waive the 100W course requirement.) The WST consists of a standardized American College Testing Service (ACT) multiple choice section, and a locally developed holistically read essay. Passing is defined by combination of multiple-choice and essay scores. There are three outcomes on the WST: Waive, Pass, Fail.

          The WST has been used at SJSU since the mid-1980’s. On several occasions the index was changed because instructors complained of the number of students passing the WST who had poor command of composition fundamentals. The percentage of students who were screened out increased sufficiently to allay some concern, yet the call persists for still another increase in the index. The WST is critical to students because they cannot enroll either in 100W or in the required nine units of Advanced General Education until they pass the WST. Enrollment in other upper-division courses is normally not affected by WST status, but some departments are considering introducing passage of WST as a prerequisite to all upper-division courses in the major.

The Current Study

Study Population: Unlike previous analyses of the WST, the current report summarized data only for first-time test takers. Those who fail the WST may repeat the test as often as they like, with the exception that before taking it a third or subsequent time students must first have earned Credit (satisfactory participation) in LLD 98/99 (see later discussion; students may earn Cr, yet not pass the WST-equivalent common exam). The mix of repeat and first-time test takers confounded previous analyses of WST performance. About 20% of takers at any given WST administration are repeaters, and repeat takers have a much higher probability of failure compared to first-time takers.

          The number of first-time takers in analyses reported here is 30,523, in 25 test administrations over a five-year period. Up to 400 of that total were discarded from one analysis or another for data-entry problems of various kinds.

Negative Findings

• Neither time of year for the test, nor gender, turned out to be important in understanding outcomes, so neither are mentioned further in this summary. Females in all categories do better than males, but differences are typically only 1 to 3%. Sub-analyses that might show an interaction between gender and ethnicity have not been conducted.

• While there were differences among majors in different SJSU Colleges, these are largely a result of the fact that WST success is strongly related to language background, and students from those Asian language groups that have the most WST difficulty are clustered in certain programs.

• Class level at the time of taking the test does not contribute to variation on test performance. Probably the best conclusion to be drawn is that only unusually well prepared lower-division students would even try the WST, thus elevating the statistical success rate of that population.

Critical Issues to Consider in the Analyses

• Language Background. By self-description, students classify themselves into one of three language groups:

ENL	-	English is my Native Language, and English is also my Primary Language
		(ENL = English Native Language, 58% of test population)
EPL	-	English was not my native language, but English is now my primary       language
		(EPL = English Primary Language, 16%)
OPL	-	English was not my native language, and a language Other than English is        still my primary language
		(OPL = Other Primary Language, 26%)

• School of Most Recent Instruction. At the time of testing students are asked to report where they completed their most recent course in composition - at SJSU, at a Community College, or at another four-year college. A student who has most recently taken a composition course at DeAnza is classified in most analyses as a "DeAnza Student," no matter the school of initial enrollment.

          Complication: During the analyses it became apparent that an important number of students who entered SJSU as freshmen chose to take their lower-division composition at a Community College, and thus reported that their most recent composition course was not at SJSU. These students then become part of that other school’s population for this report. Of just over 7,000 native SJSU freshmen during this five-year period, 2,000 reported that their most recent composition course was taken at a school other than SJSU. A given school, for instance, might have a reputation for offering easily passed GE composition courses because of its high proportion of non-English speakers, and might thus be attractive to students who anticipate problems. Yet, that school’s population might perform more poorly on the WST, thus perhaps disadvantaging the SJSU native freshmen who chose to do their composition there instead of at SJSU. Southern and Smith suggest that the number of SJSU frosh who take this path is small enough at any given school that their numbers do not influence significantly the individual school data. This remains an issue for analysis.

          Almost all agree that good writing is an outcome of teaching across the curriculum, not just in composition courses. It is at this time impossible to tease out how movement among campuses influences test success outcome for a given school. Even if they take their composition at SJSU, many native SJSU frosh nonetheless take some of their lower-division work at nearby community colleges, and, reciprocally, a few students who do most of their work at a community college take composition at SJSU. Pinning down absolute cause-and-effect relations in such a fluid environment is virtually impossible.

Result Highlights

Language Status: Certainly the most striking outcome of the study is the degree to which a student’s current primary language influences WST success (Tables 1 and 2). This outcome is not surprising, but it helps explain variation among schools and raises questions about the usefulness of a simple pass/fail, non-diagnostic, placement test in addressing language needs of an increasingly heterogeneous student population. Unfortunately, there are few, if any, widely accepted writing tests that are well validated, reasonable in cost, and sufficiently diagnostic to be of real use in remediating students who cannot pass.

Table 1. Language Status: Number

Table 2. Language Status: Percent

         

In sum, only 1 in 20, about 5%, of native-English-speakers (ENL) fail the WST. 18% of EPL students fail, while more than half, over 52%, of OPL students fail on their first attempt. Non-native speakers who by self-report now use English as their primary language still fail the test at a rate over three times higher than native speakers, yet those who are not yet using English as their primary language (OPL) fail at a rate three times higher than the EPL students, and a remarkable ten times higher than native speakers.

          One of the challenges facing the campus and its feeder Community Colleges is to convey these data to students in a manner that helps them to realize what faculty have long known but not been able to document, namely, that for academic success it is important to use English as much as possible with friends and at home, and not just in the classroom. Until English becomes their self-identified "primary" language, students fail the WST in huge numbers.

          As the Writing Requirements Committee responds to calls for higher WST passing scores, one of the dilemmas it faces is that any action taken to "raise the bar" will inflict disproportionately higher failure rates on EPL and especially on OPL students, thus reducing retention and graduation rates of programs in the sciences, engineering, and business, where there are large proportions of OPL students. The impact will be especially heavy on the large populations of Vietnamese and Chinese students at SJSU, and would have dramatic impact on all language groups except native speakers of English (Tables 3 and 4).

          The failure rate among native English speakers (ENL) is high for African Americans (13.2%), a result consistent with other testing data and an issue that demands continuing attention. By contrast, only 1.5% of Caucasian native English speakers who entered SJSU as freshmen fail the WST. As the next section will show, this percent is somewhat higher for Caucasian native English speakers whose last composition course was at a two year college, however.

          Among OPL students - non-native speakers who still do not use English as their primary language - failure rates are very high for Asian students and for Black students (this OPL group of non-English-native Black students consists mostly of African students, whose total number is quite small).

Location of Last Composition Course:

          About two out of three first-time WST takers are transfer students, the majority from local community colleges. A similar percentage of degrees is awarded to students who did not begin at SJSU as freshmen. (SJSU’s transfer rates are typical for urban CSU campuses). The transfer population is important simply in terms of numbers. The academic performance of transfer students is important in upper-division courses at SJSU because instructors assume a certain homogeneity in background preparation. In turn, community college instructors, recognizing that they teach a high proportion of students who were not eligible to attend the CSU after high school, want feedback to assure them that the preparation they offer matches CSU expectations. Three local community college districts, six colleges in all out of the ten in SJSU’s service area, are represented on SJSU’s Writing Requirements Committee.

          Unfortunately, the data in the Southern-Smith report are not encouraging for community colleges. It is clear that students who take their recent composition at a community college fail the WST at a consistently higher rate across virtually all language and ethnic categories than those who take composition at a four year college, SJSU or other (Tables 5 and 6). Yet the reasons for this difference may have more to do with the nature of the student population than with the efficacy of instruction. Self selection - which students choose to attend which colleges for what facets of their education - is, not just here but across all educational assessments, perhaps the single most difficult variable to tease out in attempts to make effectiveness comparisons among a variety of educational strategies and techniques.

Table 5. Failure rates, by language group, summed across almost all of the two year colleges
where the most recent composition course was attempted

          Table 5 highlights the problem of the Asian speaking population rather dramatically. Over 65% of those Asian students who still do not claim English as their primary language, and even 30% of those who do claim English as primary language, fail the WST on first attempt. This figure is notably higher than for those who entered SJSU as freshmen (it is also a figure confounded a little by the addition of those who, while they entered SJSU as freshmen, nonetheless went elsewhere for their most recent composition work.) Table 5 also calls attention to the problem of the native African American student who does composition work at a Community College -- 21% fail, as contrasted with lower numbers in all other ethnic groups of self-defined native English speakers.

          Table 6 provides a summary picture of how strongly the percentage of OPL students in a test population influences the WST success rate. Failure rates ranged from a low of 10 to 12% (SJSU, Gavilan, Cabrillo, other four year colleges), to a high of 41% (Mission College). The statistical correlation between the percent of OPL students in a school’s test taking population and the probability of failure is 0.93. This means that language status accounts for over 80% of the variation among schools in their students’ success on SJSU’s WST. Thus, the schools with the highest failure rates (Mission, San Jose City, and Evergreen) had OPL populations ranging from 38% to 49%, as contrasted with San Jose State’s and "other four year colleges" OPL proportion of just under 19%.

Table 6. Institution of Most Recent Composition Course,
ranked by pct of OPL students in the test population:

Further Discussion - Frosh who leave SJSU to do composition elsewhere

          The success and failure of students who started as freshmen at SJSU, and then took composition elsewhere, sheds an interesting light on the writing instruction problem, and also raises as yet unanswered questions. Twice as many OPL freshmen (45%) leave SJSU for their composition as ENL freshmen (22%).

          Among ENL students - native speakers of English - SJSU’s failure rate on the WST was only 2.5% whereas San Jose City’s failure rate was 12.8%. (Table 7). This would be consistent with, but not conclusive evidence of, the interpretation that SJSU students who expected to have difficulty in their composition courses, whether they were native speakers or non-native speakers, might choose San Jose City because it is known to have a high proportion of non-native speakers, and of instructors who are thus sensitive to students with language difficulties. One could hypothesize further into that interpretation and suggest that perhaps the inordinately high percentage of students with language difficulties in this school might in fact tend to reduce the expectations of composition instructors - who perhaps only rarely see an excellent young writer.

          From the data available it is impossible to tease out how much of the difference among districts is attributable to student preparedness and motivation, and how much attributable to efficacy of composition programs. The clear fact is that students whose recent composition work is at a two year college, as opposed to a four year college, fare poorly on the Writing Skills Test, no matter their language background. Because SAT and ACT data are usually not available on community college students, it would be difficult to get an agreed-upon measure of preparedness by which the transfer population might be compared with the population of native freshmen at SJSU. Without such a measure, it is impossible to draw conclusions about the effectiveness of instruction.

Table 7. Failure of 7095 SJSU native freshmen by school of last composition course: