|
Abstract
|
IV. III. FINDINGS AND OUTCOMES A. Survey Results(See Appendix A for questionnaire) Key results are summarized below, followed in most cases by brief interpretive comments. Percentages have been rounded up. 1. Who responded to the questionnaire? A total of 118 SJSU faculty members responded, or approximately 7.5% return rate (based on 1573 total addressees). * 65% of the respondents were female, and 35% were male. * 47% were monolingual English speakers. * The majority of respondents were in the Colleges of Humanities and Arts (27%), Social Sciences (20%) and Applied Sciences and Arts (20%).** Although all of the other colleges were represented in the survey as well, each made up only 10% or less of the total. * 56% of the respondents teach primarily undergraduates. * 63% of the respondents teach classes of 21-40 students.* However, 14% teach classes with over 41 students. * 42% of the respondents identified themselves as lecturers, while 50% identified themselves as full time faculty members. 2. Frequency of responding to student writing: The most frequent response by far was, "1-4 times a week," with 55% of respondents choosing this frequency. If we add those who say they respond to student writing every day (i.e., 5 days a week), we get 74%. The author hypothesized that the frequency of response might go up as people teach more classes, but the data do not show evidence of this. Interestingly, a test of the correlation between frequency of responding to student writing and the number of classes respondents teach per semester yields little clarity. There is only a low positive relationship (.206) between the frequency of responding to student writing and the number of classes a person teaches. Comment: A possible explanation for the low correlation is that many of the people who responded to the survey were lecturers who teach part time. Lecturers are often the ones hired to teach writing intensive classes such as 100W and the academic writing classes such as LLD 1 and 2, and LLD 98 and 99. These instructors may see their primary role as teaching writing as opposed to other content. If this is so, then even though they teach fewer classes, they might be likely to report high frequencies of responding to student writing. 3. Amount of time spent on responding to student writing: It is possible that someone might respond frequently but not spend a lot of time responding, or the reverse. Therefore this item sought to discover respondentsents to estimate "ow much time they spend responding to student writing. Most respondents (64%) reported spending more than 20 hours a month in this activity (see Table 1). Table 1: Amount of time respondents report spending on responding to student writing
Comment: Along similar lines to the comment on the previous item, the lack of association between number of classes taught and amount of time spent responding to student writing may be explained by the fact that lecturers, who are often part time and who often teach writing intensive classes, may spend more time responding to student writing for each class they teach, even though they do not teach as many classes as full time faculty. 4. Commenting on student papers: 84% of respondents say they always write comments on student papers. However, respondents are quite divided in how effective they think these comments are, as shown in Table 2: Table 2: Respondents' perception of the effectiveness of comments on student papers
What we do not know from the survey is any information about the type of comments faculty write on student papers. As Dana Ferris pointed out, a wide range of comment types exist. The data from the survey suggest that we need to "unpack" the category we are calling "comments" to find out how differentt types of comments function. This might explain why we see so much variation in responses about the effectiveness of comments; respondents may be answering based on widely varying types of comments. It is difficult to capture this type of data. To do so, we need more qualitative, in-depth study in which we look at particular comment types and how students interpret and use them. 5. Correcting all grammar or sentence structure errors: 49% of respondents often or always correct all grammar or sentence structure errors. However, 76% of respondents think that correcting all grammar or sentence structure errors is only somewhat or not effective. Comment: If, based on what we know from Ferris (2003), it is true that correcting all grammar or sentence structure errors is not an effective practice, then this is a gap between research and practice that needs to be addressed through professional development. Faculty members should be familiarized with what the research says so that they can make more fully informed decisions about their practice. 6. Writing conferences: A majority of 64% of respondents sometimes, often, or always hold individual writing conferences with students, and 67% believe that these conferences are effective in helping students to improve their writing. However, 34% rarely or never hold writing conferences, 19% believe they are only somewhat or not very effective, and 14% say they are unable to evaluate the effectiveness of writing conferences. Comments: The literature says that 1-1 writing conferences are one of the most effective methods of responding to student writing. Yet 34% of respondents say they use this practice infrequently or not at all. Responses to the open-ended questions suggest that the variation may have to do with institutional constraints -- e.g., large class sizes, (64% of respondents teach classes with 21-40 students; 13% teach classes with 41 or more students); # of classes we are expected to teach. These two constraints lead to less time available for 1-1 writing conferences. Even if we think it is effective, we may not be able to do it. This is an area where, like the "comment on student writing" category, it would be useful to unpack the concept more carefully. Different people do different things in a "writing conference." 7. Peer response groups or pairs: Responses to the two items related to peer response (or peer review) showed no clear trend. There was wide variation among respondents in both how frequently they use this practice and how effective they think it is (see Tables 3 & 4): Table 3: Frequency of using peer response groups
Table 4: Perceived effectiveness of peer response groups
Comment: Like the other categories mentioned above (writing comments and holding 1-1 conferences), it would be useful to unpack what faculty are actually doing in these peer response sessions to be able to better understand under what conditions they are/are not effective. A comment on the open-ended part of the questionnaire sheds some light on why some faculty members are dissatisfied with peer response groups: "Peer responses tend to be far more superficial than I think appropriate. They tend to focus only on grammar, spelling, typos, not on the content or organization at a deep level." Pehaps, it would be valuable for faculty members to develop strategies to "deepen" peer response. 8. Sample of questions or concerns respondents had about responding to student writing: The following comments were given in response to the open-ended questions on the questionnaire. Open-ended questions were asked at the end of each major section of the questionnaire. * Institutional structure constrains faculty's ability to support student writing. Reduce class size and teaching load. * More info on what research tells us about effectiveness of various approaches. * How to get students to understand, read, process, and use your comments to improve future writing assignment. * Many professors don't do anything with usage or style, only grade on content. Comment: This statement appears to reflect a concern in the sense that unless all of us take some responsibility for teaching writing, students will not get a clear message re how important this is. But this is countered by a respondent who says he or she does not want spend time teaching students how to write; that's the job of the English 1A and 1B instructors. So there is a clear tension here regarding whose responsibility is should be to respond to students' writing, and how much of the responsibility should be shared by non-writing instructors. * Need for more campus-wide support services for writing - e.g. tutoring, writing workshops etc. * More information on checklists. * Guidelines or benchmarks for student writing competencies after 100W (and other gatekeeper courses) * Time consuming - would like to be both more efficient and more effective in giving feedback * How to work with non-native English speaking students without correcting every error. * Screening procedures that allow students to take a course when they are not really ready for it? The limitations of the survey: One limitation is that we had a very low response rate of around 7.5% (118 faculty members). Thus, the results cannot be generalized to the whole faculty of SJSU, only to those who chose to respond. It is possible that those who responded did so because they have a particular interest in teaching writing. These descriptive statistics point to some interesting findings regarding central tendencies (or lack of central tendencies). They are able to show us some general frequencies and perceptions of the effectiveness of certain practices, but they do not explain why or even how respondents understand a particular item. To answer the why and how questions, we need to do more in-depth, qualitative studies. But the descriptive statistics do point us in the direction of some questions we might want to explore further. B. Evaluation of the Symposium Highlights of the evaluation are presented here. For a more detailed evaluation summary, please refer to Appendix C. Of the 85 people who attended the Symposium, 52 turned in evaluation forms. They were very enthusiastic about the event. Dana Ferris' keynote presentation received especially high ratings; 97.8% of respondents said her presentation was valuable, and 100% said would use some of her suggestions in responding to student writing. 95% of the respondents found the faculty discussion groups valuable, and 91.6% found the student panel valuable. When asked, "What was the most valuable thing you have learned today" the most frequent comment participants gave was that the symposium experience led them to change, refine, or question their own feedback strategies. Specific strategies they mentioned include * being specific and clear when giving written feedback; * asking students to write up responses to teacher feedback; * following up to find out how students use the feedback they receive; * giving indirect feedback (i.e. not correcting but pointing out errors or problems and expecting students to seek the best way to correct or revise); * prioritizing the type of feedback to give (i.e., content or grammar); * using rubrics and checklists to make expectations clear; distributing these when the assignment is given so that students can know how the assignment will be assessed; * making instructions for peer review clear; * saying something positive about students' writing; and * creating opportunities for 1-1 conferences. Respondents also made some suggestions for follow-up. One of the most important was the need to offer more professional development specifically targeted for disciplinary content areas. In other words, participants noticed that those whose job it is to teach writing operate in a different context than subject area instructors, who teach writing in the context of other required content. Another need that surfaced in the evaluation was that faculty wanted more follow up on the peer review process - i.e. whether it is or can be effective, and if so, how to design peer review in such a way that it really helps students improve their writing. Both of these expressed needs informed the decision to focus the next phase of the project on faculty initiated professional development in departments and on small classroom research projects. C. Analysis Of The Faculty Project Reports (Phase 2) The analysis that follows attempts to capture cross cutting themes that characterized more than one of the faculty projects. The first two points provide an overview of the projects and a sense of how many people were affected by them - a rough quantitative measure of "impact". Following that, themes that emerged in multiple projects are articulated. 1. Overview of the 8 faculty projects As noted in Section II, faculty members who applied to do projects in Fall 2005 had a choice as to whether they wanted to do professional development for their department colleagues (i.e. making the link between feedback and the discipline), or conduct a classroom research project instead, or a combination of both. Table 5 presents an overview of the faculty projects. For those wishing more information about the individual projects, Appendix E contains the abstracts written by each of the eight faculty grantees. A complete report may be requested by emailing the individual faculty member.
Table 5: Overview of faculty projects 2. Number of people affected across projects The eight faculty projects involved both other faculty and students. Other faculty members besides the eight LPP grantees were primarily involved by participating in professional development workshops. A total of 32 faculty members were involved in this way, through the projects led by Drabble, Fassett, Weinstein, and Ochoa. In addition, some faculty members were surveyed to learn about the forms of feedback they provide to their students (Ochoa). If we consider that the eight project leaders were probably the most deeply affected, then we can say that 40 faculty members were involved in and affected by these projects. Three hundred and fifty five (355) students were involved, primarily as research subjects. They were surveyed and in some cases they participated in focus groups (Drabble, Cava, McVey, Messina, Ochoa, and Schwab). In Ochoa's project, students in a 100W class played a more active role, constructing the survey instrument to give to students in other classes, and also administering the survey and analyzing it. Table 6: Number of faculty and students affected in addition to the 8 grantees
3. Collaboration across institutional boundaries
The "Responding to Student Writing" faculty projects encouraged collaboration across departments and institutional boundaries. At the very minimum, all faculty grantees met at least once with a coach from the LLD department. In addition, all but one grantee attended the final project meeting to share and discuss their projects with other grantees from different departments and colleges. In addition to these project-wide collaborations, Drabble co-led a workshop for faculty in her department with a faculty member from LLD, Betsy Gilliland. Drabble reported that this interdisciplinary leadership enhanced the quality of the workshop. Fassett held a workshop attended by the Chair of the Communication Studies Department, Graduate Teaching Assistants in her department, and community college faculty who teach communication courses. In this case, although all participants were in Communications, they represented different institutions and levels of hierarchy. 4. The faculty projects enhanced SJSU's institutional capacity to provide better writing instruction across disciplines.
The eight project leaders all engaged in substantial reflection, study, and preparation to be able to carry out their projects, whether they focused on professional development in the departments or on classroom research. As a result, they are now positioned to serve as mentors and coaches to other faculty members in the area of writing instruction, especially the issue of feedback. Some of the project leaders noted specific plans they have to do just that. Fassett reported that she plans to use and expand the prototype of training she developed in this project for GTAs in communication. She also plans to use the workshop prototype to develop future workshop formats (shorter as well as more advanced) for faculty in Communication Studies. Weinstein, who held two workshops for faculty members in Urban and Regional Planning, reported that faculty found these workshops very helpful. She may be called on in the future for similar professional development activities. Ochoa plans to offer a workshop for Social Science Department faculty in Spring 2006 in which she shares the results of her study. In addition to the human capacity developed in this project, faculty members also developed instructional and research tools that can be used or adapted by others. These are listed in Table 7. Table 7: Tools developed by faculty projects
5. The classroom research projects revealed both effective and ineffective feedback practices. In particular, faculty projects found that the following practices are helpful in motivating students to revise and improve their writing: (These practices are either based on students' self report or on comparisons of writing improvement using different methods of feedback). Feedback that is helpful * Individualized, detailed comments at the end of a paper (Ochoa, McVey) * Face to face meetings with instructor to discuss their writing (Ochoa, Drabble, Schwab). For example, Drabble found that as a result of her project, students wanted more peer review workshops; students also sought out the instructor during office hours more after the peer review session. * Handwritten comments (as opposed to typed) that make the feedback feel more personalized (McVey) * Peer review, when students are well prepared and session is well structured (Drabble, Cava, Schwab) * Practice session in which students practice giving feedback as a whole class before being expected to peer review. (Schwab) * Anonymous review by peers, because it ?greduced the sense of vulnerability and self-consciousness about their writing" (Cava) * Peer review of drafts from other classes (e.g., content classes rather than writing classes) (Cava) * Making the rubric available in advance provides guidance to those unsure of expectations, while allowing students to revise the proposed scheme gives students an automatic "buy-in" to their grades. (Messina, Schwab) * Pointing out patterns of error rather than correcting errors (Messina, Schwab) (However Messina qualifies this, noting that only students who are motivated make use of this type of feedback) * Appending tips for correcting errors that have been pointed out (Messina) * Discussing feedback as a whole group in class helps to learn from others' mistakes (Schwab) * Using red pen so students can easily see feedback (Schwab) * Being required to fix errors after a paper is returned (Schwab) * Reading exemplary student essays in class helped students to understand what was expected (Schwab) * Making positive comments as well as well as constructive criticism (Schwab) The classroom research projects found that the following practices are not helpful to students: Feedback that is not helpful * Instructor provides little or no feedback on their writing (Ochoa) * Penmanship is hard to read (Ochoa, McVey, Schwab) * Instructor uses red ink (Ochoa) * Instructor provides unclear or ambiguous comments (Ochoa) * Electronic feedback forms that are not easy to pull up via computer (a problem of articulation of different computer platforms? Only one student had a problem with this) (McVey) * Technology dependent tools (such as spell check and grammar check, or instructor corrections using the "Track changes" tool in Word) tend to be accepted and used without students necessarily learning from their mistakes. The same mistakes tend to be repeated in subsequent papers (Messina). * In peer review, the following practices were deemed unhelpful: receiving confusing advice or perfunctory comments; having reviewers be too critical; being inexperienced with the process; feeling self-conscious, vulnerable, unqualified to offer suggestions (Cava). 6. Several of the projects also point to issues that need to be taken up by students themselves. Responding to student writing, as many researchers have pointed out, is one manifestation of a theoretical lens that sees writing as conversation, or more broadly, interaction. When faculty members work hard at improving their ability to communicate, they expect a concomitant effort on the part of students. In other words, they want students to be active conversational partners, and feel disappointed when students "drop the ball" in what the faculty member sees as a reciprocal arrangement. The following comments from instructors illustrate this dynamic: If students do not understand the instructor's feedback what then- 10 do nothing, 10 ask someone else, 21 talk to instructor in class, 8 go during office hours, 9 Email instructor, 4 other. My reaction to this is mixed. 20 out of 50 students do not follow up with the instructor if they do not understand the responses - not helpful. Also, seeking a two-minute transaction after class as sufficient may well not significantly add to their broader writing understanding. For structural issues, having an actual discussion during office hours could have the most potentially helpful impact, and is rarely student initiated. This needs more attention to effect different behavior.(Schwab) The most startling realization to emerge from comparing these two sets of papers was that three of the nineteen students submitted their identical drafts as their rewritten essays for the latter assignment. Possible reasons may include students' inability or unwillingness to identify the errors themselves and/or procrastination and subsequent end-of-semester time limitations, apathy, or a combination of these and other factors. (Messina) A small, though significant proportion of students ignore instructor comments altogether, even when there's much to lose as a result. It is unclear whether this is a default choice (based on the inability to comprehend the comments, poor time management, etc.) or an action based on previous writing critiques. (Messina) Interestingly, few students reported going to the web sites listed on the template when they were told their performance was weak in an area: 17.6% never did and 26.5% only did so rarely (8.8% Always; 14.7% Usually; 29.4% Sometimes). (McVey) These comments not only provide self-report data from students, but also reflect faculty members' frustration at not being able to reach a small but significant number of students. The good news is that these faculty members were apparently successful at eliciting honest self-reports about student behavior. The bad news is that these self-reports can potentially add fuel to a deficit view of some students as simply lazy or not interested in improving through the educational opportunities provided to them. 7. Students who were involved in research felt intrigued and valued. In both Schwab's and Ochoa's study, students apparently found it interesting to be part of a research project. Whether this actually had any impact on their learning is unclear, but at least it caught their attention! In Ochoa's study, 100W students were responsible for designing and administering a questionnaire about feedback for students in other classes. This experience gave them insight into the processes of doing research. In Schwab's study, students didn't actually do the research but were very interested in having a voice in improving faculty effectiveness. Schwab writes;
8. Faculty who participated in professional development articulated the following insights and needs: * Simple shifts in thinking can be often be very powerful. Participants in Fassett's workshop found the shift from thinking about "grading" to thinking about "responding" helpful (they felt less tension and dread, greater calm). * Faculty want more support on how to assign letter grades (Weinstein) * Adequate time has to be built in to professional development so that faculty can really engage with practice in reading student papers and commenting on them. (Weinstein). * Faculty want more workshops with a focus on evidence-based strategies for responding to student writing (Drabble) * Faculty wanted more and deeper professional development on giving feedback (Fassett). 9. The institutional pressures on our campus negatively affect faculty members' ability to provide effective feedback This theme emerged in several of the professional development workshops as well as in individual project reports: * The most overworked and underpaid faculty members on our campus - adjunct faculty - are often the ones responsible for providing the most writing intensive classes (Fassett). In other words, we "outsource" our writing intensive classes to faculty members who are paid less, have less secure status, and often work two or three part-time jobs. * Writing intensive courses should have smaller enrollment caps. For example, Fassett wrote, "I teach 27 students in 100W; each student writes 8000 words;I"m to take up multiple drafts of assignments; and I'm supposed to provide substantive feedback (in addition to the preparation I do for my other three courses, research and service)." * Ochoa writes of the need for systemic, institutional change to address issues of faculty workload -both number of courses and number of students per class. She particularly cites classes with 50 or more students. If faculty are to provide meaningful, effective feedback on student writing, the institution has to support them to be able to do so. 10. Providing incentives and support to both faculty and students attracted both faculty and student participants. To begin with, this phase of the project operated on the assumption that faculty need incentives in order to volunteer for projects that demand their time. The main incentive for the eight faculty project leaders was the small stipend of $1125, which was probably inadequate given the amount of time they actually put into the projects, but nonetheless symbolized in a tangible way our appreciation of their efforts. Other less tangible incentives might have been the networking and collaboration with other faculty members; the offer of coaching by project staff; and the group human subjects application, which removed the need for each person to apply individually. The eight faculty projects in turn provided an array of incentives to encourage participation. The departmental workshops offered by Drabble and Fassett both included free lunch; Drabble also offered a gift certificate to Starbucks, and Fassett offered free parking for community college faculty. To encourage student attendance at a peer review workshop, Drabble offered students an extension on a writing assignment. In McVey's study, students were given two bonus points on the final term paper if they completed the survey seeking their opinions about instructor feedback.
|
|
