Writing Assessment

Writing Proficiency Assessment One (2008-11)

Excerpts from the WASC EER Report 2012

To measure value added, the ULO Project on Writing assessed skill attainment at three key educational levels: first-year, 100-level GE composition courses; 200- and 300-level GE writing-intensive courses; and discipline-specific senior courses that emphasize writing. The chair of the ULO Writing Committee was the English Department’s Director of Writing, whose specialty is composition assessment and pedagogy. To obtain a consistent framework, the committee developed the four-point University Expository Writing Rubric (PDF) based on five traits of effective writing: purpose, synthesis, support, style, and mechanics. The committee examined persuasive essays of four to six pages in length because curricula across all levels and majors emphasize this type of writing.

Method. The committee collected work from 56 class sections that either had a GE designation of “writing intensive” or were taught by faculty members who made writing a priority. In total, the committee collected 1,147 essays. From this pool, the committee randomly selected 272 essays for scoring: 88 from freshmen, 41 from sophomores, 54 from juniors, and 89 from seniors. 153 of the essays were from men (56%), and 119 were from women (44%), which approximates the university’s gender mix.

Figure 1.1 (below) shows the sample’s college breakdown. There were three norming and scoring sessions. Once inter-rater reliability was established, two readers scored each essay, from which all identifying information about student or class level had been removed. Because of time constraints, the two scores were averaged rather than using a third reader to resolve discrepancies. The average scores were used in the following analyses.

Figure 1.1 Numbers of Writing Participants as a
Function of College and Class Year

Class YEAR	CAED	CAFES	CENG	CLA	OCOB	COSAM	Total
Freshman	12	14	33	6	14	8	87
Sophomore	0	7	16	11	4	3	41
Junior	2	13	13	12	8	5	53
senior	18	7	11	27	3	23	89
Total	32	41	73	56	29	39	270

Results: Class Level Comparisons. A statistical analysis compared the variables of Class Level (freshman, sophomore, junior, senior), College, Gender, and Trait. Only Class Level and Trait were significant (see Appendix 1.1 for full statistical analysis).

Figure 1.2 (below) presents student scores across all traits. A follow-up analysis showed that freshmen scored significantly lower than sophomores, juniors, and seniors; no additional progress in the mean total was evident after students’ sophomore year. In other words, seniors differed from freshmen in skill attainment but did not differ from sophomores and juniors. No other significant differences were found for Class Level. The data also show that about 20- 25% of sophomores, juniors, and especially seniors did not earn a score of 2 (average attainment) in their writing overall.

Figure 1.2 Overall Mean Scores Across Class Levels on Writing Participants

• Poor/No Attainment (Score 0/1 < 2)
• Average Attainment (Score 2 < 3)
• Good Attainment (Score 3 ≤ 4)

class	N	n	%	n	%	N	%	Mean
Freshman	87	44	50.1%	38	44.7%	5	5.8%	1.97
Sophomore	41	11	26.8%	27	65.9%	3	7.3%	2.32
Junior	53	12	22.6%	36	67.9%	5	9.4%	2.28
senior	89	23	25.8%	54	60.7%	12	13.5%	2.36
Total	270	90	33.3%	155	57.4%	25	9.2%	2.21

Results: Trait Comparisons. Follow-up comparisons showed that students were significantly stronger on both Purpose and Mechanics, which did not differ from each other, than on Synthesis, Support, and Style, which also did not differ from each other. The trait results suggest that these three higher-level writing skills need further development regardless of class level.

The scores in Figure 1.3 (below) present student attainment as a function of the specific trait assessed. For each trait, the figure shows the percentages of students earning a score of 2 or better on the rubric, as well as the mean score for each trait, all as a function of Class Level. For Purpose, freshmen scored significantly lower than both sophomores and seniors. No other Class Level comparisons were significant. For Synthesis, freshmen scored lower than both juniors and seniors. For Style, only the difference between seniors and freshmen was significant, with freshmen scoring lower. Finally, for both Support and Mechanics, follow-up comparisons showed that freshmen scored significantly lower than sophomores, juniors, and seniors, with no significant differences among these latter groups. It should be noted that most students reached average attainment on at least one trait. Mechanics was especially strong, with 73% of freshmen reaching average attainment or above; this increased to 83% of seniors, 89% of juniors, and 93% of sophomores.

Figure 1.3 Percentage and Means (M) on Writing Participants Scoring at least a 2
(Average attainment) as a Function of Rubric Trait Scores and Class Levels

Class Year	N	Purpose	Synthesis	Support	Style	Mechanics
Freshman	87	68.2% (M=2.09)	59.1% (M=1.87)	48.9% (M=1.78	65.9% (M=2.00)	72.7% (M=2.10)
Sophomore	41	87.8% (M=2.51)	78.0% (M=2.13)	75.6% (M=2.20)	82.9% (M=2.26)	92.6% (M=2.51)
Junior	53	76.0% (M=2.41)	75.9% (M=2.19)	75.9% (M=2.12)	72.3% (M=2.14)	88.9% (M=2.51)
Senior	89	76.3% (M=2.45)	73.0% (M=2.23)	83.1% (M=2.37)	76.4% (M=2.28)	83.1% (M=2.47)
Total	270	75.4% (M=2.33)	69.9% (M=2.09)	69.5% (M=2.11)	73.2% (M=2.16)	82.4% (M=2.36)

In sum, analyses of the mean scores for each trait yielded the following observations:

• Seniors had higher scores across all rubric traits than freshmen.

• Juniors scored higher than freshmen on Synthesis, Mechanics, and Support.

• Sophomores scored higher than freshmen on Purpose, Mechanics, and Support.

• Sophomores, juniors, and seniors exhibited statistically equivalent levels of attainment across all traits.

Other Writing Assessments English 134.

In AY 2008-2009, the Associate Dean in the College of Liberal Arts and the ULO Writing Consultant conducted an assessment that compared students’ initial and final essays in the first-year composition course, English 134 Writing and Rhetoric. The original sample was 156 students from 7 classes. First and last essays from 56 students—8 from each section—were randomly selected for assessment. Essays were scored using an earlier, holistic draft of the expository writing rubric. Final essay scores were significantly higher than those on the initial essays. As a follow-up, scores for both initial and final essays were compared to a constant of 3, indicating average attainment on the holistic rubric. Initial essay scores were significantly lower than 3; in contrast, final essay scores did not differ significantly from the constant. A separate test showed that initial and final essay scores were both correlated with final grades. Initial essay scores were weakly correlated with final grades, whereas final essay scores were significantly correlated with final grades.

The overall pattern of results with regard to the initial and final essay scores yielded promising evidence that students significantly improved in their writing during the quarter, that this improvement moved students to an average and acceptable level of attainment, and that the final essay scores were indicative of final grades. Importantly, the data showed that students progressed from minimal to average attainment of writing skills during the quarter. This finding is consistent with the ULO-based assessment results reported above that show gains following the freshman writing experience and suggest that students retain these initial gains.

Graduation Writing Requirement. All CSU students must satisfy the Graduation Writing Requirement (GWR). Cal Poly students can meet this requirement in two ways:

• Earn a C or better and successfully complete a timed essay in a GWR-designated, 300-level, writing-intensive GE course. Students who are unsuccessful receive feedback and at least one more opportunity to complete the essay. The pass rate was 84% for AY 2010-11.

• Pass the Writing Proficiency Exam (WPE), a 350-500 word, timed, expository essay test scored by writing experts and other faculty members. The WPE pass rate was 70% for AY 2010-11.

 The essay and exam results likely constitute non-comparable samples for several reasons: students select the method of administration; the tests are administered in different environments; the content differs from test to test; the scoring differs across test types; and students taking the GWR course receive feedback and have a second opportunity to write the essay. In addition, each test may attract a different population, a factor that may interact with variables such as college, ethnicity, interest in writing, etc. To date, this question has not been looked at in a systematic way because the data have not been readily available. Finally, the essays administered in a GWR course may not be suitable for drawing university-level conclusions because they are only assessed by the instructors of record. However, multiple readers score the WPE using the WPE scoring criteria, which differ from those of the expository writing rubric. WPE readers assign a single score ranging from 1, ineffectual paper, to 6, exemplary paper, based on four traits: comprehension, organization, development, and expression. Stronger connections could be made between the WPE and expository writing rubrics. The expository writing rubric could be revised to function holistically, allowing readers to assign one score to an essay. Conversely, the WPE rubric could be revised to function analytically and thus provide more formative results. The latter approach seems appropriate as the WPE rubric was developed some time ago outside the framework of university-wide assessment.

Employer Surveys

In various surveys, Career Services has asked employers to indicate both the importance they place on certain skills, including written communication, and the degree to which Cal Poly graduates demonstrate attainment of these skills. The data in Figure 1.4 (below) show a discrepancy between the importance employers place on written communication and their perception of the skill level graduates demonstrate. For example, employers of graduates from the College of Engineering gave written communication a mean importance score of 4.41 on a scale of 1 to 5 with 1 being lowest and 5 being highest. Yet in assessing the industry readiness of engineering students, employers gave students a mean score of only 3.86.

This discrepancy is especially important because employers consistently rank communication among the skills they value most in employees. Considering the ULO data showing that senior-level Cal Poly students generally do not outperform sophomores and juniors in writing, it would seem that additional instruction or an increased emphasis on this skill may be warranted.

Figure 1.4 Written Communication on Recent Employer Surveys

College	Program/College	Survey Year	Mean Employer Importance	Demonstrated skill attainment	Rank*
CENG	College-Wide	2008-09	4.41	3.86	First
OCOB	College-Wide	2008-09	4.06	3.80	First
CAFES	NRM: Forestry &  Natural Resources	209-10	4.59	3.88	Second
CAFES	NRM: Environmental Mgmt and Protection	2009-10	4.62	3.75	First
CLA	GRC: Graphic Comm	2009-10	4.63	3.95	First

Recommended Action Items in the WASC EER Report 2012

• Ensure that Cal Poly juniors and seniors continue to improve their writing skills (p. 4, 5).
• Coordinate efforts with the University Writing and Rhetoric Center to develop and raise awareness of outreach programs that target upper-division students.
• Identify upper-division students who struggle with writing before their senior year, especially ESL students, and offer additional upper-division writing courses for these students.
• Coordinate efforts with the CTL and the WINGED (Writing in Generally Every Discipline) program to offer workshops and develop learning communities for faculty members who teach upper-division, writing-intensive courses in GE and the major.
• Emphasize the value of writing in every discipline by identifying non-GE, upper-division, writing-intensive courses in the majors and across colleges; if such courses are difficult to identify, work with departments to develop discipline-specific, advanced writing courses, possibly tied to the senior project.
• Actively support Cal Poly’s acquisition of an e-portfolio and assessment management system so that students can document and assess their own progress as writers. 2. Align learning