OCLC Meeting 4-05-17

Meeting Details:

Attendance:

Jack Phelan, Trevor Forzetting, Richard Besel, Bethany Conway, Cassandra Carlson, Katie Tool, Rachel Showalter

Business:

Prior Meeting Notes Approved - March 14, 2016
Updates on Lower Division Qualtrics Scores
- There were 14-18 disrepancies per pair of raters. They may have been off by one item or by multiple but this means that these particular speeches may have to be watched again. This problem developed in part due to an issue with the 'force response' option on the grading page, causing raters to skip items. The raters have been asked to go back and address the skipped items. A total of 40 out of 100 speeches will likely need to be viewed again by a third rater. Depending on what items the raters were off on, the entirety of each speech may not need to be watched again.
Updates on Upper Division Artifacts (prepping for Qualtrics)
- Collection of the artifacts is complete.
- New Student Assistant, Rachel Showalter, will process the artifacts to Youtube, log them, hyperlink the artifacts to a comprehensive document, and prepare the form for Qualtrics.
- Order of Execution: Meeting with faculty, norming, discrepancy scoring, feedback
  - Week 2: Jack requests "discipline-specific" rubric requests from program faculty (1-3 items max), plus open field for observations
  - Week 4: Schedule Norming session (April 28) to get raters ready to code
  - Week 5: Distributing artifacts to faculty raters using Qualtrics
  - Week 8: Three week deadline for scoring artifacts
  - Week 9: Identify discrepancies and schedule debrief meeting
  - Week 10: Complete discrepancy scoring and coding

CLA -- Communication Studies -- Faculty lead: Cassandra Carlson COMS 470
CSM – Statistics Senior Projects -- Faculty lead: Beth Chance
CAED – Architecture “Best of Show” -- Faculty leads: Thomas Fowler, Mark Cabrinha
CAFES -- Natural Resource Mgmnt & Environ.Sciences -- Faculty leads: Chip Appel ERSC 363

COMS Consultants
- Protocol / Scope of consultancy
  - Meet with OCLC to discuss scope of consultancy. 5 out of the 6 current raters will likely be needed for consulting.
  - Meet with faculty for norming sessions, begin by showing what a 101 speech looks like and how best to evaluate it. Then form breakout sessions among consultants and coders to discuss how various classes and speeches may differ in form and grading.
  - Discrepancy scoring - 3rd coder role
  - Facutly / Department Feedback: answer questions in a Q and A format. But the session should primarily be focused on assessment rather than curricular decisions.
Next Steps / Action Items
- Schedule April 28th meeting, 9am-11am, for week 4 norming session
- Schedule April 14th meeting, 9am-11am, for debrief of lower division scoring, at least one Learning Community Member will be in attendance
- Katie will send out 20 page document to OCLC
- Publish names of program faculty who are involved in the scoring process