As you finish your class log, post it here, as a response. Please include the date in the title, so we can easily access the week we want.
Thanks!
The Management
8/25/16 Part 1 Introductions: the story behind our names Recounted as best I could after I ended up signing up for today. So, seeing if Bill’s theory of its memorable-ness is sound. Bill used to be Billy and when he was younger and at camp there was another Billy with an awesome middle name of Bull (to which I thought, “Really? His parents named him Billy Bull?”). With such an awesome middle name Bill wanted to know what his middle name was. To his dismay, he discovered his middle name was Francis, a girl’s name. Edie-Marie has two hyphenated first names because her Mom doesn’t believe in giving middle names to girls (abbreviated version). Lacy is Lacy because of a compromise. Her dad really wanted a boy after two girls and had a name picked out: Robert Allen. This was transformed to Roberta once it was determined that Lacy was not a boy. Lacy’s Mom wanted Latisha after one of her Mom’s dear friends growing up. Both parents felt mutual disdain for the other’s name choice. Lacy was the compromise for which Lacy is grateful (though I think she has the stuff to rock either of those names should that have happened ) Baby boy Triana is named Mark because his parents could not come to a compromise. His birth certificate reads Baby boy Triana as a result. Mark’s brother named him some time later (one or two weeks? I can’t remember). Tabitha was also not given a middle name from conservative parents (like Edie-Marie’s) and is named after a character in the bible. Anna Karen – Karen – has two first names from a Swedish tradition but only her Mom would ever call her Anna Karen. Work and life is Karen, teaching ELL students is Anna. This became a problem when her students tried to get her attention when they saw Karen out and about and were shouting “Anna!” for quite a while before it registered. Sarah Blume (apologies if I have just misspelled both your names) is named after a biblical character as well. She’s not sure what that’s about because Sarah is known for laughing at God (which I think is kind of badass). Her last name pronounced ‘bloom’ is commonly mispronounced ‘Blum’ or ‘plum’. What this class is about: Know the theory and writing around assessment, student assessment, program assessment, self-assessment -- Assessment to drive learning Coursework – (also on the syllabus) Assessment project that is real A book review – recent book in the field on assessment Assessment-friendly assignments for our own teaching Each week after class post a discussion question (by end of Friday) http://engl508f2016.blogspot.com/
8/25/2016 Part 2 Handout: Performance assessment v. traditional grading Coffee making example – we either have coffee or we don’t at the end Class blog, either you posted or you didn’t, for example – but we get to set the agenda so take the space and make it what we want to make it Several of the assessment friendly assignments will be performance based Ex) taking points off when an assignment is late and the idea that timeliness is a life skill and just doing the danged thing is often more than half the battle So, missed deadlines will equal missed performance points Q&A for example means that students sort themselves out When it comes to grading the best way to approach it is to think outside the box Second handout – excerpt from The Mismeasure of the Man by Steven J. Gould – count the E’s on page 220 Answers: 183, 237, 228, 244, 251, 252 There is not really a way for us to know for sure how many E’s are on page 220 – there are limits to our cognitive abilities – how does that affect us? How do we process the idea that there are things we will never know while trying to know more? How we counted, obviously impacts the count as we all got different numbers. The take away is that even in something as concrete as how many Es are on page 220 there are limits and we need to stay humble. Always ask, what are the limitations? With AES (from the readings) there are serious limitations and yet there are sweeping actions made, real consequences from using the AES. SAT scores example. Some use SAT scores as placements even when SAT even says it’s not for placement. And, there is direct correlation to SAT scores and income ($$=higher scores) We could place students by how many bathrooms they had in their home this way There is a good deal of literature that suggests those students placed in remedial English courses are much less likely to complete their degree Questions on the syllabus…
Assessment knowledge is used in the workplace of education, period. …Knowledge to fend off bad assessment is as important as creating and improving assessment. With the WPA outcomes, pay attention to the difference between what and how well Outcomes are what we expect our students to be able to do, not how well In forming the WPA they realized that there is no way students will have write to the same standards. They have diverse backgrounds and are starting from different places. Ex) students from Arkansas poorer backgrounds and Michigan better off economically – to expect them to write at the same level isn’t unrealistic, unfair But they can be working toward/with the same ‘what’ Outcomes v. Common Core Outcomes was always intended to be a living document. That we will learn more and adjust to the needs of students as they evolve. Standards seem to change less than outcomes. About every 10 years tools of assessment, outcomes, standards, need to be revised! In the 90s tech was viewed as a tool for writing versus a way for communication, issues of use, access, etc. But then by 2008 access was a whole different situation and there were multimodal projects and the internet as new and multiple ways to communicate.
9/1/2016 Apologies for the delay - I've been having computer crashing issues.
Pre-class – my clarifying question Start with Guba and Lincoln, application and how things work themselves out Two different epistemologies (somewhat of a false dichotomy) When and what to choose for methodology Positivism Constructivism Only 1 Truth with a capital T truths Observer-external Observer included Methodologies: Scientific method qualitative, interviews and such Qualitative measurements Context driven and nonlinear Hypothesis, yes/no but willing/able to be proved wrong Results are questions with a very small scope open-ending, larger more complex questions Goal: knowledge as fact Goal: knowledge as mutual understanding Truth exists and can be known/constant What we need is a consensus that we can work with, act on
Ultimately Guba and Lincoln are getting at, positivism is appropriate sometimes (Zika virus example) Sometimes constructivism is the appropriate response
--Plato, knowable sophists – relative
Creating a test for writing, seems like positivism, but really falls somewhere in between the two Ex) our placement exam at WSU
A humility and testing hypotheses over and over, over time as central to Guba and Lincoln -Mark I don’t know what I don’t know but I know what I need to ask (positivism) and the humility part falls under constructivism where we admit we aren’t at capital T truth yet (regardless of whether it is knowable)
Example of the lightbulb. Knowledge evolves, and evolves in a straight line (positivism) Knowledge evolves like a spiral (constructivism) –teaching is that thing that no matter how much you know there is more to know “We’ve never taught the same course twice” (they say/we say)
Why did Guba and Lincoln put positivism and constructivism as opposing each other? They have a bias toward constructivism, that it’s more sophisticated and harder Reality is they are both difficult
A fourth generation assessment: the hermeneutical circle of/for information and then the stakeholders act. After a while it’ll need to be done again (more information and things change) [the spiral goes round perhaps]
Does constructivism allow us to put the different positivism approach in conversation to each other? -Lacy Answer: yes. And yes, ideally. Example, timed writing Constructing the prompt, then given to the test-taker, and the readers (three groups) Test-takers that wrote to how the readers read the prompt did better – but how to bridge the gaps? The prompt writers and readers can sit down together but the test-takers remain in the gaps Hitting the audience of the test-takers is something we can’t quite ever really know Example of prompt development with the four-parts, ends up with about 100 prompts because – example of prompt involving a zoo. For rural and certain countries, it isn’t a good prompt because the person being tested isn’t exposed to or have access to zoos. Prompts have to be continually updated because the student demographic shifts – constructivist – but it is studied, assessed, and changed off of positive methods. If a prompt excludes even one group of people, it needs to go – figure out the deficit in the prompt Methods and information will differ with what our goals are
9/1/2016 part 2 Consumerism and education – are the students consumers or beneficiaries? (parents as consumers) Parents demands v. student demands Then, with consumerism and funding, there are donors and so we have (potentially) a hermeneutic circle – how to develop education to appease all the players: students, teachers, parents, donors Example of business and “real world” people giving feedback on what the (value) of what classrooms were doing – what it looked like to the real world Managers/assessing have a lot of control – so there are problems (immediately) with running a university like a business Trends of putting business-background people in charge of education and so far results are disastrous (so it’ll work itself out – yeah!) U of Phoenix as one that is (kinda) making the business model work So, the methods (positivism and constructivism) help us figure out what is working and what isn’t (overall business method don’t work)
Positivism – relatively simple, a question we can ask Constructivism – complex, the question might not be known
From pages 180-81, Tabby, what to actually do with constructivism and the value of vicarious experience Also, theory, practice, and praxis – Condon The common understanding accounts for a programs overall success – if parts are hidden or left out, there are a lot of divergent understandings it can’t operate as well Example of revision of mentoring tenure track professors in the English department (because it was identified as not working) – then it worked well and other administrators want to take a look at that Then there is a set of procedures for non-tenure – and with both it’s something that needs to be revisited regularly because things change and because that keeps people familiar with the procedures Shared and involving more stakeholders, for things like mentoring program, is needed for it to work well Discussing, example of the portfolio system for English 101 at WSU Condon – studies show students work harder with no grade/delayed grading Portfolio makes students a stakeholder, gives them choice Break
Lecture 3-Assessment-Friendly Assignments Thursday, September 8, 2016 2:58 PM • Assessment-Friendly Assignments o Means for assessing the effectiveness of writing assignments o Every time there is an iteration of a course, we need to evaluate how to improve the course in the future • Creating Assignments o Weekly, performance-based assignment • The same task, done weekly (or regularly and frequently), as long as it involves facilities worth developing, can show progress from week to week, thus demonstrating students' learning. You could consider every week, or every day, pending on the situation Can show progress incrementally Allows you to examine progress of understanding--provides lots of information on what they're learning and how they're learning it Can give you evidence they're learning beyond • First/last week "paired" assignment Giving similar tasks (again, tasks worth doing) at the beginning and at the end of the course can provide evidence of 'value added.' • Doesn't have to be identical, the pre- post- assessments • Assignment to be assessed with scoring guide Developing and sharing a detailed rubric for evaluating students' work helps them address the dimensions of good writing (or other learning) that the rubric describes. If the assignment sets a task that can be evaluated fully with the rubric o Example: Speakeasy Café • Developed for World Civilizations course • Students can go in, select a question, and respond to it • When designing, you want this space to be for students, not for the instructor • Students posed: Fact-based questions Class process questions Discussion-based questions • When looking at the examples, one student in particular demonstrated the following: Provided more context Narrowed focus Greater care and consideration was given to forming questions over time • Students weren't graded for the question, other than participation Allows students to sort themselves out on how much they want to learn • This type of assignment is low-risk, meaning students can try new things, play around with content and language and won't lose points Gives them a place to explore Seems more fruitful in upper-division courses • Provides a more organic means of assessment, rather than just appealing to the rubric o Example: Objectives • Three objectives
Course Compact objectives Personal Class Teaching, Learning, and Technology • Have students read the objectives critically and reflect [forward] on their reactions to them (first week) • Have students look at their initial statements and reflect on how their expectations played out in the course (last week) Students don't have to just draw on their writing process changed, but other aspects as well (e.g. reactions to group work) • Assignment that Addresses Rubric o Functions • Clarifies expectations for students Students will know what exactly they're being graded on • Helps faculty devise better assignments Can see what works and what doesn't What are students struggling with? What are they exceling in? How can those considerations be incorporated into future assignments? • Improves students' performance When students know what is expected of them, they can better focus and work on those elements Design the rubric to be "usable"--don't include more than 9 elements for grading Students can even help with rubric development Instructor can be "secretary" to developing the rubric • Focuses grading Can help you grade in a more timely manner Circled numbers on a rubric can function as feedback on their own • Facilitates communication about quality When we talk about quality, we are not talking about stasis Students gauge quality by the amount of time put into the assignment Rubric gives you language to talk about quality • Promotes improvement over time Assignments and timelines need to allow for this • Helps fight plagiarism Allows students to know what the professor is looking for When they are unsure, they may plagiarism o Need to align with course and assignment objectives o Numbers on the rubric can tell about the success/understanding of the assignment o Want to look for (according to Elbow): • Not good enough • Good enough • Praiseworthy o Rubric and the WSU portfolio • Rubric should communicate a consistency in evaluation throughout the portfolio statement • Could consider using a scoring guide with the dimension of element presence: Absent Present Sometimes
Frequently Present Consistently Present o Rubric expresses what you and your class value in writing • Want to center your rubric building around those values • Creating Assignments o Inquiry-based assignment • Asking questions that we don't know the answer to can provide rich evidence of students' abilities as lifelong learners. These assignments necessarily move beyond having students divine "What the professor wants." • Allows students to put important topics into public, class discussions o Collaborative assignment • Since collaboration gets the learning process out in the open, we can devise richer and more varied ways to document that learning, and we can more easily see where the learning is going right or wrong • When students work together, they have to negotiate everything o Peer response/process assignment • These activities also get the learning process out in the open, where we can document it. They also provide evidence of important life/work skills. • Example: An environmental science professor assigned a writing project in which the initial draft was due 100 points and the revision (due a few weeks after the professor returned the works with feedback) was worth 50 points • It's a good idea to make your classes a feedback rich environment • Have to develop assignments that require all members to contribute equally EX: Students have 200 accountability points. When a student does something that lets the group down, they have to "give" their groupmates their points • Goal is to maintain the 200 points • Students could end up with a negative score • Can build on process by having students build and revise off of a piece all semester; receive feedback regularly Consider only the last 5 grades Helps the students determine how hard they want to work Don't receive a grade unless it's a C or higher o Remember that as you develop your practice and assignments, it takes at least five (5) iterations to fully realize the pedagogy • Class logs o Having consistent writing assignments can help students internalize information o Allows students to approach the lecture and the content differently and perhaps more carefully • Key Factors for Success o Quality of task o Time of task (1) • The more time a student puts into school, the better they do o Interaction with peers (2) o Interaction with faculty (3) • Find a way to interact with the students without ending the conversation based on their assumption that the instructor's understanding is the end all be all o Promotion of course objectives • Assignments Alter Dimensions
o Examining native speakers and non-native speakers and the features of their writings in particular disciplines o Concern of collecting and analyzing data o Could focus on high, medium, and low o Examine how language and structure influences a student's understanding of an assignment • E.g. how might a term like "discuss" be interpreted differently based on the student's home culture o Reference Murphy and Ruth o Look into CompPile.org • Good for gathering resources o This project could serve as an ideal pilot for further research • Mark: WPA Outcomes and Technology o Examining how technology is discussed with WPA outcomes over time o Hardware and design • Similar to the difference between hardware and software • Introduces questions of access • Hardware: more material • Design: Examining the aesthetics (pulling from Kress, Rhodes, and Alexander) o How did the assumptions on hardware and designs influence the language of the WPA outcomes? o Methodology: Still solidifying; phone or email interviews with those tasked with defining access, and how we went beyond questions of class to questions of applications o Project has the ability to test or observe the distinctions in action • Looking at post-embargo Cuba and DIY culture • Reference a DIY collected across the island o Explore how writing assessments address technology • Could bring in CCCCs statement • Explore how much power an administrator can have over the technology students use o Mark Weisser: Ubiquitous computing; college campuses are a battle ground for tech use; the presence of
SYLLABUS QUESTION: Design one assessment-friendly assignment that's aimed at your practice; have peer review with our seminar; could inform your 508 project and/or the classes you teach; needs to be done by September 24
Assessment at WSU – 9/15/2016 Pt. 1 Assessing the portfolio – foundation at WSU. It was a moment in our history that allowed us to describe the evolution of the writing programs. Beyond Outcomes Attempts to bring in the theory of portfolios. It holds essentialist discussions for portfolio assessment. What makes a portfolio? What makes it work in the context of a course then think about the program level? What makes the portfolio system work? Portfolios as assessment have been around since 1970s. The collection of work is thought to be better than a single piece of work for many reasons. What makes a collection more appealing than assignment by assignment? • If you want to show all your important attributes, no one can do that in one thing. There’s no one exhibit that can show everything. There’s a historical thing to consider here. We come from a past – I came from FYC where the writing was done in bluebooks in class. Product-based, time constraints, all that stuff. If you read about the college experience of writers who have written memoirs, that practice goes all the way to the 1900s. 1960s, people started having what was done be composed outside of class. It’s not surprising that people started playing around with portfolios in the 1970s. Another thing that’s inherent in a collection is time. If you’re grading piece by piece, you have to grade it at once. With a collection, you have time to revise. What you see emerging in 1960s-70s is where comp incorporates process and drafting. Linda Flower, Jimmy Moffitt – the notion that a piece of writing is never finished, it’s just due; that it can continue to change. 90% of writing happens in revision. Having a collection buys time for revision. Which buys time for learning. SO you’re not caught trying to make up for earlier low marks and mistakes. The piece by piece grading system assumes that no one is going to get better. Ability is ability and you’re showing it in different ways – narrative, description, compare and contrast, different modes and they got complicated as you got along, but your writing ability was just a thing. But students learn, progress, and how do we accommodate that? In response to feedback, students revise and edit. It turns 5 papers into 10. How do you accommodate revision and learning through assessment and revision? o Allows students to achieve the outcomes you’re after and select the best of that collection that matches those outcomes. o You can’t grade paper by paper and get anything about metacognition. You get what they did. Either a student could or couldn’t do X, but you don’t know what it is. That’s why reflection is so important. o With selection, you can find how they’re reaching outcomes. That process of selection is a very considered process and it involves a lot of inherent metacognition – how they’re strengths match the overall outcomes. Edie-Marie – I did portfolio based class with choice, but you didn’t have delayed grading. You got to choose which ones, but you could revise. In revising it, the improved grade would replace the lower grade. There was incentive to pick the ones you struggled with and grow from it. The final portfolio HAS to be at least half the grade. What are the implications of that system? • There’s nothing wrong with students going back and doing their writing, but they value grading. Letting them know where they stand as they go along may help. What are my motivations if I’m in this system? Mark – responses to feedback. • Bill –figure out how to create a class that holds lots of feedback – it will improve diagnostic skills, among other skills.
Pt. 2 Bill - Teaching without anything but the final grade is hard to do. It’s a real challenge. You can read everything about the harm that grades do. Grades encourage a be good mindset instead of get better. “A” is a ceiling. Why would you want to revise? Well, I’ve arrived. Portfolio is one of the things you need to teach in that context. You’re going to learn a lot and I’m not going to grade it until you make it as good as you can. The research on ungraded classes shows that helps but it doesn’t make the students comfortable. o Bill – you’re always holding them accountable – whether in the grade or feedback. Motivation for starting with a lousy draft, that motivation assumes that there’s extra grading reward for that improvement. Part of getting an A in the class is the idea that it’s necessary to improve. Someone who gets As in their writing will write a C draft. But if there’s no ceiling and what they’re going for is objectives, then they don’t need to play that game. You just have to be careful in handing revision so you don’t place incentive on bad drafting. • I think it’s great that the standard is how it is, but we need to think harder about how to scaffold it so it works. The portfolio has a lot of elements, and how do you get students to internalize what good writing means in such a way that it helps them reach that. • Mark – I thought about incorporating the outcomes in specific ways. I made a forum on blackboard that is grade commentary separated in the five outcomes. Using their work as examples from this particular perspective based on the outcomes. They’re not only practicing the outcomes, but evaluating themselves. Mark – reflection on outcomes. What range of reflective prompts have you seen? o Bill – the level of challenge is the key factor. The weaker ones say write a reflection. The better ones will tell the students to look at the outcomes, and how those pieces reflect those outcomes, what did you learn? What did you go through? How did you get there? How do the exhibits show that you’ve achieved these objectives? They’re created in a way to talk openly and deeply about the process. Not the diary entry version. Reflection is opportunity for self-assessment. It’s probably because they couldn’t take that opportunity. Reflection is where students get to talk about what’s in the box. (122). o Formal reflection gives them chances to show metacognition. Look at the depth of their learning. • Bill – Read The reflection instead of the entire portfolio. If you build that reflection well, then everything you need to know will be in the reflection Mark – genre of reflection – They’re able to talk to me • in their writing, but I haven’t tried any other way. o Bill – we started off with the executive summary or cover reflection. My favorite reflection was in the form of a closet drama. It was done as a couple sitting over dinner talking about the writer’s portfolio. But in the dialogue, everything came out. SO the possibilities are endless. I’ve tried to ask students to be as creative as they could possibly be. It gives them agency. BREAK Programmatic Assessment USING ASSESSMENT TO SUPPORT INSTRUCTION – Beyond Outcomes – graphic in introduction. • Left side is linear inclusion of assessment and instruction • right side is the amount of writing. How little writing should a student work on in order to achieve this. It adds up to at least 100 pages of finished work – but really 400. • Last category on the right has to do with effects and impacts within the reach of the program. Effects on people who operate it. • The real version is wonderfully color-coded. It gives you a programmatic view. Short history with Elbow.
• Attention to Provost's statement of relevance of Q-value • 2 handouts: o "Office of Writing Assessment Washington State University Eleventh Findings (June 2013-May2015)" • Every year Writing Program submits report on portfolio assessment • Policy briefs by college • What sorts of inferences do you have? ♣ Credits you should have before submitting portfolio is 60 credits ♣ Mark: Those students are only doing half? ♣ Anna-Karin: How many credits have they completed? ♣ Condon: To completion; Have to do one or the other to register for courses; then later, cannot register until they've completed portfolio ♣ Lacy: How long to completion of degree? 4 or 5 years? ♣ Condon: 64% graduate in 6 years ♣ Edie: Statistics of those who graduate somewhere else? ♣ Condon: 70% finish college somewhere; roughly 80-20 ♣ Lacy: Do these numbers include non-native speakers? Held to the same standards? Cultural considerations? Writing Center? ♣ Condon: transfer, non-native, non-English speakers all included; same standards ♣ Edie: What is the purpose/what motivated the Passed with Distinction characteristic? ♣ Condon: The carrot; make something a graduate requirement and the administration consideration is to think about how to not make it another hoop and how to ensure it doesn't hinder degree • Motivate them to do it on time • 10-15% • Either Pass or Pass with Distinction on transcript or Incomplete • Complete it means they do UH 202, and once they complete that work, then they pass the portfolio • A little extra thing to put on their resume • Good for employers: employers say they want some form of writing among top 3 desirable traits (e.g. communication, writing skills, work independently, and work well with others) ♣ Mark: Page 5 ♣ Condon: At Tier I, all their reading is blue book; Haswell's genius is making rating process as efficient as possible • 3 pieces of writing marked at least Acceptable, and then blue book comes in as a pass, then 5 pieces of writing with 4 different faculty looking at it • No one single faculty member reads it all • Takes 60% out of the middle • 20% on Needs Work end; 20% with Distinction • Tier 2 decides ♣ Annie: Teacher feedback of excellence? ♣ Condon: 3 Outstandings = Tier 2 • Read class papers to see if they Pass with Distinction ♣ Lacy: Page 5 regarding ethnic identification; any trends observed of why that is and how to improve these numbers? ♣ Condon: Gap is closing; international students in particular have lower rate of Pass with Distinction and higher rate of Incomplete, and smaller rate is simple Pass • Some reasons are pretty obvious: when you look at programs, high International student rates have high Needs Work (criminal justice vs international business); look program by program; try to give better supplementary instruction and instruct faculty--> more success in sciences than humanities, as sciences are used to it and more accommodating • Look at data and see that in some cases, General Studies is a lot less populated than it used to be--old dumping ground; small number of students ♣ Condon: Look at final page; the ones who just gave the only 3 papers they had do worse than those who chose from 5 or 6; came across many students in phone surveys or front desk who just don't have them; challenge is there has to be enough writing in curriculum to have enough papers to choose from • 2 factors to consider: 1) sheer number of students involved in each class and 2) students put in papers they feel best about; the experience of producing that paper was a good one • Carlton portfolio system modeled on WSU: found that when professor has done workshop and pedagogy experience, students were most likely to choose papers from that course; the more the faculty learned to teach writing effectively, the more worthwhile the students felt their papers were and more likely to put it in their portfolio
♣ Condon: If you contrast the fact that students are taking 101 within first 30 credits (1st year and a half, on average) and you look at time to completion (have 79.5 credits), 101 paper was a long time ago; why dig into distant past than choose one when they're more experienced? • All students have to take 101 and teachers do good job and students feel good about papers ♣ Annie: Include teacher comments? ♣ Condon: Students can submit with all markings; students can go to professor and ask to sign off to submit paper to portfolio; prof might ask to make revisions before it goes into portfolio; students sometimes put cover sheet on portfolio and that's what authenticates score; profs initial each page to show they've read it ♣ Condon: Based on Top Paper Submissions by Program, what inferences do you have about papers submitted in science? • Lacy: The more writing is emphasized (English and History), the more central the writing process is to the course itself and the prof's pedagogy, the more likely the students will view their papers as a fulfilling piece--more of a connection • Condon: Consider how active the program faculty are as part of the portfolio assessment and their participation in Writing Program • How many trained as portfolio raters ♣ Anything surprising? • Annie: Physics • Condon: Math ♣ Condon when director of Writing Programs, then working with Doug Baker, tried to do something about math offerings; took a while but used undergraduate opportunity grants to focus on math (quantitative approach); difficulty is math courses everyone takes are taught by NTT instructors • People teaching GenEd requirements are much easier to work with • Delighted that 87 came from Math ♣ Edd: Woman's Studies is low. Not as many people enrolled in course? ♣ Condon: Look at enrollment; if it does enroll a comparable amount to other programs with 100+ submissions, then why not more writing? ♣ Mark: Do students know what portfolio expectations are within their programs? ♣ Condon: Look at completion data; something going on keeping students from completing faster ♣ Edie: Music has higher credit hours, takes longer to accrue • Condon: Data gives you ability to ask questions; talk specifically to department heads and ask why there are issues with time to completion; numbers give you the chance and if you're Dept chair or Program Director, you can see numbers and see what's happening and ask why-- content with answer or change with response? Assign more papers? Do better job of assigning papers? Data gives you question, and then answer takes longer • Condon: In English, things are pretty much what you'd expect- 60% are Complete, simple pass; 20% Complete with Distinction; 20% Needs Work ♣ What correlates more highly with success in first-year writing is SAT math section, rather than verbal section ♣ Overall portfolio performance: Among highest are English, German, Women's Studies o "Appendix C: Paper Submission by Prefix and Course Number 2013-2015" • Condon: What inferences can you draw? • Not in alphabetical order • Condon: Favorite acronym is here FSHN- Fashion? Fishing? Actually Food Science and Human Nutrition • Condon: In some programs, it doesn't matter when they take it sometimes and enrollment is so tight and backlogged, they'll take it whenever they can get in ♣ College asked how much it will take to get rid of backlogs in number of courses we teach/more sections ♣ Todd: $250,000, and College was expecting only $30,000 • Mark: Do they track how many submissions from online courses? • Condon: Don't know. May be tracked separately • Edie: Tracked by campus
• Annie: Less writing in upper division • Condon: Don't want to see a lot of papers in upper division ♣ Looking at Writing in the Major courses, as they have more writing • Lacy: With 402 they're starting to develop major-specific courses, and curious if development changes the amount of submissions from the different majors • Condon: Doubt it; same number of students, but kind of writing will differ ♣ Can look at this data and ask: Where is writing happening in my curriculum? Don't want to see it reduce ♣ In Engineering, schedule is so tight, they try to take 400 after 101, as a sophomore ♣ ABET requirements (Engineering accreditation body) are so rigid that students need to take courses exactly as prescribed ♣ Funding at level of deans to shift money allocations between departments--> transfer of funds between colleges • Mark: Chatter about Communications taking 102 • Condon: Mechanical engineering had been negotiating with Communications for them to offer technical writing • Condon: In English, all upper division courses are Writing in the Major courses ♣ A lot of papers coming from them; English has pretty narrow range; reflects the fact that for an English major, all courses are fair game; English completion point is 78.1, which is just a little behind but not much; English majors are getting lots of writing and getting enough to turn in • Condon: Molecular Biosciences; even though it's 101, it doesn't have big enrollments; that's where writing is happening coming into the degree; 304 is M course; not much writing going on as a whole, but chair decides what to do next ♣ In sciences, undergrad is banking model, just learning the facts • Condon: Natural Sciences are doing pretty well with writing and comparable to English; Animal Science doing well ♣ Tier 2 portfolios qualify for nomination for best writing portfolio; raters read nominees and pick 5; Best portfolio awards come from a wide variety of students from different disciplines and a few (7 or 8) come from Animal Science. They do a good job of getting them in publication pipeline and lots of mentorship as co-authors ♣ 120 students sometimes during graduation are setting foot on campus for the first time--> Global Ed • Edd: AMDT are waiting until 420 to wait to do writing • Condon: Chair can wonder why; small program; lots of papers from 420, which is supposed to be a senior level course; maybe can move to first year, 107 or 208 ♣ One of biggest gains is in Human Development • Lots of writing and students are choosing their papers for portfolios • 204 and 205 designed by Kidwell and won awards for it • Lots of writing in that curriculum; chair could go to dean and show success • Anna-Karin: Integrated Plant Management small numbers • Condon: I don't know what IPM is, maybe a subset of Horticulture • Edie: That major has quite a few courses at U of I. Can students use papers from transferred courses? • Condon: Cross-listed/campus courses can count; closer to home than papers from community college ♣ Whenever I go to someone else's 2-year college campus, I go to the bookstore and see WSU portfolio packet-- our reach as a program is far o Condon: The purpose of looking at these things is on the Writing Program page since 1995 • First read in 1993 and first buy-in was 1995 • Passed 80,000th portfolio o Annie: When a student gets an Incomplete, do they need to take 1 credit courses to complete? Or resubmit portfolio? o Condon: Meant to catch students who struggle at mid-career; want to prevent them from taking upper division before they are ready so they can get help from trained writing faculty; upper division instructors in the majors are not always trained to teach writing • Early research shows that we were placing 1/3 of students in 102 • Consider context: WSU incoming GPA average is 3.4; in Michigan 3.9 needs a prayer to get in • Of the students that place in to ENGL 102, only 10% of them stay at bottom ♣ 90% who do well at mid-career were not placed in supplemental at entry
• "Pick up a rock and see what crawls out": can be a range ♣ Transfer students do better in portfolio than native-enrolled student ♣ For the most part, those who enroll in 102 get in good help, get 4 credit-hours, 4 hours a week, and rise ♣ Condon has been teaching Basic Writing since it was Remedial English, and notices that they just need difference approach; not dumb, but just not experienced; by mid-course, they are ready for 101; could have been placed or choose to do 102 ♣ One of Condon's favorite overheard conversations between two basketball players: freshman mad at 102 placement, sophomore said he was "down" with getting help once a week; 102 designed to be a positive experience; when you give students that, they thrive ♣ There's always going to be a bottom 10%, but if it's the same 10%, then you have problems--you're not having an effect • Portfolio is biggest, bestest "rock" you can turn over; shows what you need to work on • Look at statistics and data by college and department and think like a dean, department chair, or program director and look at the richness o Condon typically shows data like this when talking to program heads, and their jaws drop because they want to be able to get data like this • Wanted to get to program level in discussion-- Writing Program here • Directing a Writing Program big or small, you have to have data ♣ Not just for program but institution also needs desperately • Assessment Project Updates • Edd: Trying to do a project on service learning assessment o Methodology: Lit Review, Polarize issue between quantitative and qualitative (narrative, discourse analysis, and genre analysis) assessment • Criticisms of qualitative: Hard to judge how much students have grown; diff contextualizing experiences; subjective ; lacks precision and magnitude • 2 scholars: Cushman and Schulz • Positivist vs constructivist ♣ Land somewhere more constructivist, but still incorporate some positivist • Argument: Do need hourly logs, pre-/mid-/post- (positivist assessments) along with qualitative assessments, and learning objectives o Condon recommends: comp folks developing service learning pedagogy; search Comppile • Center for Civic Engagements outcomes ♣ Don't stress discourse analysis ♣ Hopefully get something to shift there o Most service learning in Biology and Education • Contacted faculty for syllabi o Condon: Some faculty leery of giving syllabus o Question: What service learning objective will you use to line up with outcomes of class? • Annie: Looking at reflections from people who wanted to change the outcomes of their junior writing portfolio o Why students got placed where they got placed o Condon: That might be an interesting concept in itself; some students just don't understand requirement • Characteristics of that group: discipline, genre o Annie: Don't know what it looks like to put together this information o Condon: quantitative end- research on the teaching of English; don't look at heavily statistical ones; assessing writing can be fairly quantitative; more along the lines of discursive, not heavily quantitative; those two should give you good intro to genre; form follows function; what Condon likes about data collection process is collect, then decide what to do; naturalistic development
• Anna-Karin: Looking at how non-native speakers write lab reports o Contact in Bio department- haven’t heard from him • Looked at the data and found students in English 105 to get some comparison between good, middle, low of American students versus good, middle, low of those who take 105 • Waiting for data o Looking at methodologies and how to do framework o Condon: method to compare 105 and 101, to get baseline of where they were and how they do discourse; cannot stalk, but don't be bashful about checking in; differences between research agenda and teacher's agenda o Lacy: spoke to Patty and curious about how notion of self-placement might influence non-native students placed in 105; community college transfers aren't placed in 403 and then seen at 402, and their grasp and mastery of English language aren't where it is supposed to be for them to best succeed in course. How does the student who perhaps transfers to WSU from other universities or community colleges differ from those who receive suggested placements? o Condon: If you can think about that demographic information, you can get it through institutional researcher; student ID #s; short list of questions and ask students through survey • Tabitha: Distributed handout of plan of pedagogical underpinnings of UOG Placement Rubric o Condon: • Start with qualitative, then posit quantitative--> then suggest a survey • Ask instructors typical patterns noticing--> commonalities in specific demographics • Textual analysis through topic modeling • Focus on what you can get • Offering a ton of free labor • Carve something out of the pieces--> in order to arrive at a rubric o Lacy: Suggests survey of students and instructors about rubric outcomes o Mark: Coordinating schedules may be cause of lack of communication; Qualtrics --> send a link and then have students submit o Condon: Developing an instrument is a publication-worthy event • Mark: Role of hardware in composition o CCCC's position statement on Teaching, Learning, and Assessing Writing in Digital Environments • Last updated in 2004 o Role that they ascribe to reading environments o Looking at how these things hold up to 2016 and 2017 o Hardware-based pedagogy versus design-based pedagogy • Rhetorical obstacles • Moving from bridging digital divide to what we're doing with technology now o Opening Spaces - lenses for looking at writing technology • Good framework • Could go into Raul Sanchez, but it would go somewhere else o Looking at two statements and suggestions over what happens with position statement • Role of digital divide in hardware o Edie: Students maybe affected by fear, rather than access; distrust of tech o Mark and Lacy: distributed survey on choice and preference with regard to technology; kinds of things students have to navigate o Lacy: How do students see students see technology in the classroom? Curious how secondary ed pedagogies and practices have influenced how students feel about incorporating technology in the classroom • Lacy: Curious how slowly integrating students into notions of agonism and slowly integrating civic issues o Asking students to seek out additional perspectives that they are different or contrary to their own o Being knowledgeable of ow to participate in civil discourse • Disagree, but in a civil, productive way o Working with Dixie State o Better to look at department outcomes? Or broader lens (WPA)? o Condon: Outcomes are not directly rhetorical, yet what students produce are heavily rhetorical, more so than what the outcomes require • Institutions' sense of outcomes different from what students do • Different curriculum in play: curriculum in paper, what teacher does in classroom, and testing group ♣ In most situations, those don't line up well • You have these different outcomes, but how much congruence is there? If there is not much, what do you do?
• Edie: Used survey monkey to ask how peer review process went o Ultimately revising and polishing to give to all 101 instructors to distribute to students to see what they think about peer review o Going to use Qualtrics o Running into limitations of Survey Monkey • Eg. "Were the Peer Review Questions clear?" o Data project with 101 o Mark: Conducted face-to-face peer review? o Edie: One Peer review so far, face-to-face in AML with Word • Spent 8-10 minutes on Word tutorials o Mark: In methodology, clarify how peer review took place o Condon: We know peer review is good practice, but we know it's not really practiced well--> develop data o Edie: Learning to be comfortable with the uncomfortable parts of teaching • Next class: bring in assessment friendly assignments o Frame them in a way that makes sense--> no standard way o Contextualize them for us so we know what's going on o Bring copies for class
Sorry if my Class Log looks confusing! My bulleted system looks strange in HTML, so here's a link to download the document in Word: https://www.dropbox.com/s/xjahlzmx7fyk3ga/ENGLISH%20508%20CLASS%20LOG%209.22.16.docx?dl=0
September 29, 2016 Assessment Friendly Assignments Discussion-
Lacey- has posted a link to a Google Doc on the class blog. Ran into technical problems, and asked to wait to go.
Ed – gave handout on his assignment. Has questions about he purpose of the assignment. Concerned if anyone did a rhetorical analysis on the document. Assignment about rhetorical analysis. Open to feedback on the assignment. Also shared the rubric that his students designed for what they are to be graded on. Annie (A)– asks how the rubric? Works into the portfolio. Bill (B) – notices that there are missing points on the rubric, and whether the peer review & workshops are counted twice. Mark (M), Lacey (L) contribute to what should actually be in assignment descriptions for students. M – suggests to have a flow chart to clarify the steps of the assignment. Tabitha – suggests to have bullet points to describe each part of the assignment. M – suggests using Blackboard to help break up the assignments. Tabitha – suggests to have the component and due dates separated. Lacey – suggests adding portfolio objectives at the bottom of each page of the assignment descriptions. M – suggests contextualizing how points are allocated for the grade. B – suggests contextualizing the course objectives on the assignment sheet. Notes that the handout is not really performance based. Lacey – asks how it goes when students help develop the rubric in the class E – shares how students have to argue and justify what should be in the rubric. Shared how he adjusted the rubric after the class discussion to meet the objectives better. T – shares how she has tried to understand how get students to truly understand what rhetorical analysis really is. B – suggests changing the paragraph on description paragraph, praises Ed on including the little stuff out. Arranging all the things that have to appear in the final product in one location. Asks about the rubric. Thinks the weighting works. Thinks that a student would not understand how they would get 50 points. Asks when is peer review, and how much time between proposal and final. Wonders if students could give quality language for the peer review. Mentions that doing a collaborative rubric helps students to understand what they can actually do in writing. Allowing students to see what is achievable. The teacher should use descriptors that come from their work. Better to give students strengths to achieve than weaknesses to avoid. Asks what Ed would learn from the students. Ed – shares that he would learn what groups prioritize in their groups. M – notes that there is not much on a definition of research in the research & analysis category. A - asks if rubric is only for the one assignment. T – wonders what the benefits of having a blank chart, to duke out the point values and what the descriptors should be. This would help put the onus on the students to take notes and pay attention to do the assignment to lessen the need for the instructor take notes. B – suggests to double up the groups so that you get sets of words in constructing the criteria. M – asks if Ed asks if students use the rubric as well. Ed says ‘yes’. L – wonders if students would change their values for different assignments throughout the semester. Ed – shares that he did that. The weighting was standard throughout the semester other than the focus of the given assignment. B – identifies that the lack of change in the rubric related to homogeneity of genres. M & B talk about the change in genres
L – shares that her blog post is available. Hoping to get feedback on the format before she writes the formal description. Shares how she normally focus on the argument at the end of 101. Interested in discourse. Interested in how students would respond to arguments. Getting students to construct arguments based on what they have seen. Seeing how students synthesize the argument construction. Anna-Karin – asks about the assessment, as it seems a bit like a “needle-in-a-haystack” Curious whether introducing argument style would influence how students develop arguments. T –shares using WordCloud for what makes “good writing”. Then compares before & after WordClouds on the class time. Students able to change in vocabulary. B – Interesting way to check on concepts and retention of old concepts. Wonders what are students getting that L is not overtly teaching. Talks about the 3 types of curriculum – paper curriculum, taught curriculum and tested curriculum. Teachers need to look for what students are learning and what they are not doing. What are students that they might not be required to do. L – wonders how are students engaging with others that the framework might not address. Ed – gets that L values rhetorical listening rather than rhetorical reply. Maybe make it RL more explicit. B – states that maybe better to not explicitly state to see if students have learned the tasks. Wonders how many incremental assignments. L – 4 incremental assignments; middles assignments being in-class assignments; still trying to manage 6 major things. B – is the class online or physical L – could be either; has done some in the current class that she is teaching M – likes the idea of having collaborative assessments not just individual B – since online courses increase time-on-task in the online portal; physical class students can distinguish between classwork and homework. L – invites people to post comments
Annie – shares wanting to learn/try peer-review assignment. Considering having students include one the peer-review in their portfolio. Wants students to include one the best peer review in the portfolio as well to monitor the peer-review process. Wants 3 questions after the peer-review process for the author. AK- clarifies about 3 questions A – aware of conversational tone in the description M – asks whether the rubric needs to change from topic to topic B – wonders what students think about the rubric after using the rubric multiple times. T – shares her method of wanting to read all the documents that students include in the portfolio; gives students terminology to use when doing peer-reviews. For 1st drafts have students answer the questions on the prescriptive handout holistic, analytic, narrative & _____. Uses check plus, check or check minus to grade peer review. A – Thinking about not wanting to grade the peer-review. Has had her 102 group give verbal positive feedback to each other. Had students also include anonymous feedback about what they wished would have been in their pr feedback. L – mentioned having sent A links from a class a few summers ago. Have students write a letter to the reader reflecting on their own writing. Have students ask for 2 local and 2 global comments. B – thinking about having students get a heuristic rather than a rubric. A – wonders how students would react to having their work shown in class for good reviews vs. bad reviews. M – Eli Review does what A thinking about. Maybe share good/bad from other classes. T – Have a chart for good comments for rhetorical vs. usage so the class can agree on what kind of feedback the members’ value. B – using things that are modeling good practices, but avoid highlighting the “bad” things L – Nancy summers suggest having students say “I think I did __X__ really well in this draft” will give reviewer something to start from in their own comments. M – bringing a wide range of student comments. Have students rate comments from different classes. B – At Michigan, have high, middle, low portfolios for reviewing in workshop. Gives participants ability to see what was actually valued by reviewers.
Tabitha – misunderstood syllabus; gave 3 different assignments. Pre-reflection – have students determine how they would want to achieve the learning outcomes, borrowing from B’s presentation Post-reflection – have students reflect on how the students achieved the goals without the course goals The assignment would help T identify what she can share that students have learned in the course. A – wonders if a 3 - 5 page paper would freak students out B – shares that it will challenge them but help them to focus on the writing M – assignment would help the students prepare for their portfolio T – wondering if assignments could be included in the portfolio B – takes the view that the assignments could be used to help prepare students for the reflection in their portfolio. Asks whether giving students the pre-reflection before writing the post-reflection would allow the student to identify. Suggests framing the 2nd assignment to invite discourse rather than yes/no questions. M – framing the questions as ‘in what ways..’ will cause students to think about their writing more B – asking questions that at the end that students could actually answer L – thinking about how an incremental reflection to identify reflective comments M – shares that he had done that idea in 101 and that the comments were much stronger throughout the term. B – reflective piece most important piece in the portfolio. Reflective piece needs to be first in the portfolio because it sets the tone of the portfolio. Jeff Summers? Had ½ students put reflections at beginning in portfolio and ½ at end but reviewers read reflections first anyways. T – liked the comments from peers Ed - curious about how T’s previous students wrote reflection in different modes T – One student wrote an extended metaphor about running; one student wrote a script between student & teacher; realizing students want to impress her in their writing B – identifies that the assignments give students way to identify what works in the course. Students will be brutally honest
Mark – handout in color because all student handouts he gives are colored. Explains what is needed. Rating scale questions for Eli Review to give comments on IMRaD format for assignment. Identifies that he is still trying to figure out how to use peer review. L – impressed at how specific M is on his file names M – states that the file names help students & T – asks about how beneficial on each handout rather than just on the syllabus. M – shares that it is beneficial to reinforce on each assignment. B – not repeating defies human nature M – question: what should be in evaluation criteria? T – likes the chart, as it lays out the points M – likes creating an ecology of the assignment and review comments. Helps alleviate Blackboard issues B – change bullets into steps to accomplish M – PR is worth 10 – 20% of final grade. B – likes the evaluation of the quality Ed – wonders how peer review process, B – likes the vocabulary included in the handout; Research on testing found that if question 1 is in question 2, ½ students got question 1 wrong L – asks if students are given a handout on what to respond to M – wonders if anyone has ideas on using style manual
B – wondering what would be added; as the handout is a social contract T – talks about peer review as a scientific crucible. Wonders how much grace period M gives students on submissions. M – states if there is consistency in submissions there is no grace period; some consideration given in special circumstances; points given for completing peer reviews of all group members. A – wondering if peer review groups change or stay the same through the semester B – compliance of group; looking at comments to group members; how students teach other to do peer review
Anna-Karin – weekly blog/discussion board post for course in Hong Kong. Wanting to give students an opportunity to post questions from lecture or tutorial, and to practice writing in English. Things discussed related to whether should “grade” monitor student’s use of English in posts or just allow them to have “natural” improvement over the 13 week semester. Also how to incorporate the questions into the class whether in an exam or in class discussions. How to monitor the discussion forum – state good question or model good questions and/or responses in the classroom so students learn from each other.
For next week revise assignments and submit on the class blog.
-Karin: Changes to the assignment: Incorporated Lecture into the blog responses as a form of incentive -Tabitha: Streamlined and added due dates. This changed the way that students interact with the assignment since the information is visually accessible right at first. Added the evaluative criteria. So that it become clearer why student are using the reflective pieces to tie into the main prompt. It is so difficult to assess reflective writing. Included a rubric for reflective writing. Did they do it? Did they not do it? Did they kind of do it? Clarified the grading process so that students are aware of how their pre and post assignments are graded. The grades will be averaged. Point value for both pre and post are equivalent. Post: Less perfunctory more provocative. Did you do this? In what ways? Pre-Reflection goals add a column to the rubric that they build on throughout the semester (making it pre, during and post) and the during-course reflections add to the post-reflection. -Mark: With due dates is this separate enough from the last paper to leave time for reflection? -Bill: Right after this reflective session would be a great time to do course evaluations -Mark: In inviting students to be creative, you might want to invite students to choose their audience. Since this is one of the first assignments It could set a great tone and tap into rhetorical awareness. Maybe Change it up for the final one. -Bill: Move the length information down into the submission data, out of the rhetorical situation -Edie: How do you define offensive language? -Tabitha: that would be established prior to this assignment. Reiterating expectations from the syllabus.
Discussion on Evaluations Evaluations: We are putting them online this year for the first time. Students respond in text rather than filling in the bubbles. -Karin: With typing it’s harder to decipher what feedback came from what student -Edie: it’s nice to have the results compiled and given to you, rather than receiving a PDF looking at it as a whole eval. -Bill: We did this in Honors, it wasn’t the best. No one did it without class time being allotted for it. Had to do two, honors and English and then class specific evaluation. It takes a lot of time. -Lacy: Dixie State offered incentives to complete evaluations in a timely manner. How will it go without incentives? -Edie: this university is HUGE -Bill: It might be possible to tie it to student’s access to their grades
Mark: Included information between peer reviews. Situation peer review as a valuable and important genre of writing. Tying Peer Review into the portfolio outcomes. This works in the classroom but addressing the bold terms, and having students articulate the how this works in a peer review. Inviting students to problematize and define terms as part of the peer review process. This assignment sheet would not be a handout: it would be on Blackboard Peer Review is formative: and formative for a specific issue that has been clarified for each peer review Pg. 5 Incorporating an image into the prompt that show the different aspects of the ImRaD style Pg. 2 Steps rather than numbers, cleaning up the information -Bill: It is a better designed text Shows the point system to show students what is important and valuable -Tabitha: On pg. 3 Could you model and provide examples of different styles and modes of peer review. Contextualizing the peer review process. Similar to what you did with IMRaD. -Bill: If you thought of this as a shell, you could use it in any class. 101 may need scaffolding, but 201 and 301 may not. It makes a good heuristic for any writing class. -Edie: the significance of the color? -Mark: Students are able to locate colored handouts more easily and we have access to the colored paper. It’s possible to color code types of handouts, but it would take A LOT of planning. Images that have copyrights, how to cite that in assignment design. -Tabitha: you can do a role-over or make the image a hyperlink -Lacy: This looks designed to be a hand out. How does it play out with students using this and getting this done since it is not a handout. -Mark: this isn’t a peer review that I would do in class. The peer reviews that I do in class are much more tactile. i.e. The colored pencils and the identifying exercise. In class has directions that are less cumbersome. Simpler tasks for in class. -Mark has a module specifically for questions about assignments. Modelling students to complement each other during in class and then have that transfer to the peer review outside of class.
Edie-Marie: -Tabitha: Are you asking students to evaluate and summarize? Maybe separate those two terms to clarify that this summary is not objective listing of facts it is also evaluative. -Edie: objectivity is not very possible. Also these articles themselves are evaluative -Mark: I like the first lead-up, encouraging students to look at author credibility -Edie: students did it on their own, found that one author had his own patented vaccine. -Lacy: End goal is synthesis; do you encourage students to find sources that are opposing? How do you frame what type of articles they should be looking for? -Edie: the two articles are general and broad, dealing with historical aspects and one with expectations. Students can take it from there. Some students give a historical overview, some argued for reform. One student just took three random sources. But most students focused in on one aspect that interested them specifically. They don’t have to use scholarly sources, which is why author evolution is crucial. -Bill: How is this assessment friendly? -Edie: this is the first assignment, I have transfer students, some international, some traditional students, they have different backgrounds, this assignment is a norming process and provides a baseline. Baby Bib is a small assignment just like the one that is coming later in the semester. This Bibliography is based on completion and feedback is given. -Karin: Do they do anything with the sources that they find? -Edie: they summarize the sources for an annotated bibliography -Tabitha: Do they know how to do an annotated bibliography in a 201 class? Would modeling or breaking down the components be useful? -Edie: Showed them her own master’s annotated bibliography -Bill: If this is a baseline assignment and you want to know how much you teach them. You don’t want to provide them with this information before the assignment. -Edie: I don’t punish students for what they don’t know, but they need to address it. This is the breakdown of the unit. This is the 3 week. They have already done some reading. This is to address the smaller parts leading up to a larger synthesis essay. -Mark: Breaking information up into bullet points would increase readability. Frame it and then steps that are separate would make it more accessible. Two Weeks from beginning of assignment to a reviewable draft. -Edie: Some of the feedback I addressed in class: The balancing act, what to include in the handout and what to leave for in class instruction -Bill tells a kind of freaky Greek Myth to illustrate the balance between brevity and necessary information Lesson—Bill appreciates brevity -Tabitha: Is it beneficial to provide the criteria on which they will be assessed at this stage? -Edie: Doesn’t use a rubric, uses track changes and then has conversation -Mark: Designs rubric based on the instructions, are things absent or not?
Lacy: Posted the updates on her assignment on the shared doc form last week. Focused her changes on the start of the semester and end of semester analysis. Wanted an artifact that is used at beginning and end. Decided using March CNN Republicans discussions. Provides different perspectives and speakers. Asks students to look at the effectiveness of the arguments. It will let Lacy know where students are in there understanding of argument. Ted Cruz was much easier to analyze then Trump. Students can apply terms to dissect that argument. Added Word Count Expectation for responses to avoid the one word responses. End of Term: Going with Donald Trump. Now students have more experience engaging in the scholarly sphere and the public sphere to be able to engage in this more complex interview. Increased the word count for the end of the semester. Students should have more tools to respond to an argument by this time. -Lacy: Is the language of her prompt clear? She wants students to ask who is the speaker, with whom are they engaging and how? Students watch interviews in class, and write in class, but have time after class as well. So students are only responding to the very brief part of the interview. They can select which portion of the town hall meeting they respond to. -Mark: Could add question about the interview as well as the candidate. Their allegiances and how they deport themselves. -Bill: by specifying word minimums, you don’t know if you are achieving a goal of having students engage more deeply or if students are just meeting a specification.
Having students do directive self-placement. A good assessment need more than one sample. It can turn out to be a computer mediated interaction between students and institution. Here at WSU students sit for a placement and confirm to the results. Other Institutions students stay where they self-placed
No one right now is really measuring the reliability of directed self-placement. This is basically waiting for someone to write for a grant to look into this. People are thinking this is the way to go, but someone need to validate it.
Writing-Less placement: and indirect test of writing used for placement. Scores are used for placement into first year composition. -Lacy: Ivy League, doing away with ACT and SAT for admissions. How might this effect student success at an institution? -Bill: Harvard students don’t take a writing course. (interesting right?) Where Most students take the most writing? Community Colleges. At WSU we do placement, but we don’t really have places. Most students learn to write at community college, fewer go to large state universities. Such a tiny, tiny percent of people go to elite schools. Ivy League schools are on the peripheries of the writing instruction. When we look at what works, we don’t need to look at elite instructions. We need to look at places that represent the heart of the enterprise. Small private colleges are rich enough to do what they want. For smaller schools it is so much easier to design an assessment program, you only need 6 or 7 faculty members to get on board to make a change. Bill facilitated a workshop at a small private college and in an hour the faculty dreamed up a portfolio now in use. We, a bigger institution are the ones that have the numbers and are trying to do right by the students. So directive self-placement makes a lot of sense. Asking students to look at the information and place themselves. Data shows that most students place themselves conservatively rather than ambitiously. Most students are most concerned about their writing. To back up self-placement, you need a writing sample from students. If students place themselves conservatively and then the system tells them they can go higher how does that effect students? Validity started out as something you take the test-makers word for. IQ test and army testing. In 1950 it changed to validity as something assumed in the test, and that it dealt with how the test promoted the test takers wellbeing and experience. If the test was screwing things up for the test takers it wasn’t valid. If it helps it was valid. 1990s construct validity. If you are not assessing writing, then you are not assessing writing. Writing must be involved in writing assessment. Multiple choice cannot, does not assess writing. Seems obvious, right? Apparently it’s not! What are the outcomes of the assessment? 1998-1994 . Unified validity no single determination, validity is a process. Have to keep examining to see if validity if there. How does it fit into the curriculum how does it affect the test taker? There is a law that WSU cannot teach remedial writing. So Victor Villanuevea did away with it. There were always students that couldn’t pass 101. Had to bring basic writing back. Students were dropping out. Had to develop a rating system that could identify basic writers and make accurate placements.
Class Log: 10/20 Thursday, October 20, 2016 2:52 PM
• Bill distributed handouts from his conference in Indianapolis • Tom Angelo Handout: Survey from Chronicle of Higher Ed • Compare percentages between what we think and reality--> What do these percentages tell us? • If students want to learn more about things they’re interested in and want to be more knowledgeable and cultured, how are we promoting that in our classroom? • Angelo presentation helped to better frame pre-/post- reflection, wherein students are asked what obstacles they may encounter when approaching the goals (course or personal); students make a plan for overcoming obstacles; at the end, students evaluate the ways they overcame their obstacles • Find tasks that allow yourself, and your students, to find joy or wonder at least once a day --> helps to build momentum to finish tasks and see how they build into the final product • Assessment activity o Have students set their own goals and see how they blend in with course goals • Discussion from handout o Have 4 instead of 3 assignment • Giving students the agency to select their assignments can be empowering for them as they get to decide how to represent themselves and their writing • Signature strengths: strengths you rely on the most; asks students to identify those strengths and articulate them; the last essay should be a more traditional academic essay ♣ Kind of like backwards engineering ♣ Balance can be found in blending teaching with the things students are interested in ♣ References course built around "positive psychology" ♣ Can teach the basics without dictating a genre; can establish the genre while allowing the students to select the genre • Mark: Page 4 • How do we enable students to follow the Seven Research-based Guidelines for Assessment to Improve Teaching and Learning? • Encounter difficulties with the amount of project 4 • Bill: Plan with the end in mind; build in assignments that help students value their own learning • Edd: "Parrot Test" on Page 4 • Bill: Handout meant for those doing program-wide assessment Constructed alignment- Students come up with teaching philosophy; Bill encouraged students to think about what students would encounter then
• point out in draft syllabus where philosophy influenced development of syllabus • Weigle • Page 2: Questions to consider in designing assessment; Think about the WSU writing placement • Edie: Placement essays have to be mind-blowing to be exempt • Bill: Fits with our philosophy; tests our philosophy that it tests what they're going to be learning in English 101 • Semester doing assessment with Hong Kong Society of Accountants: develop an assessment system that would work within an accounting firm and would link accountants with university system to advance in ways they need to advance ♣ 4 levels: (5-7) partners, (20) managers, (199) senior accountants, (200) jr accountants ♣ Question: How do you handle promotion? • Peter principle: In a hierarchy, people will continue to be promoted until they reach of level of incompetence and cannot do the job. How do you avoid this? • Reality: communicate with one another and with clients; up and down communication involves some kind of writing, typically on paper so that it survives all the different links • Involves evaluating the person's language skills: give clear directions and relay good results--> low stakes, mostly institutional • They were promoting people who couldn't do the job; becomes an even bigger issue when face-saving is more significant in China than in the US • Testing language ability at different levels • Did not need much to test junior accountants moving to senior accountants • Answering questions of what we want to test and what we want to know entails looking at if people are qualified to do the job they're hired to do • How can the employees reach a level of competence that would help them get promoted? • The purpose of testing in placement is to decide whether to put them in 101, or 101+102 (100 numbers inconsequential) • Bill: It is impossible to test writing without also testing reading--> How central do you want reading to be in writing assessment? ♣ Take authentic tasks and build prompts from them • Lacy: How might integrity of the test be compromised if you consider the population? Differences between states in how they approach education. How do you still maintain credibility of exam while still giving students to opportunity to succeed • Bill: 1) presenting them with a task that is congruent with what they need to have a successful beginning in 101; give them something to read that's not overly complicated but still gives them room to make an argument and 2) avoid cultural discrimination; try not to alienate people through a question ♣ ACT test had a question about computing cab fare --> who is being privileged? ♣ Important that test takers have a point of reference; designing a question without a cultural or geographical bias is important • Establish a threshold that students will realistically confront in the first week or two from the course • Mark: Students draw from a common reading • Bill: Be careful not to raise the level of complexity too much and that there's just enough time to get enough handle on the issue involved and use some of that information in their essay • Bill: 2 terms get thrown around a lot in book • Performance assessment: tasks in environment they will be in • Authentic assessment: take work products they're doing and assess those • These do not make you stop learning to make you take test; they repurpose your learning environment so that your learning environment can be assessed • If takers are succeeding at an overly high rate, then there is something wrong with the test--> validation studies • Make adjustments • The hardest thing in writing assessment is coming up with prompts • WSU: 4 frames and 20 prompts per frame= 100 or more prompts that fit in frame ♣ Share the prompts ♣ ETS does not share prompts because of hours time and money that meet criteria of fairness for millions of test takers
• Analytical way of evaluating writing: Samples of kids from Done, NE ♣ Explain ways in which your life will be better than your parents' lives ♣ NE very traditional place; farm country; change happens slowly; people live frugally--> to ask these students to complete this task is asking them to criticize their parents' lives • Students on that prompt were half as competent as compared to the other thoughts • "Know the test takers well enough to not blow their minds" • Expert rater system: rater makes the judgment • Call for raters in the spring term: $30 to 35, depending on the tier • Training involves deciding whether a students can handle 101 in his/her own or if he/she needs assistance • Mark: Tier 1: Not making official designation of where student is going; put in Tier 2 for questionable; don't do much outside of putting them in different piles • Bill: Tier 1 is certain students are going to 101 ♣ Tier 2 gets settled by more experienced raters • First summer of reading is using experience in 101 to place students; 60% in Tier 1; Tier 2, who are higher paid, get fewer to deal with in determining placement • Bill: Hong Kong accounting company • Junior accountant: Evaluating the writing of the accountants to determine whether they get promoted or have to go through additional education • Senior accountant: demonstrate they can do memos and trying out things that managers write; managers evaluate • Mark: Digital placement tests? o Bill: system wouldn't change very much; just a logistics thing; IMOTE, not have to do it physically as part of ALIVE; doing remotely; no pressure in getting sample rated • Raters would get things that are easier to read • No need to change format of exam • Go to directed self placement eventually • Bill: Who uses this info that the test provides? o Lacy: the program and raters o Mark: other researchers and programs o Annie: the university o Anna-Karin: students--> Bill: Not so much o Lacy: state governments to note progress--> Bill: not so much; process fulfills one of seven ways institutions do these things responsibly o Bill: Registrars office; they program it into WSU and it affects every students' record; advisors use it; students are only aware because of what it allows them to enroll in o Edd: surprised state legislature doesn't evaluate in whether or not they're meeting standards in higher ed • Bill: high schools ask for data; different high schools across the area meet with program twice a year; teachers want to know how their students are doing; all state wants to know is what we're doing fits with what they want us to do • Bill: If we had a lot of time, would we still do English placement test? Or something better? o As direct assessment in timed writing genre, our test is the gold standard • Students do it physically as part of ALIVE, placements are known, and then students enroll • Logistics easier when you conduct placement yourself • Outsourcing (to COMPASS or ERATER) gives up validity o University of Michigan • Norming session in the morning, then by 3 or 3:30, the group's blue books are rated so that next morning, students can sit down with advisor to enroll • Lacy: All seniors had to do professional writing portfolio in her old school; Do any universities determine enrollment by senior portfolio? Is it simply timed assessment? o Bill: No, and rightly so. • Daughter had to do showcase portfolio that had certifications of things student has done, but not actual work
• When portfolio in Michigan installed, asked for 3-5 pieces of writing: ♣ One most representative of writing ability ♣ One from a class other than English ♣ One student identified as best or favorite piece with reflection on why • If schools just took portfolios from high schools, universities would not know what they're dealing with--> How would they rate that? ♣ Easier to assess Michigan's context, since that is where students want to succeed • With those three samples, students' work could be judged on whether they would succeed in English 125 • If students couldn't come up with one of the three, they must account for it in the reflective piece ♣ 81% managed to provide writing from class other than English • Bill: Every exit is an entry ♣ Portfolio to get out of high school is different from portfolio to get into college ♣ Good that HS prepares students with portfolio ♣ Pomeroy: community members coordinate with teachers to rate HS portfolios-- engages the community with the schools in profitable ways and generates support by showing them the great work students are doing • Basic questions to consider in assessment on page 2 • Terms in Page 48 on Test Usefulness are helpful • Weigle is only comprehensive source of writing assessment that takes into TESOL writing assessment and does best job of explaining writing as a construct o Give Pages 2 and 48 "extra love" • Edie: Validity o Language and power with students; power of students choosing o Liberate students and their writing, but also help students write in the language of power o Is construct validity sitting in that same camp or not really? • Bill: construct validity asks of the test; questions of validity ask of test; does test measure what it claims it does? ♣ If you're not testing writing, you're not testing writing; if you don't have your writers write, you're not having takers give writing to be assessed--> got to have them write something to test that construct ♣ Political economy conspires to ignore that, but sometimes it comes back • California: Legislature declared that if ETS didn't include writing sample, they would not use SAT • Result: SAT2 had writing test instead of lose California's patronage ♣ ACT and SAT writing parts are now optional ♣ Bill: With different tests, what are they testing? • CC use COMPASS (ACT) and (ETS) ERATER • COMPASS is editing skill • ERATER collects 20 minute sample (utmost automated essay scoring can handle) on tightly controlled and delineated topic • Computer looks for 42 linguistic features associated with what human raters associate with writing • Bill: Moving WSU placement online o Les Perlman found that you could rate samples from across the room: the correlation between the length of the sample and the score was 90% o With online, students can take as much time as they want o Les figured out when correlation between length and score reaches 50% (correlation goes away), if you remove time constraint and let people write as long as they want, is 4 hours • Correlation still exists with WSU placement exam • If we move test online, we could do more interesting things with it and the correlation between length and score would go away, which would be good • Bill: with validity, is it testing what it says it tests; other questions: o Is the test adequate to the results that it gives? • If CC, the writing placement exam may or not be able to place students with that so fine-toothed a comp • WSU has only 2 placements ♣ If students place into 098, they will not get to take 101 (2.5 years of English for a 2-year degree) • Test has to be able to make high-stakes decisions • Consequential validity: is the test able to engage the consequences of the assessment? • Validity: Keep checking; validity is not a things but an ongoing process ♣ Takers, context change ♣ Every 2 years: validity studies; biennial portfolio report
• Bill: Reinventing 100 • Victor came in as Director of Comp and did away with 100; went 4 years without placing students in 100 o Complication: students tried their hardest and were still failing 101 o To appease legislature, made 100 look like a college course o Worked though curriculum stuff and went back and pulled blue books of old 100 students and train new raters to understand 100 placement o Edie: Most of the time when I think blue book exam needs to be in 100, as Tier 2 reader, it's more or less imaginary because I never taught 100 (took class to, but never have) • Bill: Bottom 10% at University of Michigan (still high achievers) • Frustration of teachers: spending a lot of time making students' writing a little bit better • Bill knew what basic writing is and what word salad is • Job was to push these students up • Edie: Fall schedule has less than 60 students enrolled in 100 o Bill: sliding scale; create a new category of who belongs there o Over 5-6 years, for a time under Lane Rowlands, the goal was to raise caliber of students • Complications: land grant is meant to expand access • Elson and budged cuts came and needed to raise tuition in spite of legislature ♣ Used to give 54% and now only 26%--erodes functionality; raised tuition to compensate • If it costs $8,000 dollars (state and student each pay $4,000), you have to educate student at half price • If you raise tuition, you can get more students ($6,000 from student, $2,000 from legislature) and losing legislature funding is inconsequential o Sam Smith--Lane Rowlands--Elson Floyd • Scale shifted: More college ready to less college ready ♣ System has to be able to accommodate that ♣ Cannot just declare test is valid; in a few years later, you may have a different set of test takers • Bill: Element of writing assessment that is about language competencies • Different from other places in US o Focus on competency and leave language acquisition to other people • Anna-Karin: students in HK start learning English in elementary o Students grew up speaking Cantonese (no writing) and written form of Chinese based off of Mandarin form • By the time they start learning English, it's their 3rd language o By 1974, there had to be equal opportunities to language: Chinese or English medium • Power dynamic of English in HK • 1997: students have to be trilingual (Mandarin, Cantonese, and English) o Mandatory exit exam determines which universities students go to • Community college more expensive that university, as state doesn't pay for students; students pay for themselves o If students came from English-medium school, their English is fairly competent; if Chinese-medium school, there were issues in writing and speaking • English 1000 (equivalent to EN 100) or 1500 (equivalent to EN 101) • Bill: construct is language acquisition and not writing ability ♣ In HK, it's language acquisition as writing as after thought o Teach students to think and express argument in Western linear progression and process o Bill: We have developed, because of print in 16th century; once people began to have printed text, you no longer had to translate to learn but could buy it and then could compare and teach texts--write commentary and publish • Linear: knock down someone who's written before; you don't do this in Asian cultures; Asian thought is venerated-- honor that and say what you want to say; accommodate what people before have said into what you want to say (cultural competency) o Those in English-medium schools already accustomed to
o Bill: biggest influence to international assessment is colonialism--sun never set on British empire, wherever you went in the world, people spoke English for various purposes • For colonial society to work, you had to teach native people to use English well; first wave ♣ Some places work spookily well like India and adequately well like HK • For EU, it became possible and normal for a person in one country to go to university in another--establish competency levels; second wave ♣ Bologna accords: developed in 80s and 90s to meet assessment needs--assessment all about language acquisition; write the language enough to cope and succeed in the university ♣ Whole system that turned up the heat on assessing language acquisition--tests outside of US aim at primarily o In colonial system, you learned English to succeed; if you didn't you remained on lower levels of economy • Anna-Karin: people aren't on their phones to the same degree as in HK-- possibly due to population density? • Bill: Weigle book is useful because when encountering non-native speakers, it helps you know how to approach it o Lets you know why international students don't care about the same thing you care about • Their focus has been language acquisition, while teachers ask them to do rhetoric • Lacy: Germanic and Asian languages differ • If you took someone from Germany and someone from Hong Kong, how do we account for differences in language (German closer to English). Do we put them in the same class? o Bill: Germanic language less likely to be EN 105, as they study and use language with native speakers • Edie: Friend missed learning Russian language when wall fell • Bill: HK, China, Japan, Korea have Native English Teachers; not TESOL trained but train students to speak with native speakers • Students coming from Asian languages didn't grow up with verb tenses or articles • Tabitha: understand variance between ELLs from different regions and even among different countries o Know their background with English to see how they approach writing assessment • Bill: they're not testing construct on ways that students will need it, but what we want them to know o Asvar, past graduate student, last true Marxist on Earth; Bangladeshi, brilliant, smoker o Facebook: Anis Rahman becomes Bill's Facebook friend and asks questions about university; wanted to get PhD from first world country, looking to immigrate; Bill coached him through TOEFL to get TESOL masters degree and second in rhetoric • Bill: The question will be- How long will the US hold out in building other language competency into their writing assessment? Probably not long o World is growing more mobile; communicating across thresholds is easier o No longer geographically isolated o 60% of graduate students in STEM fields are international students o WSU Education Program being maintained by international students o Oil rich Middle Eastern countries get paid to go to school in US o As the richest country in the world, why can't we do something like that? • If questions remain for Weigle, keep them in mind, but if not, we will move on to Elliot • Tabitha: How about Assessment Projects? • Bill: Next week; first hour work collaboratively to discuss progress of Assessment Projects; then Elliot, which may spill over into two sessions (with Very Like a Whale) o Middle of November: Have conference-style write-up (you get as far as you get); doing something real-world takes more than a semester and spills over a lot; engage with methodology and new topics; Bill willing to keep working on it through his retirement o No deficit for not having brought Assessment Project to closure-- bring it up as possibility
• Announcements ○ Write-up due on November 10 § 7.5-10 pages (conference page length) § Should be written up as a conference paper, even if the project is still going on § Post to class blog § Will set up peer review after that § Should still keep your reviewers informed • Overview of final assignments ○ Lacy: ○ Annie: Appealing Junior Writing Portfolio decision § Looking for more opportunities for additional status § Only students placed in 1 credit courses appealed § Few numbers/data is making it difficult to draw specific conclusions § Examining the process of assessing portfolio § 16 artifacts for analysis may indicate some promising insight § Can go into greater detail on factors of success based on observations § Research "regression analysis" § Can look into Haswell's work, he recommends following-up with students and the writing they do after graduation □ Could go through Alumni Association □ IR § Could develop a system to test over time § Current research indicates students trust the portfolio process § All students have been assessed twice (at both Tier I and Tier II) § Should include more detail about the nature of the junior writing portfolio § Look at the outcomes discussed in Beyond Outcomes and how a program can explain its purpose and objectives more clearly § Could foreground it in the findings that WSU has the more "sophisticated" portfolio system ○ Tabitha: UOG English Placement Rubric § Reconstruction of project § Looks at population, praxis, pedagogy, and purpose § Context: □ Population ® Race demographics □ Planning and Purpose ® Referenced Zimmerman and the theories that informed MA thesis and creation of the rubric ® Wrigel to assess limitations of assessment ® Looks at limitation of research □ Pedagogy ® Millman and Zimmerman ® Why they constructed the rubric in the way they did ® Zimmerman: students are unprepared; correction is important; ® Millman: we need to orient students to American/Western discourse and practices ◊ Guide for Correcting Composition □ Practices ® Moving away from the deficit model ® Current model stresses the deficit model, even at greater levels of proficiency ® Deficit model pathologizes ◊ Labels both writing and writer as deficient ◊ Can occur across disciplines § May frame conclusions as "frames to consider" during revamp of program and outcomes □ Ties into the "so what" of the piece/why it's important § Could also look at how much change (and much has not) since the last iteration in 1991
○ Anna-Karen § Has 5 complete sets of lab reports § Recipient of lowest score had difficulty with paragraph construction, making it difficult to parse through at times § Examining methodologies for examining science writing □ Book by Biebers □ Articles referenced come more from bio-chemistry □ Thinking about looking at moves of bio-chemistry articles § Consider the TA role in evaluating the writing □ Looking for close comparisons to the writing of the discipline § Studies show that those who had several surface level mistakes, did not receive as great of credit on larger, global aspects of writing § Could look into genre expectations and how those expectations (held by professors and TA's) influence the evaluation of writer's lab report content § Can possible access more partial (but not full) sets of data □ Will help with "might be" statements □ 10 complete sets could better indicate existing patterns § Could do this on a larger, more official scale next semester ○ Edd: Service Learning Assessment § Has lit review done § Looking at instances at WSU § Identifying issues within assessments methods of service learning § Assessment model is based on quantitative growth § Formal write-ups (provided by students) are short reflections □ Relies on a professor's qualitative analysis on quantitative data § Been working with Center for Civic Engagement § Moving towards more of a constructivist approach □ Discourse analysis: being able to understand the conventions of community they're writing for during instances of service learning □ Will also emphasize the importance of qualitative research during service learning assessments, rather than relying so heavily on quantitative research § Look at how they're selecting ongoing service learning activities □ How students integrate these projects into writing outside of the classroom □ Examine process for selecting projects and organizations for service § Would like feedback on structure □ Has lit review □ Examine the shortcomings of WSU □ Propose methods for improvement
• Reading Discussion ○ Companions for On a Scale § O'Neil, Moore, and Huette § Robert Connors's works (historian in the field) § Tom Miller (historian in the field); work is more ideological; more interesting but perhaps less reliable ○ Text begins with NCTE rubric § The field had already been going for 30 years ○ Some background/context § Pre-printing press was meant to record something that had been said aloud § Post-printing press, writing was a site of rhetoric □ Rhetoric had to account for the differences between speaking and writing □ Greater rates of literacy § Writers (even into the 19th century) wrote as if to be read aloud □ EE Cummings was one of the first writers to write for the written page, not to be read aloud □ Difference between oral and written communication and their rhetorical expectations can be English took on text production, while communication took on the performance § Rhetoric began finding its ways into other disciplines after the invention of the printing press § More recent theory pays attention to rhetoric in its written forms § Speech has a limited distribution, while writing can extend over more temporal and physical contexts ○ Expanding enrollment § Students didn't share the same preparations as they had before; had different experiences and approaches to discourse § Created a deficit model with students § Harvard referenced as an example of entrance writing exams § Teachers were removed from most of the assessment process § Referenced Mismanagement of Man □ First IQ test and its effects when picked up by Stanford □ Not initially thought of as a fixed quantity; yet as a means for identifying those who may need extra help □ Revised applications removed the upward mobility aspect of it
○ NCTE was expanded out of MLA § MLA did not take the teaching of writing seriously, so NCTE was developed to address this apprehension § Many rubrics started appearing around the development of NCTE ○ California testing system § Subject A Exam § Test would determine preparedness for entrance into the UC system § Tests were developed later to more directly assess writing § More direct writing exams (like WSU writing placement exam) emerged from this ○ Transaction writing began to take the place the role of creative writing in the college classroom § Creative focus had greater problems fitting within an industrialized society ○ Discussing differences between writing-based (more common in Europe) and standardized assessment (more common in the US) ○ Assessment's effects on writing: students had to write as if their text was going to be read individually, rather than read aloud like in the past ○ WWII § Two impacts □ Perfection of the system: began with Army IQ test; used tests to rank individuals; resulted with determining who should be placed where based on qualities demonstrated in tests ® Proved effective ® Turned into the SAT □ After effect: GI Bill; created a moment like 1870s moment at Harvard; at the end of the war, millions of men came home wanting to attend college with their GI bill; many used it as a support to go to college, rather than vocational training; in 1951, Michigan State was a 2 year community college; by 1955, Michigan State was a 4 year state university (land grant); small land grant institutions turned into larger universities because more people had the support to attend college § Learned that assessments from WWII were efficient § Trouble began when they started basing exams off of material for those who had historically been in college: white, upper-middle class men □ Assessments, then, were designed for this demographic, thus preventing those outside these groups from doing as well as possible □ Did not allow for additional experiences to be considered § Open admissions began to combat this privilege □ Allowed those from previously excluded populations to enroll in academic programs
○ Open class discussion of text § Tabitha: how is assessment motivated by consumerism? □ Before WWII, larger schools had maybe 3,000-4,000 students □ When capacity is increased, that means an increase in scale □ The development/design of standardized test examined ideas that cannot really be assessed by an exam (designing learning strategies, asking for help, critical thinking, etc.) § Annie: what are the stakes for students testing in middle/high school? □ Students are required to participate but have no incentive to do so □ No stakes are assigned to students □ Even though students prepped for exams nearly half of the year, assessments had no direct correlation with improvement § Lacy: what are the main differences between NCLB and Common Core? What can we expect from students raised in a Common Core setting? □ Common core better focuses on demonstrating how skills are transferable to college or professional settings □ NCLB threatened schools with funding by putting all students on the same scale □ Saw a greater emergence of charter schools ○ Value of On a Scale: exposes both the advantages and disadvantages of assessment § Began as a gateway § Turned into a barrier for those wanting to get in § Reveals political nature and the intentions of standardized tests
Starts with how did Very like a Whale go? Book is a good reference that will not likely go out of date.
Assessment book would have been too technical to publish in the mid-90’s. Book helps to bridge the divide between writing administrators and research Authors did a good job with questions Book is more usable than readable.
Reading introductions? Edie admits that she usually reads intros but missed the Hamlet reference to the title of the book.
Trope? Why did they choose to use this term? Trope being used in the Aristoleian sense as well as in the way Toulmin – related to things that may become a warrant. It’s ideological common place.
DFA framework? How does this design for assessment (DFA) framework work throughout the book. Designing first makes it easier to know what kinds of outcomes that the program wants to achieve. Outcomes might range from finding more sections, identifying if students are making progress. Example from WSU – Haswell identified two environmental prompts to see if there was change in the students who had the two prompts. The raters were to identify which prompts were better – a little better or a lot better. 79% essays showed that students at portfolio did a little better than at entry. At a program level the individual student doesn’t matter, in a longitudinal frame the program wants to show that students are getting better and better. Having a blue book in the portfolio helps show that students are getting better over time. You need to show that the writing program is worth the money involved. Another WSU example, is looking at the 29% of students placed into 102, are still at the bottom 10% at mid-career. 90% of the students from 102 were no longer in the bottom 10%. Portfolio system was designed to NSF grant for Vancouver campus for writing assessment of engineering students. Setting up syllabi at the beginning to design for transfer to engineering student writing of lab reports. Having everyday evaluation after each workshop as part of the DFA sounds very similar to backward design. However, DFA is more about identifying a more fluid end product unlike backward design for an assessment.
WAC metaphor from physics – particle exists a thing, or as a wave (in motion). A writing program as a particle you will find the people/location. If looking at writing programs in motion it’s not easy to find its location. Need to identify how the program is working in motion not in a static location.
WPA – assessment research is not like the research in the humanities. Need to rethink how Often DFA happens mid-way in the process, not at the beginning. WSU started a project shortly after Bill’s arrival looking at the critical thinking involved in the writing. Two students identified a rubric to evaluate the critical thinking involved in the portfolio writing. Looking at the correlation between critical thinking and problem solving. The students identified a rating scale to help promote high reliability. Then the rubric was considered as useful to train faculty to incorporate critical thinking into their syllabi.
How is student success defined? How is this program working? Research paradigm based around the null hypothesis. If you’re looking for success and you find success, it could be that you are just confirming what you. Starting with a null hypothesis helps you disprove the opposite of what you are looking for.
Universities are hierarchies.
p. 56 scoring sheet for technical writing Scoring sheet as part of a survey design identifying whether e-portfolios are working. What would the data from a holistic score like this tell? Helps you identify what is working and to what extent that is working. p.57 shows how the individual parts lead to the whole assessment. As a wpa it’s good to have an outside evaluation can allow you to get the funding you desire.
1st week impromptu writing is a bad scenario for everyone involved (teachers and students). Very Like a Whale helps provide good reasons to end this kind of scenario.
iMoat example from Irv Peckham, at Louisiana State University. He designed a way to test
p.81 validation: evidence & categories The thing you want people to do is what needs to be valid. Validity depends on the instruments, and it also depends on what you want it do. The construct needs to be relevant to the place where the Figure 3.1 p. 75, the writing construct in general is the area that the writing construct is happening.
Debates between Peter Elbow and Barthalmey(?) Writerly knowledge, group characteristics, as well as the biological characteristics that lead into the writing construct. Outcomes for all students, as well as outcomes for individual students. Writing construct design is a boundary to identify how students are to perform. Validity is decisions that can be trusted.
Writing in a portfolio allows you see a student’s writing within the context of why it was written. p. 81 the construct needs to
p. 85 diagram collecting big data, you need to ask the same questions about subsets of the data (disaggregation). Is the construct fair to all subsets of the data set? At WSU – performance by major animal science majors do well in the portfolio, international business students do not. Disaggregation allows you identify if there are outside influences to the difference between writing ability between animal science and the international business major.
p. 22 quote from Bill in the text An instrument you use to test a writing construct The writing construct should be a rich Standardized test results are a poor representation of actual writing ability. A 20 minute sample of writing use a miniscule amount of the writing process. NAEP tests represent more of the construct, but is still a limited representation of the writing construct. The writing portfolio is a much richer representation of the construct because it allows students to show more their ability involved in the writing process. The first two tiers in Bill’s handout at the beginning of the term, stop learning for a test. The portfolio looks at all the products a student has written over the course of a semester. Correlation is not causation.
Changes in higher education have worked to eliminate “weeder” courses eliminating the pool of students. The pedagogy is changing to allow for accessibility for students rather than elimination of students.
WSU example – All university writing committee wanted a mid-career writing test since that was what was being used at entry. CLA – rich writing assessment that does a pretty good job, but it doesn’t reach all the constructs of what is expected of a student at graduation because it gives the topic to the students, only focuses on the specific readings provided.
If an assessment is designed to stop learning to take the test, it is not very valid.
AP tests – Where do AP and IB fall in the rich vs. artificial assessments? Start with AP – AP tests are geared towards an elitist group of students. The exam is similar to CLA or iMote exams. Universities are willing to give credit for the exam but may not exempt the courses.
AP exams train for a specific task of writing, but may not give the full picture of the writing constructs. Arguments for decisions are based constructs. Bill was one of the 1st reviewers of the book. Ended up being a 2nd reviewer. This book would be a good resource as a WPA. Design issues often confronted in WPA roles, are addressed in the questions at the end of each chapter. Five years is a minimum for identifying change in a students writing ability. The WPA wants to show that program is getting max “bang” for the cost to offer the courses. How is a writing program characterized by the department offering it as well those outside the department? If faculty senate are pleased with the writing program, the program is safe. Questions at end of Ch. 1, are good questions for a writing program consultant. The questions are more heuristic than questions that you can get answers to at one setting.
WPAs should network with others to find out what other programs are doing.
p. 86 is a good identifier for evaluation on a yearly basis.
Construct modeling – prerequisite for relational modeling – developing a set of exemplars. Students need to know what a good piece of writing looks like. How do you get students know what it is? Getting students to identify what was good in the exemplars. Rubric model demonstrated by Bill is a good way for getting students to know good writing looks like. Telling students what good writing is, is not as useful as having students explore/identify what good writing is.
Relational modeling – not possible to teach the whole construct. Desire to teach what will be most representative of the whole. Similar to regression analysis of the parts of the whole construct.
A regression analysis allows you to compare several outcomes of a construct outcome to each other. What elements are most likely to cause improvement? Regression allows you determine whether an element is useful to the larger outcome. Are there things that you spend time on in class that do not impact students’ final outcome.
Providing students with ways to access to perform the tasks being assigned. Separate out the things that you have to do with the thing you can change. Give students a level of expectation to access the material.
Identify constructs for a course that will give students an idea of what they need to do the assignment.
Critical thinking: Using the language for goals and outcomes in the assignment, language and prompts. Label it when they do it, so they can identify it within themselves. Conclusion is a great place to emphasize critical thinking. It should not be a simple repeat of facts already learned. Should say implications and consequences that you weren’t justified to say before the research. Studies show that students are weakest at articulating their own opinions. Difficulty going beyond reiteration of facts. Edie-Marie—do students not have the confidence to say “I am doing critical thinking” Mark: Creating a research place: The CRS, John Swales Edie-Marie: A teacher has to train themselves to use the language of the outcomes statement, seems somewhat unnatural. What do we have at our disposal to have them thinking about outcomes Edd: What is a good reflection question, that gets students engaged in reflection effectively. Lacy: reader letters: Tell me what you think you do well. Global revision and local revision that you want feedback on. Gives a turn toward revision. Giving students a stake in the situation. Something that is actionable in student’s life and process. (No paper is finished it just do. Edie-Marie—ethnography, culture of college writing. Were your expectations met or not, what are the actual expectations and how do those match up with what you anticipated. Getting students to articulate their understanding of the genres. Self-assessment, collaboratively made rubrics to help students identify how they are engaging with learning. Reelection is designed to give students a voice, if it doesn’t it’s not effective reflection. Having students articulate what grade they think they have earned in the class. Students usually placed low. Grades extrusive motivation, looking to internalize motivation.
Mark: Asking questions about how students see genres Critical Think: John Bean, chapter on critical thinking, supporting claims. Reflection also means it’s more difficult to get the paper done just the night before. It’s a cheap trick but it works. No one has done a serious date driven research on reflection. Been doing it since the 90’s. We have lots of “lore” to back up its effectiveness, but less research. Numbers and data vs. theory and nuances. Administratively deans need data so they can justify continuing funding. When you put together information you always keep in mind the audience, presenting learning in a way your audience needs to hear. Presenting data numerically and effectively is a great way to distinguish yourself, and get funding for your project. A program is built over the years by data. Mark: How did you generate that data? Bill: Look at the what matters. What is something that a dean or administrator might care about. Look at the strategic plans/goals that the university has outlined and use that language. For a writing program—student contact is important. Setting up English 103, University 302, made the writing center generate credit hours. Luckily—this is not that hard. You’ve just got to do it. Your first goal has to be improvement—and measure for it. All the stuff you report to a dean is stuff you need to know anyway. Things that have changed: We used to think that the status quo would be preserved. Positions are not simply going to be replaced any longer. Things are more fluid; everyone needs to be in a positions of justifying what they do. There are across the board budget cuts, the money is then in a central pot, ready for redistribution. You’ve got to look like you know what you’re doing. Evaluation vs. Assessment Bill: Evaluation is a universe in which assessment is a planet. Assessment is more narrow. Evaluation inquires to find the strengths and weakness. Appreciative Inquiry is the first move. In assessment eventually you get to ranking, but it’s not where you start. A, B, C, No credit would be great, plus and minus is too much. We often grade on the notion of limited good. That giving too many A’s somehow devalues them. Assessment is aimed at ranking. Who gets what at the end of the class. Relative performance between one student and another. Evaluation is aimed at things much bigger. Assessment is reductive and Evaluation is generative. Evaluation isn’t closed it’s open. It provides information that produces a line of action. There are real consequences to scores. Too low and you’ll discourage them right out of college. Assessment is often done rather than evaluation because it is more cost effective. In a well-designed assessment there is hardly any cheating. It means students are trusting the process and investing in their work. The grades are institutionalized requirements, and that is a fair compromise. There is a way to go before eliminating grades. We know that external motivators don’t produce the kind of learners that we want to see running the world, but intrinsic motivation does. Video games: the better you are the longer you get to play. Assessment has just the opposite effect. We use assessment to shut people out of the next step.
How do we/should we stop students from translating any feedback that is given to them via rubrics into grades. Rubrics that are produce outside of the learning process and then are imposed are reductive and detrimental. You have to take something that is locally developed and then tweak it to fit the context for which it is applied. It takes time and expertise to develop and effect rubric. Develop a rubric with your class. How do we define what good writing is as a construct? What is okay writing? What is crappy writing? That way the language comes from the community. Mark: What do you think about page number and word count? Bill—I give a page range, a piece of writing needs to be as long as it needs to be. I don’t think that you can do it in less than four, six would be great. It’s more of a guideline. What we do as teachers should be in response to our students. Mark: reflection is the only component of the portfolio that doesn’t have a unit dedicated to it. What do you think of a segment just dedicated to drafting reflection? Bill—Seems great, why not do this during dead week. You could have them bring in a draft and then ask them what part of it seems like and argument for a grade? Take it out and engage more deeply. Mark has created an assignment on blackboard with unlimited submission. They can put things into those slots as they go. Initial drafts etc.
This comment has been removed by the author.
ReplyDelete8/25/16 Part 1
ReplyDeleteIntroductions: the story behind our names
Recounted as best I could after I ended up signing up for today. So, seeing if Bill’s theory of its memorable-ness is sound.
Bill used to be Billy and when he was younger and at camp there was another Billy with an awesome middle name of Bull (to which I thought, “Really? His parents named him Billy Bull?”). With such an awesome middle name Bill wanted to know what his middle name was. To his dismay, he discovered his middle name was Francis, a girl’s name.
Edie-Marie has two hyphenated first names because her Mom doesn’t believe in giving middle names to girls (abbreviated version).
Lacy is Lacy because of a compromise. Her dad really wanted a boy after two girls and had a name picked out: Robert Allen. This was transformed to Roberta once it was determined that Lacy was not a boy. Lacy’s Mom wanted Latisha after one of her Mom’s dear friends growing up. Both parents felt mutual disdain for the other’s name choice. Lacy was the compromise for which Lacy is grateful (though I think she has the stuff to rock either of those names should that have happened )
Baby boy Triana is named Mark because his parents could not come to a compromise. His birth certificate reads Baby boy Triana as a result. Mark’s brother named him some time later (one or two weeks? I can’t remember).
Tabitha was also not given a middle name from conservative parents (like Edie-Marie’s) and is named after a character in the bible.
Anna Karen – Karen – has two first names from a Swedish tradition but only her Mom would ever call her Anna Karen. Work and life is Karen, teaching ELL students is Anna. This became a problem when her students tried to get her attention when they saw Karen out and about and were shouting “Anna!” for quite a while before it registered.
Sarah Blume (apologies if I have just misspelled both your names) is named after a biblical character as well. She’s not sure what that’s about because Sarah is known for laughing at God (which I think is kind of badass). Her last name pronounced ‘bloom’ is commonly mispronounced ‘Blum’ or ‘plum’.
What this class is about:
Know the theory and writing around assessment, student assessment, program assessment, self-assessment -- Assessment to drive learning
Coursework – (also on the syllabus)
Assessment project that is real
A book review – recent book in the field on assessment
Assessment-friendly assignments for our own teaching
Each week after class post a discussion question (by end of Friday) http://engl508f2016.blogspot.com/
8/25/2016 Part 2
ReplyDeleteHandout: Performance assessment v. traditional grading
Coffee making example – we either have coffee or we don’t at the end
Class blog, either you posted or you didn’t, for example – but we get to set the agenda so take the space and make it what we want to make it
Several of the assessment friendly assignments will be performance based
Ex) taking points off when an assignment is late and the idea that timeliness is a life skill and just doing the danged thing is often more than half the battle
So, missed deadlines will equal missed performance points
Q&A for example means that students sort themselves out
When it comes to grading the best way to approach it is to think outside the box
Second handout – excerpt from The Mismeasure of the Man by Steven J. Gould – count the E’s on page 220
Answers: 183, 237, 228, 244, 251, 252
There is not really a way for us to know for sure how many E’s are on page 220 – there are limits to our cognitive abilities – how does that affect us? How do we process the idea that there are things we will never know while trying to know more?
How we counted, obviously impacts the count as we all got different numbers. The take away is that even in something as concrete as how many Es are on page 220 there are limits and we need to stay humble.
Always ask, what are the limitations?
With AES (from the readings) there are serious limitations and yet there are sweeping actions made, real consequences from using the AES.
SAT scores example. Some use SAT scores as placements even when SAT even says it’s not for placement. And, there is direct correlation to SAT scores and income ($$=higher scores)
We could place students by how many bathrooms they had in their home this way
There is a good deal of literature that suggests those students placed in remedial English courses are much less likely to complete their degree
Questions on the syllabus…
Assessment knowledge is used in the workplace of education, period. …Knowledge to fend off bad assessment is as important as creating and improving assessment.
With the WPA outcomes, pay attention to the difference between what and how well
Outcomes are what we expect our students to be able to do, not how well
In forming the WPA they realized that there is no way students will have write to the same standards. They have diverse backgrounds and are starting from different places.
Ex) students from Arkansas poorer backgrounds and Michigan better off economically – to expect them to write at the same level isn’t unrealistic, unfair
But they can be working toward/with the same ‘what’
Outcomes v. Common Core
Outcomes was always intended to be a living document. That we will learn more and adjust to the needs of students as they evolve.
Standards seem to change less than outcomes.
About every 10 years tools of assessment, outcomes, standards, need to be revised!
In the 90s tech was viewed as a tool for writing versus a way for communication, issues of use, access, etc. But then by 2008 access was a whole different situation and there were multimodal projects and the internet as new and multiple ways to communicate.
9/1/2016
ReplyDeleteApologies for the delay - I've been having computer crashing issues.
Pre-class – my clarifying question
Start with Guba and Lincoln, application and how things work themselves out
Two different epistemologies (somewhat of a false dichotomy)
When and what to choose for methodology
Positivism Constructivism
Only 1 Truth with a capital T truths
Observer-external Observer included
Methodologies:
Scientific method qualitative, interviews and such
Qualitative measurements Context driven and nonlinear
Hypothesis, yes/no but willing/able to be proved wrong
Results are questions with a very small scope open-ending, larger more complex questions
Goal: knowledge as fact Goal: knowledge as mutual understanding
Truth exists and can be known/constant What we need is a consensus that we can work with, act on
Ultimately Guba and Lincoln are getting at, positivism is appropriate sometimes (Zika virus example)
Sometimes constructivism is the appropriate response
--Plato, knowable sophists – relative
Creating a test for writing, seems like positivism, but really falls somewhere in between the two
Ex) our placement exam at WSU
A humility and testing hypotheses over and over, over time as central to Guba and Lincoln -Mark
I don’t know what I don’t know but I know what I need to ask (positivism) and the humility part falls
under constructivism where we admit we aren’t at capital T truth yet (regardless of whether it is
knowable)
Example of the lightbulb. Knowledge evolves, and evolves in a straight line (positivism)
Knowledge evolves like a spiral (constructivism)
–teaching is that thing that no matter how much you know there is more to know
“We’ve never taught the same course twice” (they say/we say)
Why did Guba and Lincoln put positivism and constructivism as opposing each other?
They have a bias toward constructivism, that it’s more sophisticated and harder
Reality is they are both difficult
A fourth generation assessment: the hermeneutical circle of/for information and then the stakeholders act. After a while it’ll need to be done again (more information and things change) [the spiral goes round perhaps]
Does constructivism allow us to put the different positivism approach in conversation to each other?
-Lacy
Answer: yes. And yes, ideally. Example, timed writing
Constructing the prompt, then given to the test-taker, and the readers (three groups)
Test-takers that wrote to how the readers read the prompt did better – but how to bridge the gaps? The prompt writers and readers can sit down together but the test-takers remain in the gaps
Hitting the audience of the test-takers is something we can’t quite ever really know
Example of prompt development with the four-parts, ends up with about 100 prompts because – example of prompt involving a zoo. For rural and certain countries, it isn’t a good prompt because the person being tested isn’t exposed to or have access to zoos.
Prompts have to be continually updated because the student demographic shifts – constructivist – but it is studied, assessed, and changed off of positive methods.
If a prompt excludes even one group of people, it needs to go – figure out the deficit in the prompt
Methods and information will differ with what our goals are
9/1/2016 part 2
ReplyDeleteConsumerism and education – are the students consumers or beneficiaries? (parents as consumers)
Parents demands v. student demands
Then, with consumerism and funding, there are donors and so we have (potentially) a hermeneutic circle – how to develop education to appease all the players: students, teachers, parents, donors
Example of business and “real world” people giving feedback on what the (value) of what classrooms were doing – what it looked like to the real world
Managers/assessing have a lot of control – so there are problems (immediately) with running a university like a business
Trends of putting business-background people in charge of education and so far results are disastrous (so it’ll work itself out – yeah!)
U of Phoenix as one that is (kinda) making the business model work
So, the methods (positivism and constructivism) help us figure out what is working and what isn’t (overall business method don’t work)
Positivism – relatively simple, a question we can ask
Constructivism – complex, the question might not be known
From pages 180-81, Tabby, what to actually do with constructivism and the value of vicarious experience
Also, theory, practice, and praxis – Condon
The common understanding accounts for a programs overall success – if parts are hidden or left out, there are a lot of divergent understandings it can’t operate as well
Example of revision of mentoring tenure track professors in the English department (because it was identified as not working) – then it worked well and other administrators want to take a look at that
Then there is a set of procedures for non-tenure – and with both it’s something that needs to be revisited regularly because things change and because that keeps people familiar with the procedures
Shared and involving more stakeholders, for things like mentoring program, is needed for it to work well
Discussing, example of the portfolio system for English 101 at WSU
Condon – studies show students work harder with no grade/delayed grading
Portfolio makes students a stakeholder, gives them choice
Break
Ack. Xyan, is how Xyan spells her name, just so y'all know. My bad.
ReplyDeleteLecture 3-Assessment-Friendly Assignments
ReplyDeleteThursday, September 8, 2016
2:58 PM
• Assessment-Friendly Assignments
o Means for assessing the effectiveness of writing assignments
o Every time there is an iteration of a course, we need to evaluate how to improve the course in the future
• Creating Assignments
o Weekly, performance-based assignment
• The same task, done weekly (or regularly and frequently), as long as it involves facilities worth developing, can show progress from week to week, thus demonstrating students' learning.
You could consider every week, or every day, pending on the situation
Can show progress incrementally
Allows you to examine progress of understanding--provides lots of information on what they're learning and how they're learning it
Can give you evidence they're learning beyond
• First/last week "paired" assignment
Giving similar tasks (again, tasks worth doing) at the beginning and at the end of the course can provide evidence of 'value added.'
• Doesn't have to be identical, the pre- post- assessments
• Assignment to be assessed with scoring guide
Developing and sharing a detailed rubric for evaluating students' work helps them address the dimensions of good writing (or other learning) that the rubric describes. If the assignment sets a task that can be evaluated fully with the rubric
o Example: Speakeasy Café
• Developed for World Civilizations course
• Students can go in, select a question, and respond to it
• When designing, you want this space to be for students, not for the instructor
• Students posed:
Fact-based questions
Class process questions
Discussion-based questions
• When looking at the examples, one student in particular demonstrated the following:
Provided more context
Narrowed focus
Greater care and consideration was given to forming questions over time
• Students weren't graded for the question, other than participation
Allows students to sort themselves out on how much they want to learn
• This type of assignment is low-risk, meaning students can try new things, play around with content and language and won't lose points
Gives them a place to explore
Seems more fruitful in upper-division courses
• Provides a more organic means of assessment, rather than just appealing to the rubric
o Example: Objectives
• Three objectives
Course Compact objectives
ReplyDelete Personal Class
Teaching, Learning, and Technology
• Have students read the objectives critically and reflect [forward] on their reactions to them (first week)
• Have students look at their initial statements and reflect on how their expectations played out in the course (last week)
Students don't have to just draw on their writing process changed, but other aspects as well (e.g. reactions to group work)
• Assignment that Addresses Rubric
o Functions
• Clarifies expectations for students
Students will know what exactly they're being graded on
• Helps faculty devise better assignments
Can see what works and what doesn't
What are students struggling with? What are they exceling in? How can those considerations be incorporated into future assignments?
• Improves students' performance
When students know what is expected of them, they can better focus and work on those elements
Design the rubric to be "usable"--don't include more than 9 elements for grading
Students can even help with rubric development
Instructor can be "secretary" to developing the rubric
• Focuses grading
Can help you grade in a more timely manner
Circled numbers on a rubric can function as feedback on their own
• Facilitates communication about quality
When we talk about quality, we are not talking about stasis
Students gauge quality by the amount of time put into the assignment
Rubric gives you language to talk about quality
• Promotes improvement over time
Assignments and timelines need to allow for this
• Helps fight plagiarism
Allows students to know what the professor is looking for
When they are unsure, they may plagiarism
o Need to align with course and assignment objectives
o Numbers on the rubric can tell about the success/understanding of the assignment
o Want to look for (according to Elbow):
• Not good enough
• Good enough
• Praiseworthy
o Rubric and the WSU portfolio
• Rubric should communicate a consistency in evaluation throughout the portfolio statement
• Could consider using a scoring guide with the dimension of element presence:
Absent
Present Sometimes
Frequently Present
ReplyDelete Consistently Present
o Rubric expresses what you and your class value in writing
• Want to center your rubric building around those values
• Creating Assignments
o Inquiry-based assignment
• Asking questions that we don't know the answer to can provide rich evidence of students' abilities as lifelong learners. These assignments necessarily move beyond having students divine "What the professor wants."
• Allows students to put important topics into public, class discussions
o Collaborative assignment
• Since collaboration gets the learning process out in the open, we can devise richer and more varied ways to document that learning, and we can more easily see where the learning is going right or wrong
• When students work together, they have to negotiate everything
o Peer response/process assignment
• These activities also get the learning process out in the open, where we can document it. They also provide evidence of important life/work skills.
• Example: An environmental science professor assigned a writing project in which the initial draft was due 100 points and the revision (due a few weeks after the professor returned the works with feedback) was worth 50 points
• It's a good idea to make your classes a feedback rich environment
• Have to develop assignments that require all members to contribute equally
EX: Students have 200 accountability points. When a student does something that lets the group down, they have to "give" their groupmates their points
• Goal is to maintain the 200 points
• Students could end up with a negative score
• Can build on process by having students build and revise off of a piece all semester; receive feedback regularly
Consider only the last 5 grades
Helps the students determine how hard they want to work
Don't receive a grade unless it's a C or higher
o Remember that as you develop your practice and assignments, it takes at least five (5) iterations to fully realize the pedagogy
• Class logs
o Having consistent writing assignments can help students internalize information
o Allows students to approach the lecture and the content differently and perhaps more carefully
• Key Factors for Success
o Quality of task
o Time of task (1)
• The more time a student puts into school, the better they do
o Interaction with peers (2)
o Interaction with faculty (3)
• Find a way to interact with the students without ending the conversation based on their assumption that the instructor's understanding is the end all be all
o Promotion of course objectives
• Assignments Alter Dimensions
o Examining native speakers and non-native speakers and the features of their writings in particular disciplines
ReplyDeleteo Concern of collecting and analyzing data
o Could focus on high, medium, and low
o Examine how language and structure influences a student's understanding of an assignment
• E.g. how might a term like "discuss" be interpreted differently based on the student's home culture
o Reference Murphy and Ruth
o Look into CompPile.org
• Good for gathering resources
o This project could serve as an ideal pilot for further research
• Mark: WPA Outcomes and Technology
o Examining how technology is discussed with WPA outcomes over time
o Hardware and design
• Similar to the difference between hardware and software
• Introduces questions of access
• Hardware: more material
• Design: Examining the aesthetics (pulling from Kress, Rhodes, and Alexander)
o How did the assumptions on hardware and designs influence the language of the WPA outcomes?
o Methodology: Still solidifying; phone or email interviews with those tasked with defining access, and how we went beyond questions of class to questions of applications
o Project has the ability to test or observe the distinctions in action
• Looking at post-embargo Cuba and DIY culture
• Reference a DIY collected across the island
o Explore how writing assessments address technology
• Could bring in CCCCs statement
• Explore how much power an administrator can have over the technology students use
o Mark Weisser: Ubiquitous computing; college campuses are a battle ground for tech use; the presence of
SYLLABUS QUESTION: Design one assessment-friendly assignment that's aimed at your practice; have peer review with our seminar; could inform your 508 project and/or the classes you teach; needs to be done by September 24
Assessment at WSU – 9/15/2016
ReplyDeletePt. 1
Assessing the portfolio – foundation at WSU. It was a moment in our history that allowed us to describe the evolution of the writing programs.
Beyond Outcomes
Attempts to bring in the theory of portfolios. It holds essentialist discussions for portfolio assessment.
What makes a portfolio? What makes it work in the context of a course then think about the program level? What makes the portfolio system work?
Portfolios as assessment have been around since 1970s. The collection of work is thought to be better than a single piece of work for many reasons.
What makes a collection more appealing than assignment by assignment?
• If you want to show all your important attributes, no one can do that in one thing. There’s no one exhibit that can show everything. There’s a historical thing to consider here. We come from a past – I came from FYC where the writing was done in bluebooks in class. Product-based, time constraints, all that stuff. If you read about the college experience of writers who have written memoirs, that practice goes all the way to the 1900s. 1960s, people started having what was done be composed outside of class. It’s not surprising that people started playing around with portfolios in the 1970s. Another thing that’s inherent in a collection is time. If you’re grading piece by piece, you have to grade it at once. With a collection, you have time to revise. What you see emerging in 1960s-70s is where comp incorporates process and drafting. Linda Flower, Jimmy Moffitt – the notion that a piece of writing is never finished, it’s just due; that it can continue to change. 90% of writing happens in revision. Having a collection buys time for revision. Which buys time for learning. SO you’re not caught trying to make up for earlier low marks and mistakes. The piece by piece grading system assumes that no one is going to get better. Ability is ability and you’re showing it in different ways – narrative, description, compare and contrast, different modes and they got complicated as you got along, but your writing ability was just a thing. But students learn, progress, and how do we accommodate that? In response to feedback, students revise and edit. It turns 5 papers into 10.
How do you accommodate revision and learning through assessment and revision?
o Allows students to achieve the outcomes you’re after and select the best of that collection that matches those outcomes.
o You can’t grade paper by paper and get anything about metacognition. You get what they did. Either a student could or couldn’t do X, but you don’t know what it is. That’s why reflection is so important.
o With selection, you can find how they’re reaching outcomes. That process of selection is a very considered process and it involves a lot of inherent metacognition – how they’re strengths match the overall outcomes.
Edie-Marie – I did portfolio based class with choice, but you didn’t have delayed grading. You got to choose which ones, but you could revise. In revising it, the improved grade would replace the lower grade. There was incentive to pick the ones you struggled with and grow from it.
The final portfolio HAS to be at least half the grade. What are the implications of that system?
• There’s nothing wrong with students going back and doing their writing, but they value grading. Letting them know where they stand as they go along may help. What are my motivations if I’m in this system?
Mark – responses to feedback.
• Bill –figure out how to create a class that holds lots of feedback – it will improve diagnostic skills, among other skills.
Pt. 2
ReplyDeleteBill - Teaching without anything but the final grade is hard to do. It’s a real challenge. You can read everything about the harm that grades do. Grades encourage a be good mindset instead of get better. “A” is a ceiling. Why would you want to revise? Well, I’ve arrived. Portfolio is one of the things you need to teach in that context. You’re going to learn a lot and I’m not going to grade it until you make it as good as you can. The research on ungraded classes shows that helps but it doesn’t make the students comfortable.
o Bill – you’re always holding them accountable – whether in the grade or feedback. Motivation for starting with a lousy draft, that motivation assumes that there’s extra grading reward for that improvement. Part of getting an A in the class is the idea that it’s necessary to improve. Someone who gets As in their writing will write a C draft. But if there’s no ceiling and what they’re going for is objectives, then they don’t need to play that game. You just have to be careful in handing revision so you don’t place incentive on bad drafting.
• I think it’s great that the standard is how it is, but we need to think harder about how to scaffold it so it works. The portfolio has a lot of elements, and how do you get students to internalize what good writing means in such a way that it helps them reach that.
• Mark – I thought about incorporating the outcomes in specific ways. I made a forum on blackboard that is grade commentary separated in the five outcomes. Using their work as examples from this particular perspective based on the outcomes. They’re not only practicing the outcomes, but evaluating themselves.
Mark – reflection on outcomes. What range of reflective prompts have you seen?
o Bill – the level of challenge is the key factor. The weaker ones say write a reflection. The better ones will tell the students to look at the outcomes, and how those pieces reflect those outcomes, what did you learn? What did you go through? How did you get there? How do the exhibits show that you’ve achieved these objectives? They’re created in a way to talk openly and deeply about the process. Not the diary entry version. Reflection is opportunity for self-assessment. It’s probably because they couldn’t take that opportunity. Reflection is where students get to talk about what’s in the box. (122).
o Formal reflection gives them chances to show metacognition. Look at the depth of their learning.
• Bill – Read The reflection instead of the entire portfolio. If you build that reflection well, then everything you need to know will be in the reflection
Mark – genre of reflection – They’re able to talk to me
• in their writing, but I haven’t tried any other way.
o Bill – we started off with the executive summary or cover reflection. My favorite reflection was in the form of a closet drama. It was done as a couple sitting over dinner talking about the writer’s portfolio. But in the dialogue, everything came out. SO the possibilities are endless. I’ve tried to ask students to be as creative as they could possibly be. It gives them agency.
BREAK
Programmatic Assessment
USING ASSESSMENT TO SUPPORT INSTRUCTION – Beyond Outcomes – graphic in introduction.
• Left side is linear inclusion of assessment and instruction
• right side is the amount of writing. How little writing should a student work on in order to achieve this. It adds up to at least 100 pages of finished work – but really 400.
• Last category on the right has to do with effects and impacts within the reach of the program. Effects on people who operate it.
• The real version is wonderfully color-coded. It gives you a programmatic view.
Short history with Elbow.
ENGLISH 508 CLASS LOG
ReplyDeleteThurs., Sept. 22
2:50 p.m.
• Attention to Provost's statement of relevance of Q-value
• 2 handouts:
o "Office of Writing Assessment Washington State University Eleventh Findings (June 2013-May2015)"
• Every year Writing Program submits report on portfolio assessment
• Policy briefs by college
• What sorts of inferences do you have?
♣ Credits you should have before submitting portfolio is 60 credits
♣ Mark: Those students are only doing half?
♣ Anna-Karin: How many credits have they completed?
♣ Condon: To completion; Have to do one or the other to register for courses; then later, cannot register until they've completed portfolio
♣ Lacy: How long to completion of degree? 4 or 5 years?
♣ Condon: 64% graduate in 6 years
♣ Edie: Statistics of those who graduate somewhere else?
♣ Condon: 70% finish college somewhere; roughly 80-20
♣ Lacy: Do these numbers include non-native speakers? Held to the same standards? Cultural considerations? Writing Center?
♣ Condon: transfer, non-native, non-English speakers all included; same standards
♣ Edie: What is the purpose/what motivated the Passed with Distinction characteristic?
♣ Condon: The carrot; make something a graduate requirement and the administration consideration is to think about how to not make it another hoop and how to ensure it doesn't hinder degree
• Motivate them to do it on time
• 10-15%
• Either Pass or Pass with Distinction on transcript or Incomplete
• Complete it means they do UH 202, and once they complete that work, then they pass the portfolio
• A little extra thing to put on their resume
• Good for employers: employers say they want some form of writing among top 3 desirable traits (e.g. communication, writing skills, work independently, and work well with others)
♣ Mark: Page 5
♣ Condon: At Tier I, all their reading is blue book; Haswell's genius is making rating process as efficient as possible
• 3 pieces of writing marked at least Acceptable, and then blue book comes in as a pass, then 5 pieces of writing with 4 different faculty looking at it
• No one single faculty member reads it all
• Takes 60% out of the middle
• 20% on Needs Work end; 20% with Distinction
• Tier 2 decides
♣ Annie: Teacher feedback of excellence?
♣ Condon: 3 Outstandings = Tier 2
• Read class papers to see if they Pass with Distinction
♣ Lacy: Page 5 regarding ethnic identification; any trends observed of why that is and how to improve these numbers?
♣ Condon: Gap is closing; international students in particular have lower rate of Pass with Distinction and higher rate of Incomplete, and smaller rate is simple Pass
• Some reasons are pretty obvious: when you look at programs, high International student rates have high Needs Work (criminal justice vs international business); look program by program; try to give better supplementary instruction and instruct faculty--> more success in sciences than humanities, as sciences are used to it and more accommodating
• Look at data and see that in some cases, General Studies is a lot less populated than it used to be--old dumping ground; small number of students
♣ Condon: Look at final page; the ones who just gave the only 3 papers they had do worse than those who chose from 5 or 6; came across many students in phone surveys or front desk who just don't have them; challenge is there has to be enough writing in curriculum to have enough papers to choose from
• 2 factors to consider: 1) sheer number of students involved in each class and 2) students put in papers they feel best about; the experience of producing that paper was a good one
• Carlton portfolio system modeled on WSU: found that when professor has done workshop and pedagogy experience, students were most likely to choose papers from that course; the more the faculty learned to teach writing effectively, the more worthwhile the students felt their papers were and more likely to put it in their portfolio
♣ Condon: If you contrast the fact that students are taking 101 within first 30 credits (1st year and a half, on average) and you look at time to completion (have 79.5 credits), 101 paper was a long time ago; why dig into distant past than choose one when they're more experienced?
ReplyDelete• All students have to take 101 and teachers do good job and students feel good about papers
♣ Annie: Include teacher comments?
♣ Condon: Students can submit with all markings; students can go to professor and ask to sign off to submit paper to portfolio; prof might ask to make revisions before it goes into portfolio; students sometimes put cover sheet on portfolio and that's what authenticates score; profs initial each page to show they've read it
♣ Condon: Based on Top Paper Submissions by Program, what inferences do you have about papers submitted in science?
• Lacy: The more writing is emphasized (English and History), the more central the writing process is to the course itself and the prof's pedagogy, the more likely the students will view their papers as a fulfilling piece--more of a connection
• Condon: Consider how active the program faculty are as part of the portfolio assessment and their participation in Writing Program
• How many trained as portfolio raters
♣ Anything surprising?
• Annie: Physics
• Condon: Math
♣ Condon when director of Writing Programs, then working with Doug Baker, tried to do something about math offerings; took a while but used undergraduate opportunity grants to focus on math (quantitative approach); difficulty is math courses everyone takes are taught by NTT instructors
• People teaching GenEd requirements are much easier to work with
• Delighted that 87 came from Math
♣ Edd: Woman's Studies is low. Not as many people enrolled in course?
♣ Condon: Look at enrollment; if it does enroll a comparable amount to other programs with 100+ submissions, then why not more writing?
♣ Mark: Do students know what portfolio expectations are within their programs?
♣ Condon: Look at completion data; something going on keeping students from completing faster
♣ Edie: Music has higher credit hours, takes longer to accrue
• Condon: Data gives you ability to ask questions; talk specifically to department heads and ask why there are issues with time to completion; numbers give you the chance and if you're Dept chair or Program Director, you can see numbers and see what's happening and ask why-- content with answer or change with response? Assign more papers? Do better job of assigning papers? Data gives you question, and then answer takes longer
• Condon: In English, things are pretty much what you'd expect- 60% are Complete, simple pass; 20% Complete with Distinction; 20% Needs Work
♣ What correlates more highly with success in first-year writing is SAT math section, rather than verbal section
♣ Overall portfolio performance: Among highest are English, German, Women's Studies
o "Appendix C: Paper Submission by Prefix and Course Number 2013-2015"
• Condon: What inferences can you draw?
• Not in alphabetical order
• Condon: Favorite acronym is here FSHN- Fashion? Fishing? Actually Food Science and Human Nutrition
• Condon: In some programs, it doesn't matter when they take it sometimes and enrollment is so tight and backlogged, they'll take it whenever they can get in
♣ College asked how much it will take to get rid of backlogs in number of courses we teach/more sections
♣ Todd: $250,000, and College was expecting only $30,000
• Mark: Do they track how many submissions from online courses?
• Condon: Don't know. May be tracked separately
• Edie: Tracked by campus
• Annie: Less writing in upper division
ReplyDelete• Condon: Don't want to see a lot of papers in upper division
♣ Looking at Writing in the Major courses, as they have more writing
• Lacy: With 402 they're starting to develop major-specific courses, and curious if development changes the amount of submissions from the different majors
• Condon: Doubt it; same number of students, but kind of writing will differ
♣ Can look at this data and ask: Where is writing happening in my curriculum? Don't want to see it reduce
♣ In Engineering, schedule is so tight, they try to take 400 after 101, as a sophomore
♣ ABET requirements (Engineering accreditation body) are so rigid that students need to take courses exactly as prescribed
♣ Funding at level of deans to shift money allocations between departments--> transfer of funds between colleges
• Mark: Chatter about Communications taking 102
• Condon: Mechanical engineering had been negotiating with Communications for them to offer technical writing
• Condon: In English, all upper division courses are Writing in the Major courses
♣ A lot of papers coming from them; English has pretty narrow range; reflects the fact that for an English major, all courses are fair game; English completion point is 78.1, which is just a little behind but not much; English majors are getting lots of writing and getting enough to turn in
• Condon: Molecular Biosciences; even though it's 101, it doesn't have big enrollments; that's where writing is happening coming into the degree; 304 is M course; not much writing going on as a whole, but chair decides what to do next
♣ In sciences, undergrad is banking model, just learning the facts
• Condon: Natural Sciences are doing pretty well with writing and comparable to English; Animal Science doing well
♣ Tier 2 portfolios qualify for nomination for best writing portfolio; raters read nominees and pick 5; Best portfolio awards come from a wide variety of students from different disciplines and a few (7 or 8) come from Animal Science. They do a good job of getting them in publication pipeline and lots of mentorship as co-authors
♣ 120 students sometimes during graduation are setting foot on campus for the first time--> Global Ed
• Edd: AMDT are waiting until 420 to wait to do writing
• Condon: Chair can wonder why; small program; lots of papers from 420, which is supposed to be a senior level course; maybe can move to first year, 107 or 208
♣ One of biggest gains is in Human Development
• Lots of writing and students are choosing their papers for portfolios
• 204 and 205 designed by Kidwell and won awards for it
• Lots of writing in that curriculum; chair could go to dean and show success
• Anna-Karin: Integrated Plant Management small numbers
• Condon: I don't know what IPM is, maybe a subset of Horticulture
• Edie: That major has quite a few courses at U of I. Can students use papers from transferred courses?
• Condon: Cross-listed/campus courses can count; closer to home than papers from community college
♣ Whenever I go to someone else's 2-year college campus, I go to the bookstore and see WSU portfolio packet-- our reach as a program is far
o Condon: The purpose of looking at these things is on the Writing Program page since 1995
• First read in 1993 and first buy-in was 1995
• Passed 80,000th portfolio
o Annie: When a student gets an Incomplete, do they need to take 1 credit courses to complete? Or resubmit portfolio?
o Condon: Meant to catch students who struggle at mid-career; want to prevent them from taking upper division before they are ready so they can get help from trained writing faculty; upper division instructors in the majors are not always trained to teach writing
• Early research shows that we were placing 1/3 of students in 102
• Consider context: WSU incoming GPA average is 3.4; in Michigan 3.9 needs a prayer to get in
• Of the students that place in to ENGL 102, only 10% of them stay at bottom
♣ 90% who do well at mid-career were not placed in supplemental at entry
• "Pick up a rock and see what crawls out": can be a range
ReplyDelete♣ Transfer students do better in portfolio than native-enrolled student
♣ For the most part, those who enroll in 102 get in good help, get 4 credit-hours, 4 hours a week, and rise
♣ Condon has been teaching Basic Writing since it was Remedial English, and notices that they just need difference approach; not dumb, but just not experienced; by mid-course, they are ready for 101; could have been placed or choose to do 102
♣ One of Condon's favorite overheard conversations between two basketball players: freshman mad at 102 placement, sophomore said he was "down" with getting help once a week; 102 designed to be a positive experience; when you give students that, they thrive
♣ There's always going to be a bottom 10%, but if it's the same 10%, then you have problems--you're not having an effect
• Portfolio is biggest, bestest "rock" you can turn over; shows what you need to work on
• Look at statistics and data by college and department and think like a dean, department chair, or program director and look at the richness
o Condon typically shows data like this when talking to program heads, and their jaws drop because they want to be able to get data like this
• Wanted to get to program level in discussion-- Writing Program here
• Directing a Writing Program big or small, you have to have data
♣ Not just for program but institution also needs desperately
• Assessment Project Updates
• Edd: Trying to do a project on service learning assessment
o Methodology: Lit Review, Polarize issue between quantitative and qualitative (narrative, discourse analysis, and genre analysis) assessment
• Criticisms of qualitative: Hard to judge how much students have grown; diff contextualizing experiences; subjective ; lacks precision and magnitude
• 2 scholars: Cushman and Schulz
• Positivist vs constructivist
♣ Land somewhere more constructivist, but still incorporate some positivist
• Argument: Do need hourly logs, pre-/mid-/post- (positivist assessments) along with qualitative assessments, and learning objectives
o Condon recommends: comp folks developing service learning pedagogy; search Comppile
• Center for Civic Engagements outcomes
♣ Don't stress discourse analysis
♣ Hopefully get something to shift there
o Most service learning in Biology and Education
• Contacted faculty for syllabi
o Condon: Some faculty leery of giving syllabus
o Question: What service learning objective will you use to line up with outcomes of class?
• Annie: Looking at reflections from people who wanted to change the outcomes of their junior writing portfolio
o Why students got placed where they got placed
o Condon: That might be an interesting concept in itself; some students just don't understand requirement
• Characteristics of that group: discipline, genre
o Annie: Don't know what it looks like to put together this information
o Condon: quantitative end- research on the teaching of English; don't look at heavily statistical ones; assessing writing can be fairly quantitative; more along the lines of discursive, not heavily quantitative; those two should give you good intro to genre; form follows function; what Condon likes about data collection process is collect, then decide what to do; naturalistic development
• Anna-Karin: Looking at how non-native speakers write lab reports
ReplyDeleteo Contact in Bio department- haven’t heard from him
• Looked at the data and found students in English 105 to get some comparison between good, middle, low of American students versus good, middle, low of those who take 105
• Waiting for data
o Looking at methodologies and how to do framework
o Condon: method to compare 105 and 101, to get baseline of where they were and how they do discourse; cannot stalk, but don't be bashful about checking in; differences between research agenda and teacher's agenda
o Lacy: spoke to Patty and curious about how notion of self-placement might influence non-native students placed in 105; community college transfers aren't placed in 403 and then seen at 402, and their grasp and mastery of English language aren't where it is supposed to be for them to best succeed in course. How does the student who perhaps transfers to WSU from other universities or community colleges differ from those who receive suggested placements?
o Condon: If you can think about that demographic information, you can get it through institutional researcher; student ID #s; short list of questions and ask students through survey
• Tabitha: Distributed handout of plan of pedagogical underpinnings of UOG Placement Rubric
o Condon:
• Start with qualitative, then posit quantitative--> then suggest a survey
• Ask instructors typical patterns noticing--> commonalities in specific demographics
• Textual analysis through topic modeling
• Focus on what you can get
• Offering a ton of free labor
• Carve something out of the pieces--> in order to arrive at a rubric
o Lacy: Suggests survey of students and instructors about rubric outcomes
o Mark: Coordinating schedules may be cause of lack of communication; Qualtrics --> send a link and then have students submit
o Condon: Developing an instrument is a publication-worthy event
• Mark: Role of hardware in composition
o CCCC's position statement on Teaching, Learning, and Assessing Writing in Digital Environments
• Last updated in 2004
o Role that they ascribe to reading environments
o Looking at how these things hold up to 2016 and 2017
o Hardware-based pedagogy versus design-based pedagogy
• Rhetorical obstacles
• Moving from bridging digital divide to what we're doing with technology now
o Opening Spaces - lenses for looking at writing technology
• Good framework
• Could go into Raul Sanchez, but it would go somewhere else
o Looking at two statements and suggestions over what happens with position statement
• Role of digital divide in hardware
o Edie: Students maybe affected by fear, rather than access; distrust of tech
o Mark and Lacy: distributed survey on choice and preference with regard to technology; kinds of things students have to navigate
o Lacy: How do students see students see technology in the classroom? Curious how secondary ed pedagogies and practices have influenced how students feel about incorporating technology in the classroom
• Lacy: Curious how slowly integrating students into notions of agonism and slowly integrating civic issues
o Asking students to seek out additional perspectives that they are different or contrary to their own
o Being knowledgeable of ow to participate in civil discourse
• Disagree, but in a civil, productive way
o Working with Dixie State
o Better to look at department outcomes? Or broader lens (WPA)?
o Condon: Outcomes are not directly rhetorical, yet what students produce are heavily rhetorical, more so than what the outcomes require
• Institutions' sense of outcomes different from what students do
• Different curriculum in play: curriculum in paper, what teacher does in classroom, and testing group
♣ In most situations, those don't line up well
• You have these different outcomes, but how much congruence is there? If there is not much, what do you do?
• Edie: Used survey monkey to ask how peer review process went
ReplyDeleteo Ultimately revising and polishing to give to all 101 instructors to distribute to students to see what they think about peer review
o Going to use Qualtrics
o Running into limitations of Survey Monkey
• Eg. "Were the Peer Review Questions clear?"
o Data project with 101
o Mark: Conducted face-to-face peer review?
o Edie: One Peer review so far, face-to-face in AML with Word
• Spent 8-10 minutes on Word tutorials
o Mark: In methodology, clarify how peer review took place
o Condon: We know peer review is good practice, but we know it's not really practiced well--> develop data
o Edie: Learning to be comfortable with the uncomfortable parts of teaching
• Next class: bring in assessment friendly assignments
o Frame them in a way that makes sense--> no standard way
o Contextualize them for us so we know what's going on
o Bring copies for class
Hi Everyone,
ReplyDeleteSorry if my Class Log looks confusing! My bulleted system looks strange in HTML, so here's a link to download the document in Word: https://www.dropbox.com/s/xjahlzmx7fyk3ga/ENGLISH%20508%20CLASS%20LOG%209.22.16.docx?dl=0
Hope that helps! :)
September 29, 2016
ReplyDeleteAssessment Friendly Assignments Discussion-
Lacey- has posted a link to a Google Doc on the class blog. Ran into technical problems, and asked to wait to go.
Ed – gave handout on his assignment. Has questions about he purpose of the assignment. Concerned if anyone did a rhetorical analysis on the document. Assignment about rhetorical analysis. Open to feedback on the assignment. Also shared the rubric that his students designed for what they are to be graded on.
Annie (A)– asks how the rubric? Works into the portfolio.
Bill (B) – notices that there are missing points on the rubric, and whether the peer review & workshops are counted twice.
Mark (M), Lacey (L) contribute to what should actually be in assignment descriptions for students.
M – suggests to have a flow chart to clarify the steps of the assignment.
Tabitha – suggests to have bullet points to describe each part of the assignment.
M – suggests using Blackboard to help break up the assignments.
Tabitha – suggests to have the component and due dates separated.
Lacey – suggests adding portfolio objectives at the bottom of each page of the assignment descriptions.
M – suggests contextualizing how points are allocated for the grade.
B – suggests contextualizing the course objectives on the assignment sheet. Notes that the handout is not really performance based.
Lacey – asks how it goes when students help develop the rubric in the class
E – shares how students have to argue and justify what should be in the rubric. Shared how he adjusted the rubric after the class discussion to meet the objectives better.
T – shares how she has tried to understand how get students to truly understand what rhetorical analysis really is.
B – suggests changing the paragraph on description paragraph, praises Ed on including the little stuff out. Arranging all the things that have to appear in the final product in one location. Asks about the rubric. Thinks the weighting works. Thinks that a student would not understand how they would get 50 points. Asks when is peer review, and how much time between proposal and final. Wonders if students could give quality language for the peer review. Mentions that doing a collaborative rubric helps students to understand what they can actually do in writing. Allowing students to see what is achievable. The teacher should use descriptors that come from their work. Better to give students strengths to achieve than weaknesses to avoid. Asks what Ed would learn from the students.
Ed – shares that he would learn what groups prioritize in their groups.
M – notes that there is not much on a definition of research in the research & analysis category.
A - asks if rubric is only for the one assignment.
T – wonders what the benefits of having a blank chart, to duke out the point values and what the descriptors should be. This would help put the onus on the students to take notes and pay attention to do the assignment to lessen the need for the instructor take notes.
B – suggests to double up the groups so that you get sets of words in constructing the criteria.
M – asks if Ed asks if students use the rubric as well. Ed says ‘yes’.
L – wonders if students would change their values for different assignments throughout the semester.
Ed – shares that he did that. The weighting was standard throughout the semester other than the focus of the given assignment.
B – identifies that the lack of change in the rubric related to homogeneity of genres.
M & B talk about the change in genres
L – shares that her blog post is available. Hoping to get feedback on the format before she writes the formal description. Shares how she normally focus on the argument at the end of 101. Interested in discourse. Interested in how students would respond to arguments. Getting students to construct arguments based on what they have seen. Seeing how students synthesize the argument construction.
ReplyDeleteAnna-Karin – asks about the assessment, as it seems a bit like a “needle-in-a-haystack”
Curious whether introducing argument style would influence how students develop arguments.
T –shares using WordCloud for what makes “good writing”. Then compares before & after WordClouds on the class time. Students able to change in vocabulary.
B – Interesting way to check on concepts and retention of old concepts. Wonders what are students getting that L is not overtly teaching. Talks about the 3 types of curriculum – paper curriculum, taught curriculum and tested curriculum. Teachers need to look for what students are learning and what they are not doing. What are students that they might not be required to do.
L – wonders how are students engaging with others that the framework might not address.
Ed – gets that L values rhetorical listening rather than rhetorical reply. Maybe make it RL more explicit.
B – states that maybe better to not explicitly state to see if students have learned the tasks. Wonders how many incremental assignments.
L – 4 incremental assignments; middles assignments being in-class assignments; still trying to manage 6 major things.
B – is the class online or physical
L – could be either; has done some in the current class that she is teaching
M – likes the idea of having collaborative assessments not just individual
B – since online courses increase time-on-task in the online portal; physical class students can distinguish between classwork and homework.
L – invites people to post comments
Annie – shares wanting to learn/try peer-review assignment. Considering having students include one the peer-review in their portfolio. Wants students to include one the best peer review in the portfolio as well to monitor the peer-review process. Wants 3 questions after the peer-review process for the author.
ReplyDeleteAK- clarifies about 3 questions
A – aware of conversational tone in the description
M – asks whether the rubric needs to change from topic to topic
B – wonders what students think about the rubric after using the rubric multiple times.
T – shares her method of wanting to read all the documents that students include in the portfolio; gives students terminology to use when doing peer-reviews. For 1st drafts have students answer the questions on the prescriptive handout holistic, analytic, narrative & _____. Uses check plus, check or check minus to grade peer review.
A – Thinking about not wanting to grade the peer-review. Has had her 102 group give verbal positive feedback to each other. Had students also include anonymous feedback about what they wished would have been in their pr feedback.
L – mentioned having sent A links from a class a few summers ago. Have students write a letter to the reader reflecting on their own writing. Have students ask for 2 local and 2 global comments.
B – thinking about having students get a heuristic rather than a rubric.
A – wonders how students would react to having their work shown in class for good reviews vs. bad reviews.
M – Eli Review does what A thinking about. Maybe share good/bad from other classes.
T – Have a chart for good comments for rhetorical vs. usage so the class can agree on what kind of feedback the members’ value.
B – using things that are modeling good practices, but avoid highlighting the “bad” things
L – Nancy summers suggest having students say “I think I did __X__ really well in this draft” will give reviewer something to start from in their own comments.
M – bringing a wide range of student comments. Have students rate comments from different classes.
B – At Michigan, have high, middle, low portfolios for reviewing in workshop. Gives participants ability to see what was actually valued by reviewers.
Tabitha – misunderstood syllabus; gave 3 different assignments.
ReplyDeletePre-reflection – have students determine how they would want to achieve the learning outcomes, borrowing from B’s presentation
Post-reflection – have students reflect on how the students achieved the goals without the course goals
The assignment would help T identify what she can share that students have learned in the course.
A – wonders if a 3 - 5 page paper would freak students out
B – shares that it will challenge them but help them to focus on the writing
M – assignment would help the students prepare for their portfolio
T – wondering if assignments could be included in the portfolio
B – takes the view that the assignments could be used to help prepare students for the reflection in their portfolio. Asks whether giving students the pre-reflection before writing the post-reflection would allow the student to identify. Suggests framing the 2nd assignment to invite discourse rather than yes/no questions.
M – framing the questions as ‘in what ways..’ will cause students to think about their writing more
B – asking questions that at the end that students could actually answer
L – thinking about how an incremental reflection to identify reflective comments
M – shares that he had done that idea in 101 and that the comments were much stronger throughout the term.
B – reflective piece most important piece in the portfolio. Reflective piece needs to be first in the portfolio because it sets the tone of the portfolio. Jeff Summers? Had ½ students put reflections at beginning in portfolio and ½ at end but reviewers read reflections first anyways.
T – liked the comments from peers
Ed - curious about how T’s previous students wrote reflection in different modes
T – One student wrote an extended metaphor about running; one student wrote a script between student & teacher; realizing students want to impress her in their writing
B – identifies that the assignments give students way to identify what works in the course. Students will be brutally honest
Mark – handout in color because all student handouts he gives are colored. Explains what is needed. Rating scale questions for Eli Review to give comments on IMRaD format for assignment. Identifies that he is still trying to figure out how to use peer review.
ReplyDeleteL – impressed at how specific M is on his file names
M – states that the file names help students &
T – asks about how beneficial on each handout rather than just on the syllabus.
M – shares that it is beneficial to reinforce on each assignment.
B – not repeating defies human nature
M – question: what should be in evaluation criteria?
T – likes the chart, as it lays out the points
M – likes creating an ecology of the assignment and review comments. Helps alleviate Blackboard issues
B – change bullets into steps to accomplish
M – PR is worth 10 – 20% of final grade.
B – likes the evaluation of the quality
Ed – wonders how peer review process,
B – likes the vocabulary included in the handout; Research on testing found that if question 1 is in question 2, ½ students got question 1 wrong
L – asks if students are given a handout on what to respond to
M – wonders if anyone has ideas on using style manual
B – wondering what would be added; as the handout is a social contract
T – talks about peer review as a scientific crucible. Wonders how much grace period M gives students on submissions.
M – states if there is consistency in submissions there is no grace period; some consideration given in special circumstances; points given for completing peer reviews of all group members.
A – wondering if peer review groups change or stay the same through the semester
B – compliance of group; looking at comments to group members; how students teach other to do peer review
Anna-Karin – weekly blog/discussion board post for course in Hong Kong. Wanting to give students an opportunity to post questions from lecture or tutorial, and to practice writing in English.
ReplyDeleteThings discussed related to whether should “grade” monitor student’s use of English in posts or just allow them to have “natural” improvement over the 13 week semester. Also how to incorporate the questions into the class whether in an exam or in class discussions. How to monitor the discussion forum – state good question or model good questions and/or responses in the classroom so students learn from each other.
For next week revise assignments and submit on the class blog.
Class Notes October 6, 2016
ReplyDelete-Karin: Changes to the assignment: Incorporated Lecture into the blog responses as a form of incentive
-Tabitha: Streamlined and added due dates. This changed the way that students interact with the assignment since the information is visually accessible right at first. Added the evaluative criteria.
So that it become clearer why student are using the reflective pieces to tie into the main prompt.
It is so difficult to assess reflective writing. Included a rubric for reflective writing. Did they do it? Did they not do it? Did they kind of do it?
Clarified the grading process so that students are aware of how their pre and post assignments are graded. The grades will be averaged. Point value for both pre and post are equivalent.
Post: Less perfunctory more provocative. Did you do this? In what ways? Pre-Reflection goals add a column to the rubric that they build on throughout the semester (making it pre, during and post) and the during-course reflections add to the post-reflection.
-Mark: With due dates is this separate enough from the last paper to leave time for reflection?
-Bill: Right after this reflective session would be a great time to do course evaluations
-Mark: In inviting students to be creative, you might want to invite students to choose their audience. Since this is one of the first assignments It could set a great tone and tap into rhetorical awareness. Maybe Change it up for the final one.
-Bill: Move the length information down into the submission data, out of the rhetorical situation
-Edie: How do you define offensive language?
-Tabitha: that would be established prior to this assignment. Reiterating expectations from the syllabus.
Discussion on Evaluations
Evaluations: We are putting them online this year for the first time. Students respond in text rather than filling in the bubbles.
-Karin: With typing it’s harder to decipher what feedback came from what student
-Edie: it’s nice to have the results compiled and given to you, rather than receiving a PDF looking at it as a whole eval.
-Bill: We did this in Honors, it wasn’t the best. No one did it without class time being allotted for it. Had to do two, honors and English and then class specific evaluation. It takes a lot of time.
-Lacy: Dixie State offered incentives to complete evaluations in a timely manner. How will it go without incentives?
-Edie: this university is HUGE
-Bill: It might be possible to tie it to student’s access to their grades
Mark:
ReplyDeleteIncluded information between peer reviews. Situation peer review as a valuable and important genre of writing. Tying Peer Review into the portfolio outcomes.
This works in the classroom but addressing the bold terms, and having students articulate the how this works in a peer review. Inviting students to problematize and define terms as part of the peer review process.
This assignment sheet would not be a handout: it would be on Blackboard
Peer Review is formative: and formative for a specific issue that has been clarified for each peer review
Pg. 5 Incorporating an image into the prompt that show the different aspects of the ImRaD style
Pg. 2 Steps rather than numbers, cleaning up the information
-Bill: It is a better designed text
Shows the point system to show students what is important and valuable
-Tabitha: On pg. 3 Could you model and provide examples of different styles and modes of peer review. Contextualizing the peer review process. Similar to what you did with IMRaD.
-Bill: If you thought of this as a shell, you could use it in any class. 101 may need scaffolding, but 201 and 301 may not. It makes a good heuristic for any writing class.
-Edie: the significance of the color?
-Mark: Students are able to locate colored handouts more easily and we have access to the colored paper. It’s possible to color code types of handouts, but it would take A LOT of planning.
Images that have copyrights, how to cite that in assignment design.
-Tabitha: you can do a role-over or make the image a hyperlink
-Lacy: This looks designed to be a hand out. How does it play out with students using this and getting this done since it is not a handout.
-Mark: this isn’t a peer review that I would do in class. The peer reviews that I do in class are much more tactile. i.e. The colored pencils and the identifying exercise. In class has directions that are less cumbersome. Simpler tasks for in class.
-Mark has a module specifically for questions about assignments.
Modelling students to complement each other during in class and then have that transfer to the peer review outside of class.
Edie-Marie:
ReplyDelete-Tabitha: Are you asking students to evaluate and summarize? Maybe separate those two terms to clarify that this summary is not objective listing of facts it is also evaluative.
-Edie: objectivity is not very possible. Also these articles themselves are evaluative
-Mark: I like the first lead-up, encouraging students to look at author credibility
-Edie: students did it on their own, found that one author had his own patented vaccine.
-Lacy: End goal is synthesis; do you encourage students to find sources that are opposing? How do you frame what type of articles they should be looking for?
-Edie: the two articles are general and broad, dealing with historical aspects and one with expectations. Students can take it from there. Some students give a historical overview, some argued for reform. One student just took three random sources. But most students focused in on one aspect that interested them specifically.
They don’t have to use scholarly sources, which is why author evolution is crucial.
-Bill: How is this assessment friendly?
-Edie: this is the first assignment, I have transfer students, some international, some traditional students, they have different backgrounds, this assignment is a norming process and provides a baseline. Baby Bib is a small assignment just like the one that is coming later in the semester. This Bibliography is based on completion and feedback is given.
-Karin: Do they do anything with the sources that they find?
-Edie: they summarize the sources for an annotated bibliography
-Tabitha: Do they know how to do an annotated bibliography in a 201 class? Would modeling or breaking down the components be useful?
-Edie: Showed them her own master’s annotated bibliography
-Bill: If this is a baseline assignment and you want to know how much you teach them. You don’t want to provide them with this information before the assignment.
-Edie: I don’t punish students for what they don’t know, but they need to address it.
This is the breakdown of the unit. This is the 3 week. They have already done some reading. This is to address the smaller parts leading up to a larger synthesis essay.
-Mark: Breaking information up into bullet points would increase readability. Frame it and then steps that are separate would make it more accessible.
Two Weeks from beginning of assignment to a reviewable draft.
-Edie: Some of the feedback I addressed in class: The balancing act, what to include in the handout and what to leave for in class instruction
-Bill tells a kind of freaky Greek Myth to illustrate the balance between brevity and necessary information
Lesson—Bill appreciates brevity
-Tabitha: Is it beneficial to provide the criteria on which they will be assessed at this stage?
-Edie: Doesn’t use a rubric, uses track changes and then has conversation
-Mark: Designs rubric based on the instructions, are things absent or not?
Lacy:
ReplyDeletePosted the updates on her assignment on the shared doc form last week.
Focused her changes on the start of the semester and end of semester analysis.
Wanted an artifact that is used at beginning and end. Decided using March CNN Republicans discussions. Provides different perspectives and speakers.
Asks students to look at the effectiveness of the arguments. It will let Lacy know where students are in there understanding of argument. Ted Cruz was much easier to analyze then Trump. Students can apply terms to dissect that argument.
Added Word Count Expectation for responses to avoid the one word responses.
End of Term: Going with Donald Trump. Now students have more experience engaging in the scholarly sphere and the public sphere to be able to engage in this more complex interview.
Increased the word count for the end of the semester. Students should have more tools to respond to an argument by this time.
-Lacy: Is the language of her prompt clear? She wants students to ask who is the speaker, with whom are they engaging and how? Students watch interviews in class, and write in class, but have time after class as well. So students are only responding to the very brief part of the interview. They can select which portion of the town hall meeting they respond to.
-Mark: Could add question about the interview as well as the candidate. Their allegiances and how they deport themselves.
-Bill: by specifying word minimums, you don’t know if you are achieving a goal of having students engage more deeply or if students are just meeting a specification.
Having students do directive self-placement.
ReplyDeleteA good assessment need more than one sample.
It can turn out to be a computer mediated interaction between students and institution. Here at WSU students sit for a placement and confirm to the results. Other Institutions students stay where they self-placed
No one right now is really measuring the reliability of directed self-placement. This is basically waiting for someone to write for a grant to look into this. People are thinking this is the way to go, but someone need to validate it.
Writing-Less placement: and indirect test of writing used for placement. Scores are used for placement into first year composition.
-Lacy: Ivy League, doing away with ACT and SAT for admissions. How might this effect student success at an institution?
-Bill: Harvard students don’t take a writing course. (interesting right?) Where Most students take the most writing? Community Colleges.
At WSU we do placement, but we don’t really have places. Most students learn to write at community college, fewer go to large state universities. Such a tiny, tiny percent of people go to elite schools.
Ivy League schools are on the peripheries of the writing instruction. When we look at what works, we don’t need to look at elite instructions. We need to look at places that represent the heart of the enterprise.
Small private colleges are rich enough to do what they want. For smaller schools it is so much easier to design an assessment program, you only need 6 or 7 faculty members to get on board to make a change. Bill facilitated a workshop at a small private college and in an hour the faculty dreamed up a portfolio now in use.
We, a bigger institution are the ones that have the numbers and are trying to do right by the students. So directive self-placement makes a lot of sense. Asking students to look at the information and place themselves.
Data shows that most students place themselves conservatively rather than ambitiously. Most students are most concerned about their writing. To back up self-placement, you need a writing sample from students. If students place themselves conservatively and then the system tells them they can go higher how does that effect students?
Validity started out as something you take the test-makers word for. IQ test and army testing. In 1950 it changed to validity as something assumed in the test, and that it dealt with how the test promoted the test takers wellbeing and experience. If the test was screwing things up for the test takers it wasn’t valid. If it helps it was valid.
1990s construct validity. If you are not assessing writing, then you are not assessing writing. Writing must be involved in writing assessment. Multiple choice cannot, does not assess writing. Seems obvious, right? Apparently it’s not!
What are the outcomes of the assessment? 1998-1994 . Unified validity no single determination, validity is a process. Have to keep examining to see if validity if there. How does it fit into the curriculum how does it affect the test taker?
There is a law that WSU cannot teach remedial writing. So Victor Villanuevea did away with it. There were always students that couldn’t pass 101. Had to bring basic writing back. Students were dropping out. Had to develop a rating system that could identify basic writers and make accurate placements.
Class Log: 10/20
ReplyDeleteThursday, October 20, 2016
2:52 PM
• Bill distributed handouts from his conference in Indianapolis
• Tom Angelo Handout: Survey from Chronicle of Higher Ed
• Compare percentages between what we think and reality--> What do these percentages tell us?
• If students want to learn more about things they’re interested in and want to be more knowledgeable and cultured, how are we promoting that in our classroom?
• Angelo presentation helped to better frame pre-/post- reflection, wherein students are asked what obstacles they may encounter when approaching the goals (course or personal); students make a plan for overcoming obstacles; at the end, students evaluate the ways they overcame their obstacles
• Find tasks that allow yourself, and your students, to find joy or wonder at least once a day --> helps to build momentum to finish tasks and see how they build into the final product
• Assessment activity
o Have students set their own goals and see how they blend in with course goals
• Discussion from handout
o Have 4 instead of 3 assignment
• Giving students the agency to select their assignments can be empowering for them as they get to decide how to represent themselves and their writing
• Signature strengths: strengths you rely on the most; asks students to identify those strengths and articulate them; the last essay should be a more traditional academic essay
♣ Kind of like backwards engineering
♣ Balance can be found in blending teaching with the things students are interested in
♣ References course built around "positive psychology"
♣ Can teach the basics without dictating a genre; can establish the genre while allowing the students to select the genre
• Mark: Page 4
• How do we enable students to follow the Seven Research-based Guidelines for Assessment to Improve Teaching and Learning?
• Encounter difficulties with the amount of project 4
• Bill: Plan with the end in mind; build in assignments that help students value their own learning
• Edd: "Parrot Test" on Page 4
• Bill: Handout meant for those doing program-wide assessment
Constructed alignment- Students come up with teaching philosophy; Bill encouraged students to think about what students would encounter then
• point out in draft syllabus where philosophy influenced development of syllabus
ReplyDelete• Weigle
• Page 2: Questions to consider in designing assessment; Think about the WSU writing placement
• Edie: Placement essays have to be mind-blowing to be exempt
• Bill: Fits with our philosophy; tests our philosophy that it tests what they're going to be learning in English 101
• Semester doing assessment with Hong Kong Society of Accountants: develop an assessment system that would work within an accounting firm and would link accountants with university system to advance in ways they need to advance
♣ 4 levels: (5-7) partners, (20) managers, (199) senior accountants, (200) jr accountants
♣ Question: How do you handle promotion?
• Peter principle: In a hierarchy, people will continue to be promoted until they reach of level of incompetence and cannot do the job. How do you avoid this?
• Reality: communicate with one another and with clients; up and down communication involves some kind of writing, typically on paper so that it survives all the different links
• Involves evaluating the person's language skills: give clear directions and relay good results--> low stakes, mostly institutional
• They were promoting people who couldn't do the job; becomes an even bigger issue when face-saving is more significant in China than in the US
• Testing language ability at different levels
• Did not need much to test junior accountants moving to senior accountants
• Answering questions of what we want to test and what we want to know entails looking at if people are qualified to do the job they're hired to do
• How can the employees reach a level of competence that would help them get promoted?
• The purpose of testing in placement is to decide whether to put them in 101, or 101+102 (100 numbers inconsequential)
• Bill: It is impossible to test writing without also testing reading--> How central do you want reading to be in writing assessment?
♣ Take authentic tasks and build prompts from them
• Lacy: How might integrity of the test be compromised if you consider the population? Differences between states in how they approach education. How do you still maintain credibility of exam while still giving students to opportunity to succeed
• Bill: 1) presenting them with a task that is congruent with what they need to have a successful beginning in 101; give them something to read that's not overly complicated but still gives them room to make an argument and 2) avoid cultural discrimination; try not to alienate people through a question
♣ ACT test had a question about computing cab fare --> who is being privileged?
♣ Important that test takers have a point of reference; designing a question without a cultural or geographical bias is important
• Establish a threshold that students will realistically confront in the first week or two from the course
• Mark: Students draw from a common reading
• Bill: Be careful not to raise the level of complexity too much and that there's just enough time to get enough handle on the issue involved and use some of that information in their essay
• Bill: 2 terms get thrown around a lot in book
• Performance assessment: tasks in environment they will be in
• Authentic assessment: take work products they're doing and assess those
• These do not make you stop learning to make you take test; they repurpose your learning environment so that your learning environment can be assessed
• If takers are succeeding at an overly high rate, then there is something wrong with the test--> validation studies
• Make adjustments
• The hardest thing in writing assessment is coming up with prompts
• WSU: 4 frames and 20 prompts per frame= 100 or more prompts that fit in frame
♣ Share the prompts
♣ ETS does not share prompts because of hours time and money that meet criteria of fairness for millions of test takers
• Analytical way of evaluating writing: Samples of kids from Done, NE
ReplyDelete♣ Explain ways in which your life will be better than your parents' lives
♣ NE very traditional place; farm country; change happens slowly; people live frugally--> to ask these students to complete this task is asking them to criticize their parents' lives
• Students on that prompt were half as competent as compared to the other thoughts
• "Know the test takers well enough to not blow their minds"
• Expert rater system: rater makes the judgment
• Call for raters in the spring term: $30 to 35, depending on the tier
• Training involves deciding whether a students can handle 101 in his/her own or if he/she needs assistance
• Mark: Tier 1: Not making official designation of where student is going; put in Tier 2 for questionable; don't do much outside of putting them in different piles
• Bill: Tier 1 is certain students are going to 101
♣ Tier 2 gets settled by more experienced raters
• First summer of reading is using experience in 101 to place students; 60% in Tier 1; Tier 2, who are higher paid, get fewer to deal with in determining placement
• Bill: Hong Kong accounting company
• Junior accountant: Evaluating the writing of the accountants to determine whether they get promoted or have to go through additional education
• Senior accountant: demonstrate they can do memos and trying out things that managers write; managers evaluate
• Mark: Digital placement tests?
o Bill: system wouldn't change very much; just a logistics thing; IMOTE, not have to do it physically as part of ALIVE; doing remotely; no pressure in getting sample rated
• Raters would get things that are easier to read
• No need to change format of exam
• Go to directed self placement eventually
• Bill: Who uses this info that the test provides?
o Lacy: the program and raters
o Mark: other researchers and programs
o Annie: the university
o Anna-Karin: students--> Bill: Not so much
o Lacy: state governments to note progress--> Bill: not so much; process fulfills one of seven ways institutions do these things responsibly
o Bill: Registrars office; they program it into WSU and it affects every students' record; advisors use it; students are only aware because of what it allows them to enroll in
o Edd: surprised state legislature doesn't evaluate in whether or not they're meeting standards in higher ed
• Bill: high schools ask for data; different high schools across the area meet with program twice a year; teachers want to know how their students are doing; all state wants to know is what we're doing fits with what they want us to do
• Bill: If we had a lot of time, would we still do English placement test? Or something better?
o As direct assessment in timed writing genre, our test is the gold standard
• Students do it physically as part of ALIVE, placements are known, and then students enroll
• Logistics easier when you conduct placement yourself
• Outsourcing (to COMPASS or ERATER) gives up validity
o University of Michigan
• Norming session in the morning, then by 3 or 3:30, the group's blue books are rated so that next morning, students can sit down with advisor to enroll
• Lacy: All seniors had to do professional writing portfolio in her old school; Do any universities determine enrollment by senior portfolio? Is it simply timed assessment?
o Bill: No, and rightly so.
• Daughter had to do showcase portfolio that had certifications of things student has done, but not actual work
• When portfolio in Michigan installed, asked for 3-5 pieces of writing:
ReplyDelete♣ One most representative of writing ability
♣ One from a class other than English
♣ One student identified as best or favorite piece with reflection on why
• If schools just took portfolios from high schools, universities would not know what they're dealing with--> How would they rate that?
♣ Easier to assess Michigan's context, since that is where students want to succeed
• With those three samples, students' work could be judged on whether they would succeed in English 125
• If students couldn't come up with one of the three, they must account for it in the reflective piece
♣ 81% managed to provide writing from class other than English
• Bill: Every exit is an entry
♣ Portfolio to get out of high school is different from portfolio to get into college
♣ Good that HS prepares students with portfolio
♣ Pomeroy: community members coordinate with teachers to rate HS portfolios-- engages the community with the schools in profitable ways and generates support by showing them the great work students are doing
• Basic questions to consider in assessment on page 2
• Terms in Page 48 on Test Usefulness are helpful
• Weigle is only comprehensive source of writing assessment that takes into TESOL writing assessment and does best job of explaining writing as a construct
o Give Pages 2 and 48 "extra love"
• Edie: Validity
o Language and power with students; power of students choosing
o Liberate students and their writing, but also help students write in the language of power
o Is construct validity sitting in that same camp or not really?
• Bill: construct validity asks of the test; questions of validity ask of test; does test measure what it claims it does?
♣ If you're not testing writing, you're not testing writing; if you don't have your writers write, you're not having takers give writing to be assessed--> got to have them write something to test that construct
♣ Political economy conspires to ignore that, but sometimes it comes back
• California: Legislature declared that if ETS didn't include writing sample, they would not use SAT
• Result: SAT2 had writing test instead of lose California's patronage
♣ ACT and SAT writing parts are now optional
♣ Bill: With different tests, what are they testing?
• CC use COMPASS (ACT) and (ETS) ERATER
• COMPASS is editing skill
• ERATER collects 20 minute sample (utmost automated essay scoring can handle) on tightly controlled and delineated topic
• Computer looks for 42 linguistic features associated with what human raters associate with writing
• Bill: Moving WSU placement online
o Les Perlman found that you could rate samples from across the room: the correlation between the length of the sample and the score was 90%
o With online, students can take as much time as they want
o Les figured out when correlation between length and score reaches 50% (correlation goes away), if you remove time constraint and let people write as long as they want, is 4 hours
• Correlation still exists with WSU placement exam
• If we move test online, we could do more interesting things with it and the correlation between length and score would go away, which would be good
• Bill: with validity, is it testing what it says it tests; other questions:
o Is the test adequate to the results that it gives?
• If CC, the writing placement exam may or not be able to place students with that so fine-toothed a comp
• WSU has only 2 placements
♣ If students place into 098, they will not get to take 101 (2.5 years of English for a 2-year degree)
• Test has to be able to make high-stakes decisions
• Consequential validity: is the test able to engage the consequences of the assessment?
• Validity: Keep checking; validity is not a things but an ongoing process
♣ Takers, context change
♣ Every 2 years: validity studies; biennial portfolio report
• Bill: Reinventing 100
ReplyDelete• Victor came in as Director of Comp and did away with 100; went 4 years without placing students in 100
o Complication: students tried their hardest and were still failing 101
o To appease legislature, made 100 look like a college course
o Worked though curriculum stuff and went back and pulled blue books of old 100 students and train new raters to understand 100 placement
o Edie: Most of the time when I think blue book exam needs to be in 100, as Tier 2 reader, it's more or less imaginary because I never taught 100 (took class to, but never have)
• Bill: Bottom 10% at University of Michigan (still high achievers)
• Frustration of teachers: spending a lot of time making students' writing a little bit better
• Bill knew what basic writing is and what word salad is
• Job was to push these students up
• Edie: Fall schedule has less than 60 students enrolled in 100
o Bill: sliding scale; create a new category of who belongs there
o Over 5-6 years, for a time under Lane Rowlands, the goal was to raise caliber of students
• Complications: land grant is meant to expand access
• Elson and budged cuts came and needed to raise tuition in spite of legislature
♣ Used to give 54% and now only 26%--erodes functionality; raised tuition to compensate
• If it costs $8,000 dollars (state and student each pay $4,000), you have to educate student at half price
• If you raise tuition, you can get more students ($6,000 from student, $2,000 from legislature) and losing legislature funding is inconsequential
o Sam Smith--Lane Rowlands--Elson Floyd
• Scale shifted: More college ready to less college ready
♣ System has to be able to accommodate that
♣ Cannot just declare test is valid; in a few years later, you may have a different set of test takers
• Bill: Element of writing assessment that is about language competencies
• Different from other places in US
o Focus on competency and leave language acquisition to other people
• Anna-Karin: students in HK start learning English in elementary
o Students grew up speaking Cantonese (no writing) and written form of Chinese based off of Mandarin form
• By the time they start learning English, it's their 3rd language
o By 1974, there had to be equal opportunities to language: Chinese or English medium
• Power dynamic of English in HK
• 1997: students have to be trilingual (Mandarin, Cantonese, and English)
o Mandatory exit exam determines which universities students go to
• Community college more expensive that university, as state doesn't pay for students; students pay for themselves
o If students came from English-medium school, their English is fairly competent; if Chinese-medium school, there were issues in writing and speaking
• English 1000 (equivalent to EN 100) or 1500 (equivalent to EN 101)
• Bill: construct is language acquisition and not writing ability
♣ In HK, it's language acquisition as writing as after thought
o Teach students to think and express argument in Western linear progression and process
o Bill: We have developed, because of print in 16th century; once people began to have printed text, you no longer had to translate to learn but could buy it and then could compare and teach texts--write commentary and publish
• Linear: knock down someone who's written before; you don't do this in Asian cultures; Asian thought is venerated-- honor that and say what you want to say; accommodate what people before have said into what you want to say (cultural competency)
o Those in English-medium schools already accustomed to
o Bill: biggest influence to international assessment is colonialism--sun never set on British empire, wherever you went in the world, people spoke English for various purposes
ReplyDelete• For colonial society to work, you had to teach native people to use English well; first wave
♣ Some places work spookily well like India and adequately well like HK
• For EU, it became possible and normal for a person in one country to go to university in another--establish competency levels; second wave
♣ Bologna accords: developed in 80s and 90s to meet assessment needs--assessment all about language acquisition; write the language enough to cope and succeed in the university
♣ Whole system that turned up the heat on assessing language acquisition--tests outside of US aim at primarily
o In colonial system, you learned English to succeed; if you didn't you remained on lower levels of economy
• Anna-Karin: people aren't on their phones to the same degree as in HK-- possibly due to population density?
• Bill: Weigle book is useful because when encountering non-native speakers, it helps you know how to approach it
o Lets you know why international students don't care about the same thing you care about
• Their focus has been language acquisition, while teachers ask them to do rhetoric
• Lacy: Germanic and Asian languages differ
• If you took someone from Germany and someone from Hong Kong, how do we account for differences in language (German closer to English). Do we put them in the same class?
o Bill: Germanic language less likely to be EN 105, as they study and use language with native speakers
• Edie: Friend missed learning Russian language when wall fell
• Bill: HK, China, Japan, Korea have Native English Teachers; not TESOL trained but train students to speak with native speakers
• Students coming from Asian languages didn't grow up with verb tenses or articles
• Tabitha: understand variance between ELLs from different regions and even among different countries
o Know their background with English to see how they approach writing assessment
• Bill: they're not testing construct on ways that students will need it, but what we want them to know
o Asvar, past graduate student, last true Marxist on Earth; Bangladeshi, brilliant, smoker
o Facebook: Anis Rahman becomes Bill's Facebook friend and asks questions about university; wanted to get PhD from first world country, looking to immigrate; Bill coached him through TOEFL to get TESOL masters degree and second in rhetoric
• Bill: The question will be- How long will the US hold out in building other language competency into their writing assessment? Probably not long
o World is growing more mobile; communicating across thresholds is easier
o No longer geographically isolated
o 60% of graduate students in STEM fields are international students
o WSU Education Program being maintained by international students
o Oil rich Middle Eastern countries get paid to go to school in US
o As the richest country in the world, why can't we do something like that?
• If questions remain for Weigle, keep them in mind, but if not, we will move on to Elliot
• Tabitha: How about Assessment Projects?
• Bill: Next week; first hour work collaboratively to discuss progress of Assessment Projects; then Elliot, which may spill over into two sessions (with Very Like a Whale)
o Middle of November: Have conference-style write-up (you get as far as you get); doing something real-world takes more than a semester and spills over a lot; engage with methodology and new topics; Bill willing to keep working on it through his retirement
o No deficit for not having brought Assessment Project to closure-- bring it up as possibility
Thank you so much to Lacy for her notes at the beginning of class! :)
ReplyDeleteHere is also a link to the Word document for clearer formatting: https://1drv.ms/w/s!AgjVmHf4H81igWk9nULz2Y_u19U2. :)
• Announcements
ReplyDelete○ Write-up due on November 10
§ 7.5-10 pages (conference page length)
§ Should be written up as a conference paper, even if the project is still going on
§ Post to class blog
§ Will set up peer review after that
§ Should still keep your reviewers informed
• Overview of final assignments
○ Lacy:
○ Annie: Appealing Junior Writing Portfolio decision
§ Looking for more opportunities for additional status
§ Only students placed in 1 credit courses appealed
§ Few numbers/data is making it difficult to draw specific conclusions
§ Examining the process of assessing portfolio
§ 16 artifacts for analysis may indicate some promising insight
§ Can go into greater detail on factors of success based on observations
§ Research "regression analysis"
§ Can look into Haswell's work, he recommends following-up with students and the writing they do after graduation
□ Could go through Alumni Association
□ IR
§ Could develop a system to test over time
§ Current research indicates students trust the portfolio process
§ All students have been assessed twice (at both Tier I and Tier II)
§ Should include more detail about the nature of the junior writing portfolio
§ Look at the outcomes discussed in Beyond Outcomes and how a program can explain its purpose and objectives more clearly
§ Could foreground it in the findings that WSU has the more "sophisticated" portfolio system
○ Tabitha: UOG English Placement Rubric
§ Reconstruction of project
§ Looks at population, praxis, pedagogy, and purpose
§ Context:
□ Population
® Race demographics
□ Planning and Purpose
® Referenced Zimmerman and the theories that informed MA thesis and creation of the rubric
® Wrigel to assess limitations of assessment
® Looks at limitation of research
□ Pedagogy
® Millman and Zimmerman
® Why they constructed the rubric in the way they did
® Zimmerman: students are unprepared; correction is important;
® Millman: we need to orient students to American/Western discourse and practices
◊ Guide for Correcting Composition
□ Practices
® Moving away from the deficit model
® Current model stresses the deficit model, even at greater levels of proficiency
® Deficit model pathologizes
◊ Labels both writing and writer as deficient
◊ Can occur across disciplines
§ May frame conclusions as "frames to consider" during revamp of program and outcomes
□ Ties into the "so what" of the piece/why it's important
§ Could also look at how much change (and much has not) since the last iteration in 1991
○ Anna-Karen
ReplyDelete§ Has 5 complete sets of lab reports
§ Recipient of lowest score had difficulty with paragraph construction, making it difficult to parse through at times
§ Examining methodologies for examining science writing
□ Book by Biebers
□ Articles referenced come more from bio-chemistry
□ Thinking about looking at moves of bio-chemistry articles
§ Consider the TA role in evaluating the writing
□ Looking for close comparisons to the writing of the discipline
§ Studies show that those who had several surface level mistakes, did not receive as great of credit on larger, global aspects of writing
§ Could look into genre expectations and how those expectations (held by professors and TA's) influence the evaluation of writer's lab report content
§ Can possible access more partial (but not full) sets of data
□ Will help with "might be" statements
□ 10 complete sets could better indicate existing patterns
§ Could do this on a larger, more official scale next semester
○ Edd: Service Learning Assessment
§ Has lit review done
§ Looking at instances at WSU
§ Identifying issues within assessments methods of service learning
§ Assessment model is based on quantitative growth
§ Formal write-ups (provided by students) are short reflections
□ Relies on a professor's qualitative analysis on quantitative data
§ Been working with Center for Civic Engagement
§ Moving towards more of a constructivist approach
□ Discourse analysis: being able to understand the conventions of community they're writing for during instances of service learning
□ Will also emphasize the importance of qualitative research during service learning assessments, rather than relying so heavily on quantitative research
§ Look at how they're selecting ongoing service learning activities
□ How students integrate these projects into writing outside of the classroom
□ Examine process for selecting projects and organizations for service
§ Would like feedback on structure
□ Has lit review
□ Examine the shortcomings of WSU
□ Propose methods for improvement
• Reading Discussion
ReplyDelete○ Companions for On a Scale
§ O'Neil, Moore, and Huette
§ Robert Connors's works (historian in the field)
§ Tom Miller (historian in the field); work is more ideological; more interesting but perhaps less reliable
○ Text begins with NCTE rubric
§ The field had already been going for 30 years
○ Some background/context
§ Pre-printing press was meant to record something that had been said aloud
§ Post-printing press, writing was a site of rhetoric
□ Rhetoric had to account for the differences between speaking and writing
□ Greater rates of literacy
§ Writers (even into the 19th century) wrote as if to be read aloud
□ EE Cummings was one of the first writers to write for the written page, not to be read aloud
□ Difference between oral and written communication and their rhetorical expectations can be English took on text production, while communication took on the performance
§ Rhetoric began finding its ways into other disciplines after the invention of the printing press
§ More recent theory pays attention to rhetoric in its written forms
§ Speech has a limited distribution, while writing can extend over more temporal and physical contexts
○ Expanding enrollment
§ Students didn't share the same preparations as they had before; had different experiences and approaches to discourse
§ Created a deficit model with students
§ Harvard referenced as an example of entrance writing exams
§ Teachers were removed from most of the assessment process
§ Referenced Mismanagement of Man
□ First IQ test and its effects when picked up by Stanford
□ Not initially thought of as a fixed quantity; yet as a means for identifying those who may need extra help
□ Revised applications removed the upward mobility aspect of it
○ NCTE was expanded out of MLA
ReplyDelete§ MLA did not take the teaching of writing seriously, so NCTE was developed to address this apprehension
§ Many rubrics started appearing around the development of NCTE
○ California testing system
§ Subject A Exam
§ Test would determine preparedness for entrance into the UC system
§ Tests were developed later to more directly assess writing
§ More direct writing exams (like WSU writing placement exam) emerged from this
○ Transaction writing began to take the place the role of creative writing in the college classroom
§ Creative focus had greater problems fitting within an industrialized society
○ Discussing differences between writing-based (more common in Europe) and standardized assessment (more common in the US)
○ Assessment's effects on writing: students had to write as if their text was going to be read individually, rather than read aloud like in the past
○ WWII
§ Two impacts
□ Perfection of the system: began with Army IQ test; used tests to rank individuals; resulted with determining who should be placed where based on qualities demonstrated in tests
® Proved effective
® Turned into the SAT
□ After effect: GI Bill; created a moment like 1870s moment at Harvard; at the end of the war, millions of men came home wanting to attend college with their GI bill; many used it as a support to go to college, rather than vocational training; in 1951, Michigan State was a 2 year community college; by 1955, Michigan State was a 4 year state university (land grant); small land grant institutions turned into larger universities because more people had the support to attend college
§ Learned that assessments from WWII were efficient
§ Trouble began when they started basing exams off of material for those who had historically been in college: white, upper-middle class men
□ Assessments, then, were designed for this demographic, thus preventing those outside these groups from doing as well as possible
□ Did not allow for additional experiences to be considered
§ Open admissions began to combat this privilege
□ Allowed those from previously excluded populations to enroll in academic programs
○ Open class discussion of text
ReplyDelete§ Tabitha: how is assessment motivated by consumerism?
□ Before WWII, larger schools had maybe 3,000-4,000 students
□ When capacity is increased, that means an increase in scale
□ The development/design of standardized test examined ideas that cannot really be assessed by an exam (designing learning strategies, asking for help, critical thinking, etc.)
§ Annie: what are the stakes for students testing in middle/high school?
□ Students are required to participate but have no incentive to do so
□ No stakes are assigned to students
□ Even though students prepped for exams nearly half of the year, assessments had no direct correlation with improvement
§ Lacy: what are the main differences between NCLB and Common Core? What can we expect from students raised in a Common Core setting?
□ Common core better focuses on demonstrating how skills are transferable to college or professional settings
□ NCLB threatened schools with funding by putting all students on the same scale
□ Saw a greater emergence of charter schools
○ Value of On a Scale: exposes both the advantages and disadvantages of assessment
§ Began as a gateway
§ Turned into a barrier for those wanting to get in
§ Reveals political nature and the intentions of standardized tests
November 3, 2016
ReplyDeleteStarts with how did Very like a Whale go?
Book is a good reference that will not likely go out of date.
Assessment book would have been too technical to publish in the mid-90’s. Book helps to bridge the divide between writing administrators and research
Authors did a good job with questions
Book is more usable than readable.
Reading introductions? Edie admits that she usually reads intros but missed the Hamlet reference to the title of the book.
Trope? Why did they choose to use this term?
Trope being used in the Aristoleian sense as well as in the way Toulmin – related to things that may become a warrant. It’s ideological common place.
DFA framework?
How does this design for assessment (DFA) framework work throughout the book. Designing first makes it easier to know what kinds of outcomes that the program wants to achieve. Outcomes might range from finding more sections, identifying if students are making progress. Example from WSU – Haswell identified two environmental prompts to see if there was change in the students who had the two prompts. The raters were to identify which prompts were better – a little better or a lot better. 79% essays showed that students at portfolio did a little better than at entry. At a program level the individual student doesn’t matter, in a longitudinal frame the program wants to show that students are getting better and better. Having a blue book in the portfolio helps show that students are getting better over time. You need to show that the writing program is worth the money involved. Another WSU example, is looking at the 29% of students placed into 102, are still at the bottom 10% at mid-career. 90% of the students from 102 were no longer in the bottom 10%. Portfolio system was designed to
NSF grant for Vancouver campus for writing assessment of engineering students. Setting up syllabi at the beginning to design for transfer to engineering student writing of lab reports. Having everyday evaluation after each workshop as part of the
DFA sounds very similar to backward design. However, DFA is more about identifying a more fluid end product unlike backward design for an assessment.
WAC metaphor from physics – particle exists a thing, or as a wave (in motion). A writing program as a particle you will find the people/location. If looking at writing programs in motion it’s not easy to find its location. Need to identify how the program is working in motion not in a static location.
WPA – assessment research is not like the research in the humanities. Need to rethink how
ReplyDeleteOften DFA happens mid-way in the process, not at the beginning. WSU started a project shortly after Bill’s arrival looking at the critical thinking involved in the writing. Two students identified a rubric to evaluate the critical thinking involved in the portfolio writing. Looking at the correlation between critical thinking and problem solving. The students identified a rating scale to help promote high reliability. Then the rubric was considered as useful to train faculty to incorporate critical thinking into their syllabi.
How is student success defined? How is this program working?
Research paradigm based around the null hypothesis. If you’re looking for success and you find success, it could be that you are just confirming what you. Starting with a null hypothesis helps you disprove the opposite of what you are looking for.
Universities are hierarchies.
p. 56 scoring sheet for technical writing
Scoring sheet as part of a survey design identifying whether e-portfolios are working.
What would the data from a holistic score like this tell? Helps you identify what is working and to what extent that is working.
p.57 shows how the individual parts lead to the whole assessment.
As a wpa it’s good to have an outside evaluation can allow you to get the funding you desire.
1st week impromptu writing is a bad scenario for everyone involved (teachers and students). Very Like a Whale helps provide good reasons to end this kind of scenario.
iMoat example from Irv Peckham, at Louisiana State University. He designed a way to test
p.81 validation: evidence & categories
ReplyDeleteThe thing you want people to do is what needs to be valid.
Validity depends on the instruments, and it also depends on what you want it do.
The construct needs to be relevant to the place where the
Figure 3.1 p. 75, the writing construct in general is the area that the writing construct is happening.
Debates between Peter Elbow and Barthalmey(?)
Writerly knowledge, group characteristics, as well as the biological characteristics that lead into the writing construct.
Outcomes for all students, as well as outcomes for individual students.
Writing construct design is a boundary to identify how students are to perform.
Validity is decisions that can be trusted.
Writing in a portfolio allows you see a student’s writing within the context of why it was written.
p. 81 the construct needs to
p. 85 diagram
collecting big data, you need to ask the same questions about subsets of the data (disaggregation). Is the construct fair to all subsets of the data set?
At WSU – performance by major animal science majors do well in the portfolio, international business students do not. Disaggregation allows you identify if there are outside influences to the difference between writing ability between animal science and the international business major.
p. 22 quote from Bill in the text
An instrument you use to test a writing construct
The writing construct should be a rich
Standardized test results are a poor representation of actual writing ability. A 20 minute sample of writing use a miniscule amount of the writing process.
NAEP tests represent more of the construct, but is still a limited representation of the writing construct.
The writing portfolio is a much richer representation of the construct because it allows students to show more their ability involved in the writing process.
The first two tiers in Bill’s handout at the beginning of the term, stop learning for a test. The portfolio looks at all the products a student has written over the course of a semester.
Correlation is not causation.
Changes in higher education have worked to eliminate “weeder” courses eliminating the pool of students. The pedagogy is changing to allow for accessibility for students rather than elimination of students.
WSU example – All university writing committee wanted a mid-career writing test since that was what was being used at entry.
CLA – rich writing assessment that does a pretty good job, but it doesn’t reach all the constructs of what is expected of a student at graduation because it gives the topic to the students, only focuses on the specific readings provided.
If an assessment is designed to stop learning to take the test, it is not very valid.
AP tests – Where do AP and IB fall in the rich vs. artificial assessments?
ReplyDeleteStart with AP – AP tests are geared towards an elitist group of students. The exam is similar to CLA or iMote exams. Universities are willing to give credit for the exam but may not exempt the courses.
AP exams train for a specific task of writing, but may not give the full picture of the writing constructs.
Arguments for decisions are based constructs.
Bill was one of the 1st reviewers of the book. Ended up being a 2nd reviewer. This book would be a good resource as a WPA. Design issues often confronted in WPA roles, are addressed in the questions at the end of each chapter. Five years is a minimum for identifying change in a students writing ability. The WPA wants to show that program is getting max “bang” for the cost to offer the courses. How is a writing program characterized by the department offering it as well those outside the department? If faculty senate are pleased with the writing program, the program is safe. Questions at end of Ch. 1, are good questions for a writing program consultant. The questions are more heuristic than questions that you can get answers to at one setting.
WPAs should network with others to find out what other programs are doing.
p. 86 is a good identifier for evaluation on a yearly basis.
Construct modeling – prerequisite for relational modeling – developing a set of exemplars. Students need to know what a good piece of writing looks like. How do you get students know what it is? Getting students to identify what was good in the exemplars. Rubric model demonstrated by Bill is a good way for getting students to know good writing looks like. Telling students what good writing is, is not as useful as having students explore/identify what good writing is.
Relational modeling – not possible to teach the whole construct. Desire to teach what will be most representative of the whole. Similar to regression analysis of the parts of the whole construct.
A regression analysis allows you to compare several outcomes of a construct outcome to each other. What elements are most likely to cause improvement? Regression allows you determine whether an element is useful to the larger outcome. Are there things that you spend time on in class that do not impact students’ final outcome.
Providing students with ways to access to perform the tasks being assigned. Separate out the things that you have to do with the thing you can change. Give students a level of expectation to access the material.
Identify constructs for a course that will give students an idea of what they need to do the assignment.
Critical thinking: Using the language for goals and outcomes in the assignment, language and prompts. Label it when they do it, so they can identify it within themselves.
ReplyDeleteConclusion is a great place to emphasize critical thinking. It should not be a simple repeat of facts already learned. Should say implications and consequences that you weren’t justified to say before the research.
Studies show that students are weakest at articulating their own opinions. Difficulty going beyond reiteration of facts.
Edie-Marie—do students not have the confidence to say “I am doing critical thinking”
Mark: Creating a research place: The CRS, John Swales
Edie-Marie: A teacher has to train themselves to use the language of the outcomes statement, seems somewhat unnatural.
What do we have at our disposal to have them thinking about outcomes
Edd: What is a good reflection question, that gets students engaged in reflection effectively.
Lacy: reader letters: Tell me what you think you do well. Global revision and local revision that you want feedback on. Gives a turn toward revision.
Giving students a stake in the situation. Something that is actionable in student’s life and process. (No paper is finished it just do.
Edie-Marie—ethnography, culture of college writing. Were your expectations met or not, what are the actual expectations and how do those match up with what you anticipated.
Getting students to articulate their understanding of the genres. Self-assessment, collaboratively made rubrics to help students identify how they are engaging with learning.
Reelection is designed to give students a voice, if it doesn’t it’s not effective reflection.
Having students articulate what grade they think they have earned in the class. Students usually placed low. Grades extrusive motivation, looking to internalize motivation.
Mark: Asking questions about how students see genres
ReplyDeleteCritical Think: John Bean, chapter on critical thinking, supporting claims.
Reflection also means it’s more difficult to get the paper done just the night before. It’s a cheap trick but it works.
No one has done a serious date driven research on reflection. Been doing it since the 90’s. We have lots of “lore” to back up its effectiveness, but less research.
Numbers and data vs. theory and nuances. Administratively deans need data so they can justify continuing funding. When you put together information you always keep in mind the audience, presenting learning in a way your audience needs to hear. Presenting data numerically and effectively is a great way to distinguish yourself, and get funding for your project. A program is built over the years by data.
Mark: How did you generate that data?
Bill: Look at the what matters. What is something that a dean or administrator might care about. Look at the strategic plans/goals that the university has outlined and use that language. For a writing program—student contact is important. Setting up English 103, University 302, made the writing center generate credit hours. Luckily—this is not that hard. You’ve just got to do it.
Your first goal has to be improvement—and measure for it. All the stuff you report to a dean is stuff you need to know anyway.
Things that have changed: We used to think that the status quo would be preserved. Positions are not simply going to be replaced any longer. Things are more fluid; everyone needs to be in a positions of justifying what they do.
There are across the board budget cuts, the money is then in a central pot, ready for redistribution. You’ve got to look like you know what you’re doing.
Evaluation vs. Assessment
Bill: Evaluation is a universe in which assessment is a planet.
Assessment is more narrow. Evaluation inquires to find the strengths and weakness.
Appreciative Inquiry is the first move.
In assessment eventually you get to ranking, but it’s not where you start.
A, B, C, No credit would be great, plus and minus is too much. We often grade on the notion of limited good. That giving too many A’s somehow devalues them.
Assessment is aimed at ranking. Who gets what at the end of the class. Relative performance between one student and another. Evaluation is aimed at things much bigger. Assessment is reductive and Evaluation is generative. Evaluation isn’t closed it’s open. It provides information that produces a line of action.
There are real consequences to scores. Too low and you’ll discourage them right out of college.
Assessment is often done rather than evaluation because it is more cost effective. In a well-designed assessment there is hardly any cheating. It means students are trusting the process and investing in their work. The grades are institutionalized requirements, and that is a fair compromise. There is a way to go before eliminating grades.
We know that external motivators don’t produce the kind of learners that we want to see running the world, but intrinsic motivation does. Video games: the better you are the longer you get to play. Assessment has just the opposite effect. We use assessment to shut people out of the next step.
How do we/should we stop students from translating any feedback that is given to them via rubrics into grades.
ReplyDeleteRubrics that are produce outside of the learning process and then are imposed are reductive and detrimental. You have to take something that is locally developed and then tweak it to fit the context for which it is applied. It takes time and expertise to develop and effect rubric.
Develop a rubric with your class. How do we define what good writing is as a construct?
What is okay writing? What is crappy writing? That way the language comes from the community.
Mark: What do you think about page number and word count?
Bill—I give a page range, a piece of writing needs to be as long as it needs to be. I don’t think that you can do it in less than four, six would be great. It’s more of a guideline.
What we do as teachers should be in response to our students.
Mark: reflection is the only component of the portfolio that doesn’t have a unit dedicated to it. What do you think of a segment just dedicated to drafting reflection?
Bill—Seems great, why not do this during dead week. You could have them bring in a draft and then ask them what part of it seems like and argument for a grade? Take it out and engage more deeply.
Mark has created an assignment on blackboard with unlimited submission. They can put things into those slots as they go. Initial drafts etc.
https://1drv.ms/w/s!AhhQ4hQABmCPhgUWIbrzpvMTwiXQ
ReplyDelete