Sunday, June 6, 2010

SBG Implementation: Creating Assessments

Programming note: I'm breaking this post in to two parts. This one is on the test itself and the next will be on scoring and tracking progress.

Citation: I forgot to mention that I started off with the base system outlined in Classroom Assessment and Grading that Work and have spun it off a bit to better fit my context and style.

This is the second in a series on the process I went though implementing standards-based grading in my class. Part one here. If you haven't read that one yet, you should go back. You'll need to understand the 0-4 system before you read the rest.

Just as a refresher, here are the ground rules:
  1. There is no canon for SBG.
  2. Context is everything.1
In case you're one of twelve people in the world that hasn't read it, here are Dan Meyer's templates. I take a topic-based approach versus his skills approach so our formatting is different and the meaning behind the 0-4 scale is different. However, his guiding principles are the same. Excerpted here:
  1. It doesn't matter when you learn it as long as you learn it.
  2. My assessment policy needs to direct my remediation of your skills.
  3. My assessment policy needs incentivize your own remediation.
Adding a fourth: My assessment policy should do as much of the work as possible.
And fifth: The ultimate goal is self-assessment. Any assessment policy should help students to become better self-assessors.

Jargon alert: I use the term "standards" to refer to your in class learning goals, not your state standards. I use the term test and quiz interchangeably. I'm specifically referring to a written assessment that a student takes by him/herself

Step two in my adventures into standards-based grading: Creating Assessments
Before you get overly excited, this is not the post where you're going to learn how to write really rich and interesting problems. Look to your right. Most of those blogs devote themselves to that exact purpose. Grace has a good, short post on testing conceptual development. Personally, I like to use the rich problems for teaching, but not so much for written assessment. Context is everything here and I'd love for you to comment that I'm wrong about this. I have so many English learners that I find that I'm not really testing what I think I'm testing (h/t @optimizingke). My written test questions tend to be straight vanilla.

What I am going to write about today is test formatting. Not very glamorous, but it's one of those things that will make your transition much easier.

Tests exist so you and your students can tell where they are and what needs to be done next. The test itself should provide feedback.

There are many other ways to do this but tests have a couple of advantages. One: They're darn efficient. You can knock out a test in 15 minutes and get kids working on remediation in the same period. Two: There's a paper trail. It helps students to see how their thinking has progressed and to analyze the mistakes that they've made again and again (and again).

I definitely use multiple methods of assessment, and in fact it's necessary in any quality assessment system. However, written tests are the bread and butter for most teachers so that's what we're going to focus on here.

Here's a sample test. Most of my problems are drawn from sample state test questions, textbooks, and what we develop as a department:2
atomslg2

Specific features:
1. In the top right you'll see the topic name and which assessment number it is. They're going to reassess a lot. In my class that's usually a minimum 4-6 times for a single topic in class and even more if they want to come after school. You've got to have a way to keep them/you organized.

2. I wrote the learning goal on top. I'm actually planning to remove that. I think my original tracking sheets only had the learning goal on top and not the topic name. It was my way to help the kids match up the tracking sheet to the assessment. Now that I put the topic name on both, it's superfluous. You would think having the learning goal on the top would be like a cheat sheet to help my students answer the questions. Sadly, you'd be wrong. If anyone has a good reason for me to leave it, let me know.

3. After each test question, there's a letter that matches the question to the standard listed in their tracking sheet. Mostly, the reason for this is to help you and your students track their progress on each standard. There's a second plus. It serves as a self-check when you're writing your test questions. If you can't match up a question directly to a standard, or multiple standards, you need to edit your question, your standards, or both.

4. Questions are separated into 2.0, 3.0, and 4.0 questions. The front page consists of simple ideas, usually vocabulary or basic knowledge. The 3.0 questions are the more complex ideas. The 4.0 question has not been addressed in class but if they understand the complex ideas, they can apply those to arrive at a solution.

Most teachers I've observed cluster by problem type (multiple choice together, then matching, then short answer at the end.)

If you're clustering by problem type, please stop. It's unhelpful. You and your students should be able to identify some sort of pattern by looking at the results of the tests. That's really hard to do on tests like this. Unless your goal is to help students figure out what type of question they have the most difficulty with, I can't think of a good reason for doing this.

The second most common method is to cluster by standard. I like this method, but don't use it. I really like it for some purposes and designed our department benchmarks in this way. The strength is that there's a direct line between "what I missed on this test" and "what I need to work on." Remediation is really straightforward. Since I cluster my test questions by complexity, there's an extra step they need to work through to figure out what needs to be done next. It's not a difficult step, but some students will need more hand holding than I prefer.

So why do I choose to group my tests in this method? The most obvious is that this is how my grading scale works. It's really clear both to my students and to myself what score the test will receive. There's a better reason for choosing this method though.

I often come back to how I think using topics helps create connections across standards better than skill-based assessments and here's another example. In a skill-based assessment, students can easily see what specific standard they need help on, however because of the very format of the assessment (chunked by standards and often on separate pieces of paper) it is harder to look for patterns of errors and broad misconceptions. In a topic-based system, it is easier to see systematic errors.3

Imagine a student missing questions 7 and 9 on the test above. I could ask them to go and learn more about both states of matter and the kinetic molecular theory. In a skills-based assessment that would be my default, especially if those standards were assessed at separate times. In a topic-based system, when I read the answers together I might realize that the student has a knowledge gap that's affecting both questions. For example he/she might not understand the relationship between temperature and molecular motion, which is necessary to answer both questions.

Additionally, by clustering from simple to complex, I can quickly focus on a few common problem areas. If they're stuck on the first page, it's usually something like, "He has no idea what an atom is so he's just guessing at everything" or "She is confusing velocity and speed." If the difficulties occur on the 3.0 questions, I can rule out all of the simple stuff because I've seen that they've mastered that already.

This is totally possible to do with an assessment grouped by standard. I think the trade off of more direct remediation for making connections/ease of scoring is worth it.

All of this is a really long-winded way of saying that if you're going to implement a topic-based system, it makes sense to design your assessments by topic. Since you've taken the time to build a learning progression into your topic, keep that intact as well.

Sticking Points:
You can't figure out how to cram an entire topic into a single test without making it the SAT. You're aiming for frequent smaller assessments rather than a big one at the end of a unit. Don't feel the need to cram every last question in. Get to it next time. Notice that the second question only deals with the charge of a neutron. In other assessments I would have the properties of protons or electrons as well. Generally, if a student doesn't know the neutron is neutral, they're also not going to know the proton is positive and electron is negative. I don't feel the need to put all three on the same test. I just want to get an idea for what they know so I know what to teach next.

You've made this great topic scale but don't know how to write questions for it. I've found that commonly the problem isn't that I can't come up with a question for a standard, it's that the standard doesn't lend itself to being answered in a standard testing format. You just can't assess your state's oral language standards in writing. In your classroom, anything is an assessment. You've got your topic scales, get them to do whatever it is they need to do and evaluate it against those scales. My textbook's assessment software includes questions with pictures of a triple-beam balance and different arrows pointing at the numbers. You're supposed to then tell the mass of the object on the balance. I could do that. Or I could not be the laziest person ever and get out the triple beam balances and walk around as they determine the mass of different objects. For the most part, teachers seem to be pretty good about understanding that different modes of assessment are necessary. We're not as good at realizing that those other modes should be replacing written tests. Most teachers I know would start with having the actual triple beam balances out but then pull out the written tests later. Either they don't think it should count unless it's on a test or they're worried about CYA or they're just in the habit of putting everything on a test. Break the habit. Everything doesn't need to eventually show up in multiple choice format.

 
Next up: Scoring the assessment and tracking progress


1: Forgot to mention this last time. The book Mount Pleasant is about the high school my kids feed into. Don't buy it, but if you're in California you might want to check it out from your library and read what our current insurance commissioner and aspiring governor thinks of public schools. 
2: Teaser: After the implementation series is done, I'm going to start on the joys of common assessments. Most teachers I talk to hate them. If you do, you're doing it wrong. More likely, your school is doing it to you wrong.
3: It's easier for me at least. I don't have a good system, or even a mediocre system, for teaching students to look for patterns of errors and misconceptions.

8 comments:

  1. I'm working through as much material about SBG as I can, and I'm loving it. The school I teach at does lots of and lots of essay and short answer tests/quizzes, and very few multiple choice. This means that the kids just don't know how to do multiple choice. M/C is a specific skill that needs to be taught, and so much of life is m/c style questions (at least in academia). Thus, those of us in the sciences have started to specifically teach m/c skills. This might be unique to our school, but points to a larger goal of giving the students the test taking skills they need.

    ReplyDelete
  2. Jason,
    Is atoms#2 learning goal number 2 or is this your second assessment of atoms?

    What I'm having trouble seeing is how you re-assess them on these topics/standards. Do you have several versions of this topic test with different questions or is this the only "test" and any other assessments come from different forms like discussion/lab/etc? Do you do a 2 and out - they need to score the same 4.0 level twice in order to be able to skip that in the future? If so, how do you assess them that second time?

    Am I asking the right questions or the questions the right way?

    Matt

    ReplyDelete
  3. Matt - Yeah, so atoms #2 is my second assessment. I've got a bunch of different ways to test the same concepts (remember, it's not all tests though, everything is information).

    I don't do a 2 and out. If I think a student really gets it (this is just as likely to come from talking with them or labs as tests) I'll just let them know they don't need to test anymore in this topic. We start spiraling back in as the trimester draws to a close so I'll do a quick check to make sure they've still got it. If I'm not sure, then yes, test again.

    Remember I've got a different model than Meyer or someone like that. He's got like 80 skills. I've got 12 topics. It takes awhile for someone to master a topic.

    ReplyDelete
  4. I found your example test very helpful!! I had been wondering if putting the learning target on there was giving them the answer--evidently not! For some reason I like the idea of having it there. I have a big hangup about how mysterious education is from the student's perspective--most of the time they don't know why they're learning anything, or even *what* they're learning. This is why transparency (from standards, learning targets, and sbg) is so helpful. So I guess I just like the standard right there on the test to keep the thread of transparency and focus, and also to provide a "cue" to aid students in accessing the information (if they hung it on that "peg" in memory, this will help recall).

    ReplyDelete
  5. @wholeanimal I had thought about that too, like it's a hint or something. I put it there because my tests are mostly for self-assessment. I really want them to be able to look at a test, check what they're missing, and then immediately start working on it. If you combine my belief in the power of self-evaluation with my intense hatred for grading, most of what I've done (in terms of formatting) has been to help students do as much as possible without me.

    ReplyDelete
  6. I finally just got this. I tried to use the model earlier this year, but didn't really understand how it work and it was a bit of a mess. I'm going to try again next year, but I do have a question. If you do end up doing an assessment on an individual skill, how does that come into play on the tracking sheet?

    ReplyDelete
  7. I've been searching and searching for examples of standards-based grading in a middle school science classroom...and I am SO glad I stumbled on to your's through a google search! I've been reading and rereading all of your posts on standards-based grading, and it's been extremely helpful. After reading your blog, I'm really excited and inspired to try out a similar system in my class next year.

    I do have a few questions that I hope you can answer.

    1) Do you follow a specific assessment/instruction cycle? For example, do you administer an assessment (like your atoms example you posted) every week, every other week, etc?

    2) What do you give to the students who have already "mastered" the topic early on in the unit?

    3) Could you describe a typical progression of a topic? For example, how do you typically start, how often do you assess, how often do you give practice work, how do you know when you need to move on to the next topic, etc.?

    Thanks for your advice, I would really appreciate it.

    ReplyDelete
    Replies
    1. Hi thanks for reading.

      1. I've tried different things. Most years it's weekly. Last year I did Monday quizzes. Maybe 15-20 minutes.

      2. Do you mean during quizzes or during regular class time? I guess the answer for both would be that it varies. In general for quizzes I ask that they show me twice so that I can be sure. Otherwise it depends on the kid. Some kids take older stuff. Others just do whatever they work out with me. During class time I actually move pretty fast but then block out time later for kids to catch up. So same thing. Mostly I put those students in teaching roles but other times they'll have investigate something of interest to share types of things. Usually computer stuff because I'm not great with that kind of differentiation. If you have anything you like I'd love to hear it.

      3. Hopefully this helps: http://alwaysformative.blogspot.com/2011/09/cycle.html

      Otherwise just send me an email or find me on twitter.

      Delete