All Else is Never Equal

In a previous post I mentioned one of the problems with educational research. A post by Andrew Gelman brings up another problem. I'm actually quoting Andrew Gelman quoting himself here:

The "All Else Equal" Fallacy: Assuming that everything else is held constant, even when it's not gonna be.

There are many books that try to tell us what works in schools. I certainly try to use them as a guide. However, they make the all else equal fallacy by removing the results from the context. Nearly all educational research looks like this:

1. Multiple choice pre-test
2. Treatment Group vs. Control group
3. Multiple choice post-test
4. Look at mean, calculate effect size and decide what works

 We then get results in summaries or meta-analyses and are told to do these things because they work. We don't have the context. We're told that all else being equal this will work. But as Gelman says, they're assuming all else will be held constant even when it's not gonna be. What does this mean for us? I guess a few things.

We need to read the original sources as often as possible. While this is not always possible we should try to understand the context of the original experiment as much as we can.


Use research as a map but don't let it take control of the steering wheel. While certain strategies and systems might have a higher probability of working, you can't let it get in the way of what you see in your own classroom.

Conduct your own research. With a standards-based assessment system you are better able to use data to look at differences between your classes and between other classes at your school. In a traditional system all As, Bs, and Cs are not created equal. Even within your own class it's hard to tell why each student has each grade and don't even try to compare students between classes.

Why industrial designers understand education more than we do

I was watching Independent Lens yesterday on KQED. The film was called Objectified. In the beginning a designer named Dan Formosa said this: (not quite word for word but the spirit is intact):

We have clients come to us and say here's our average customer: female, she's 34 years-old, she has 2.3 kids....we listen politely and nod and say well that's great but we don't care about that person. What we really need to do to design is look at the extremes: the weakest or the person with arthritis or the athlete or the strongest or the fastest person. Because if we understand what the extremes are, the middle will take care of itself.

Again with feelings..if we understand what the extremes are, the middle will take care of itself.

An industrial designer has summed up very nicely my feelings on both educational research and teaching in general.

So much in educational research goes to the average kid. We look at the effect size. We do meta-analyses. We care about how a giant lump of kids does on average against another giant group of kids. What we should care about are the extremes. A .4 effect size is nice. What I really care about is the outliers in the group. What type of student did this treatment make a HUGE effect on? It gets washed out because of our love of averages. When we look at research, we shouldn't care so much about the stuff that makes the group move a little on average (although that's still good to know). What we should be looking at is those students or groups of students in the treatment that were extremely positive and negative. What conditions existed for this specific student to make such an extreme gain? We can move towards truly individualized education only when we can say what works for this specific student under these specific conditions.

As for teaching, I don't know how many times I've been told by a principal/other teacher/staff developer that I should choose 3 or 4 kids in the middle and design my instruction around them. I've done that for a long time and had pretty good results. I'm becoming convinced though that I need to be more like Dan and design for the extremes. I need to take care of the really high and the really low and the middle will take care of itself.

A brief interlude...

Before I finish the third post in my series on criticisms of SBG I wanted to share an article from SciAm. I'm unwilling to pay $11.95 for one article so I'm going to have to take Scientific American's word for it on what the research actually says. The gist is that if people tried and failed to retrieve information first, they recalled it better in the end. Here's the money quote:

People remember things better, longer, if they are given very challenging tests on the material, tests at which they are bound to fail. In a series of experiments, they showed that if students make an unsuccessful attempt to retrieve information before receiving an answer, they remember the information better than in a control condition in which they simply study the information. Trying and failing to retrieve the answer is actually helpful to learning.

At the very least this seems to argue in favor of pre-tests and guiding questions. Where does this fit in SBG?

The way I structure my class is twice-weekly quizzes. It seems like a lot but I like for students to have pretty up to date information on how they're doing. It also allows me to change gears mid-week. The quiz (I actually call them progress checks) usually covers every standard they need to learn for that topic. If it's a packed topic, they'll get every standard at least once a week. They can see where they are and what they still need to learn. There's a nice practice effect as well. In SBG all that matters is where they finish so the students aren't penalized because they continually fail the quizzes. I've struggled a little with this because I worry that they're reinforcing wrong ideas in the beginning. The SciAm article even mentions that worry.

In SBG, all those quizzes don't hurt the student's grade because all that matters is where they finish. If I graded them traditionally I wouldn't be able to give all those quizzes because it'd be unfair to the student's grades if I kept quizzing them on things I hadn't even taught yet.