Sam Harris, Jon Haidt, and Bias

I was fascinated by my colleague and mentor Jon Haidt's analysis of the rate of certainty words in Sam Harris' books, and the reaction it sparked.

I recently came upon Daniel Miessler's post, where he defends Harris thusly:

Haidt makes a major mistake here by thinking (and saying) that because individuals are bad at judging their own objectivity it must mean that using science to gauge happiness and suffering is a fools errand.

The error should be obvious: Science isn’t based on individuals making judgements. It’s based on evidence that has been validated objectively by many.

Harris isn’t proposing that he, or any other individual, sits down and divines a solution to complex world problems using his logic Ouija board. He’s saying that science should be used to show how policy changes affect human happiness and suffering. Big difference.

This gets at some things that have been coming up a lot lately. I see very similar arguments from social scientists who deny political bias in the field. Ultimately I think we need to do a better job of explaining what we mean by bias, what bias looks like, the different kinds of bias that can arise, etc.

But let's focus on the Miessler passage. He thinks bias is inevitably avoided or expunged by scientists "objectively" validating evidence.

This could only be true if we knew everything there was to know about bias and how to catch it.

Another way of putting this: It could be only true if we knew all the forms of bias we were vulnerable to, and how to catch them.

We do not have any such knowledge. Our knowledge and understanding of bias is a domain of discovery, scientific discovery. We haven't sorted it all out yet. In fact, I think it would be more accurate to say that we're just getting started. I'd venture that there are biases in science, whether social science or other sciences, that we won't know about for fifty years. We know about more about bias today – and different forms of bias – than we knew in 1960. In 2060, we'll know even more.

So it's not possible for scientists to confidently say "we're not biased." Some scientists and some fields will be more justified in worrying less about bias, but social science is the most vulnerable. Social science is mostly made of words. We can string together some words, have people respond to those words, give the variable a label, correlate it with responses to other words we've strung together, give that a label, and make sweeping declarations. We can give awful sounding labels to our variables, loaded with an ideological sword, like "Social Dominance Orientation", and we can say that huge swaths of society are high in this "Social Dominance Orientation", even if they aren't. We do that all the time. We routinely link groups to beliefs they do not in fact hold (most conservatives do not endorse SDO, and researchers routinely conceal this.) This is a huge scientific and ethical problem. That kind of bias leads us to say things that are false, so bias can be very costly. This kind of bias hasn't been rooted out yet. We can say it will be, sticking to a "science is self-correcting" mantra, but for that mantra to be valid, science needs to be self-correcting at a reasonable rate of speed, and it isn't.

Note that it is social science that will provide our knowledge of well-being and happiness, the kind of knowledge Harris anticipates. We can add neuroscience and biomedical research to the mix, but that won't change much. There could be all sorts of biases in our measures of well-being and happiness. We could be stuck with a profound error, an error rooted in bias, and not know it for decades. This isn't necessarily a dealbreaker for positive psychology and the study of well-being. I'm sympathetic to Harris' project and I've always admired his work. I think he has criticized positive psychology for perhaps similar reasons. I'm nervous about his confidence in brain scans. I wouldn't talk about brain scans as a definitive measure of well-being without qualifying it as a distant future possibility. I would expect the first generation or two of neuroimaging to lead us into all sorts of errors of method and inference.

Another important point: Bias will not be caught if everyone has the same biases. Or even if a sufficiently large majority of a field has the same biases. Those kinds biases are only caught by history.

Bias will definitely not be caught if, in response to claims of bias, scientists simply say "science is self-correcting." That's a good recipe for non-self-correction, for reinforcing bias. Although in such cases I think perhaps we need to more clearly communicate what bias is, or what kinds of biases we're talking about.

In social science, many leftists acknowledge the political bias of the field, but some minority simply respond by declaring that the field is not biased. I have yet to see anyone engage the examples of biased research my colleagues and I have offered, or that I have offered separately. There's nothing but silence in response to the substantive examples. I'm not sure what's going on there, but I think in some cases they have no schema at all for social science being politically biased. They don't know what that would look like, have no account of that kind of bias as category of bias, and they also tend not to see leftist ideology as an ideology – only the other side is ideological.

In these cases they'll try to argue that reality simply has a leftist bias. It hasn't yet occurred to them that when a field is biased, people are expected to make that argument. They haven't lingered on the fact that their perception of reality being left-friendly is compatible with two realities: reality having a leftist bias, and the field having a leftist bias. Nor have they meditated on how to go about finding out which it is. I'm always dumbfounded that any social scientist would not understand that political ideology can profoundly shape and mediate the "reality" we see, and that being in a field dominated by fellow leftists could have a profound impact on their construal of said reality. That's an elementary observation, one that is intuitive to lots of carpenters, nurses, and baristas. We of all people have to understand it.

I know a few academics who think conservatives are inherently malevolent. That type of cartoon universe is probably unmovable – we'll just have to keep walking, focus on people whose minds can be engaged. It's unfortunate that academia has become so amenable to cartoon universes – it's incompatible with good scholarship. Scholars of that quality shouldn't be paid to be scholars – they're not good enough to be in academia, but we have lots of them. I think the poor quality of so much of modern scholarship outside of the physical sciences is a big problem, and one that will impose serious costs on our society for a long time.

In any case, bias comes in many forms, and we don't know about all of them. In the meantime, I think we need to do a better job of explaining what we mean by bias. We need a full account of political bias, its nature and operation. We don't have that yet.

Regarding Harris and Haidt, I think their views are closer than is assumed. I think they differ more in style than in substance. For example, I think Harris would agree with lots of things Jon says here. The difference in style comes down to the fact that Harris is a hardass about the obviousness of certain kinds of truths, and Jon is the opposite. I also think it's extremely important to always know when people are talking about morality descriptively vs. prescriptively. Harris' project is ultimately a unification of both, but I've known lots of intellectuals who seem incapable of thinking of morality prescriptively. Jon's good at distinguishing the two, but people following these debates sometimes forget what's what.

Even though Sam's a hardass, I think he deserves a lot of respect for sharing his journey. He's as open and transparent as he can possibly be. He shares his struggles, his exasperation, his inner processes, like no other scholar I've seen. He takes us on his journey, his dialogues and debates (see his recent encounter with Chomsky.) I feel for him sometimes. His exasperation and pain is quite evident as he grapples with bizarre arguments and unconscionable misrepresentations of his views. It will be interesting to see where he is in ten years, what ground he's covered. I often wonder about where I'll be, intellectually, in ten years, and Sam sparks the same curiosity in me.