Lewandowsky, S., Oberauer, K., & Gignac, G. E. (2013), titled their paper thusly:

NASA faked the moon landing—therefore, (climate) science is a hoax: an anatomy of the motivated rejection of science.

 And in their abstract they say:

"Endorsement of free markets also predicted the rejection of other established scientific findings, such as the facts that HIV causes AIDS and that smoking causes lung cancer."

This is all false, and the paper should be retracted. It should've been retracted by the authors already.

First, the title. It's not metaphorical or anything. It plainly describes a relationship between believing the moon landing was a hoax and believing climate science is a hoax, even specifying a causal direction. These are variables that were measured in this study – again the title is not a metaphor, is quite specific and descriptive. What do the data say? Well, out of 1145 participants, only 10 believed the moon landing was a hoax (likely including fakes.) We'll pause here to note that 10 in this context is essentially zero, and we couldn't validly infer links between moon hoaxism and anything else from that number. But it's worse – a majority of these 10 reject the idea that climate science is a hoax – 7 out of the 10. Only 3 participants out of 1145 endorsed the moon hoax and the climate science hoax.

Therefore, the title is false. That's a big problem. (We shouldn't decompose those 10 people, and I do so only out of illustrative necessity – the title would be nuts even if the majority of the 10 believed climate science was a hoax.) The title is not only false, it declares the opposite relationship suggested by the authors' trivial data for it. If the authors meant to say something about moon hoaxism based those 10 people, a more accurate title, given their data, would be: "NASA Faked the Moon Landing–Therefore (Climate) Science is Reliable."

The title being wildly false is bad enough, but it's made worse by the fact that it slanders millions of people as believing that the moon landing was a hoax. They don't believe any such thing, according to the authors' own data. Slandering one's participants is a serious ethical breach.

That should be enough to retract – it was just made up. No scientist should ever get away with that, with just making stuff up under the banner of science. But there's more...

Now to the abstract...

Out of 1145 participants, only 16 reject the claim that HIV causes AIDS. Out of 176 free marketeers*, only 9 reject the HIV-AIDS link – that is, 95% agreed that HIV causes AIDS. There were fake participants in the study that can be identified by their response patterns – those trivial 9 and 16 figures will drop when we delete the fakes.

Out of 1145 participants, only 11 reject the idea that smoking causes cancer. Out of 176 people who endorsed free markets, only 7 rejected the claim that smoking causes cancer. 96% of them agreed that smoking causes lung cancer. (They should've said "increases the risk of", because some intellectual types will be sticklers on that, might struggle with their answers – see the footnote.)

They didn't disclose this in the paper. They didn't tell us. Nor did they clean the fakes from their data, fakes which end up driving some of the key results. They did the opposite – they claimed effects based on these numbers, in their headline, their abstract... Their effects were artifacts of improper statistical inferences, driven by variance between "agree" and "strongly agree" answers to those science items – the opposite of "rejection".

Let's look at the whole picture. This was a scattered online study posted at political climate-related websites. Anyone in the world could participate, and we have no idea who they were. Here's the endorsement count for each of the conspiracies in their conspiracy variable, and the rejection count for the HIV and smoking facts. This is out of 1145 participants:

Oklahoma City bombing conspiracy: 289
JFK assassination conspiracy: 247
Coca-Cola conspiracy (don't ask): 151
Pearl Harbor attack conspiracy: 146
MLK assassination conspiracy: 90
New World Order conspiracy: 70
9-11 attacks conspiracy: 69
Roswell UFO conspiracy: 47
SARS disease conspiracy: 42
Area 51 UFO conspiracy: 35
Princess Diana assassination conspiracy: 25
Reject HIV-AIDS link: 16
Reject smoking-lung cancer link: 11
Moon landing hoax conspiracy: 10

Why is their title based on the variable for which they have the least data, essentially no data?

Why in the abstract are they linking free market views to incredibly damaging positions that again, they have no data for?

The answer is that they ignored the trivial numbers and ran linear correlations on data for which it was not appropriate. Their analyses are picking up on variance between people who agree that HIV causes AIDS and those who "strongly agree" that HIV causes AIDS (the same for the smoking item, and for the moon item, in reverse.) The items all used this substantively dichotomous 4-point scale:

strongly disagree (1); disagree (2); agree (3); strongly agree (4)

In the abstract, where they say endorsement of free markets predicts "rejection" of established scientific facts, like that HIV causes AIDS and that smoking causes lung cancer, they are converting agreement with those facts into "rejection". This is an egregious example of an all too common practice in social science – using linear correlation statistics to conflate direction with destination. 

This is what happened: Almost everyone agreed that HIV causes AIDS, and smoking causes lung cancer. The dissenters (11, 16) are essentially zero in the context of a wide-open, contaminated-by-fakes online study with 1145 participants. And, most free market endorsers even said they "strongly" agreed with these basic science facts, as did most people who disagreed with free markets. But "most" varied significantly between the two groups. The former was 65%, and the latter 89%. So about 30% of free marketeers said Agree instead of Strongly Agree, which is well within their rights, while about 10% of anti-free market people (the vast majority of the sample) did so. (Some people dispositionally avoid the extreme points on survey items, the stronglys and extremelys – I have no idea if there's a link between free market views and avoiding extreme opinions, but the free market scale here was terrible so we probably can't take anything from this.) That difference in levels of agreement, in saying Agree vs Strongly Agree drove the statistics they reported. When they get a negative correlation between free market endorsement and agreeing with those items, it's driven by that variance in agree vs strongly agree. They took that negative correlation and said free market endorsement predicts "rejection" of those facts, which is simply false.

Implicitly, in their language, they took this non-continuous 4-point scale of disagreement vs. agreement and converted it to a continuous (and inverse) "rejection" variable, where 3 (agree) is a greater level of "rejection" than 4 (strongly agree). We can't do that (and I have a methods paper coming soon that delves into such issues more deeply.) We see it sometimes in social science, but this case is egregious because they had virtually zero cases of actual rejection, they didn't tell us, and they falsely linked free market views to rejection of facts that those participants very strongly endorsed. Recapping:

1. 95% of people who endorsed free markets agreed that HIV causes AIDS. (167/176) Some of the 9 dissenters are fakes.
   
2. 99% of people who disagreed with free markets agreed that HIV causes AIDS. (962/969)
   
3. The sample is heavily skewed toward people who disagree with free markets.
   
4. As we saw above, almost no one disagreed with the HIV item, or the smoking item, or endorsed the moon hoax. Given the wide open and vulnerable nature of study, we have to refrain from making inferences from 7 or 9 people out of 176.
   
5. We cannot quibble with Agree or Disagree. No one is obligated to "strongly" agree or disagree with our scale items. (Simple agreement / disagreement might even be more rational, given that a lot of participants would not have heard of some of these conspiracies.) If people think there's something interesting in the fact that 89% of anti-free market people selected Strongly Agree, while only 65% of free market endorsers selected it, while essentially everyone in both camps chose some level of positive agreement, they should research it, do some IRT and so forth. But there's nothing in this data that will tell us anything about that.
   

This was a wide open online study mostly posted at environmentalist websites. We have no idea who the participants were – they could be from anywhere in the world, no demographics were reported. The paper implies some of them were minors, that 10 was the cutoff age. (And their subsequent paper at PLOSOne has self-reported minors in the data – 7 of them. Confusing.)

We know there are fake participants, or we should be very confident that there are. Let me introduce you to the all 1s guy (or girl) – this person answered 1 to every scale item, even those that were reverse coded (a 1 meant the opposite for those items, was converted to a 4). Since 3 of the 5 free market items are reverse scored, this person counted as a net endorser of markets, and of course gave a 1 to the HIV and smoking items. They show up as high leverage, naturally, and there are other likely fakes. When you only have 11 or 16 people who reject these facts, fakes are a very big deal. 11 or 16 out of 1145 is already walk-away data. Any whiff of fakes, which was so easy here, and we definitely don't make claims aboout any political camp "rejecting" these facts. (These issues clearly aren't in play out there in society – very few people disagree with the HIV-AIDS link or smoking and lung cancer. We knew that already. We have background data, polls, etc.)

A logistic regression would be the appropriate analysis here, if there was significant data, since these are substantively dichotomous variables, and heavily skewed. But since we only have 16/1145 rejecting the AIDS item, 11/1145 rejecting the smoking, any regression is irresponsible. When you run a wide-open online study that anyone in the world can participate in, or sabotage, you cannot make inferences from such trivial numbers. It doesn't matter if a logistic regression of the HIV or smoking items on free market endorsement (as a continuous variable) shows a significant effect. It's not significant – p-values don't matter if you have no data. We know a search for fakes will reduce those trivial numbers – there's just no way the claims in the title, abstract, or body of the paper can be supported with this data (and eliminating the likely fakes eliminates the significant logistic regression coefficients – try it.)

The title is wildly false as a desciption of any reality or profile of person – it's also quite defamatory and unethical. There's no data, no analyses, to support it. (10 moon hoaxists total, including fakes, and only 3 of them endorsed a climate hoax. That means the title is false.) The abstract is false in linking free market endorsement to "rejection" of these uncontroversial facts, and will only become more false if we clean the data. The body of the paper repeats these false associations, speaks of "denial", and conceals this stark data in overly complex SEM models that we will not be able to validly reproduce (longer story.) Virtually none of their analyses will survive evaluation. This paper must be retracted. The fact that that these false links are very damaging to people, to large swaths of the population, makes an even stronger case for retraction (if we needed one.) When a headline is false, when an abstract is false, when a paper is false, we must retract that paper. When it smears innocent people and falsely attributes ludicrous and damaging beliefs to them, there is no excuse to not vacate it. It's unethical to invite people to participate in a study and then do this to them.

---------------------------------------------------------------------------------------------------------------------------------------------

NOTE: There's a lot of older content below. I'm iteratively updating this post, and will make it much shorter and cleaner soon. I just don't have a lot of time right now – my own research is more important to me. I'm not happy with the quality of some of the writing below, but I'm keeping all the older content for now because there are some non-redundant points below (although much of it is redundant.)




The researchers have had two years to come clean, to admit that there was no significant data regarding belief in the moon hoax or rejection of the HIV-AIDS or smoking-lung cancer links. They've had two years to remove the very-likely-to-be-scam participants identitifed by people who have looked at the data, which will further reduce those trivial numbers at the bottom, and they've not done so. I'm not sure they even talk about it. Lewandowsky still won't tell the public that fewer than 10 participants rejected the moon hoax or HIV and smoking claims – after all this time Lewandowsky is still evading those basic facts and distracting his readers by bragging about the p-values of invalid statistics. Pearson correlations on essentially dichotomous data skewed 1135 to 10? The paper should have been retracted by the authors long ago.

We're not going to be able to do anything with 11 and 16 people out of a sample of 1145. Or with 7 or 9 people out of a sample of 176. Moreover, when you have 1145 participants, and 11 or 16 or 10 or 7 agree or disagree with something, we don't know what that is, even if it was a cleanly controlled study instead of the scattered online study it was – we certainly don't know it's "denial", as they assume in the paper. It could be several different kinds of error. You cannot declare a link to anything based on such numbers. To do so is a scam, made worse by the fact that they didn't tell us, and reported none of the diagnostics that you would need in this case. It's amazing that they didn't tell us. It's incredible that they claimed an effect.

Moreover, their analytical methods are wrong for this data – they shouldn't be treating these variables as continuous, or using linear regression based methods. If they had screened out the scam participants and run a logistic regression, the effects would disappear (I've done it.) This would matter more if they had data.

One participant answered 1 to every item, including conspiracy endorsement and the HIV and smoking facts (which are in the opposite direction of the conspiracy items), then 0 for every question asking for an estimate of the consensus on each of HIV, smoking and CO2. Another participant answered 1 (strongly disagree) to all three science claims, and then gave extremely high consensus estimates for each of them (95%, 95%, and 98%). Perhaps he or she was showing remarkable, dispassionate integrity, acknowledging every consensus as a matter of descriptive reality, but strongly disagreeing with every single one. Perhaps he or she is a neo-Kuhnian, but I have my doubts. In any case, when you're making impossible inferences from 7, 9, 11, or 16 observations in a sample of 1145 online participants, planted individual cases among those 7 or 11 should matter a great deal to you (in this alternate fantasy universe where we would even do anything with such trivial numbers), and it's not good that the authors apparently did no checking. How can you publish data that is so obviously contaminated?

Imagine we had 70 of 176 free marketeers rejecting the smoking-lung cancer link. In that case, we might be able to generate a valid correlation, depending on some diagnostics, but it would be an example of a recurrent, although not common, problem – reporting a linear correlation or regression to imply that people high on one variable are high on the other, when the majority of them are not. For example, self-esteem can predict bullying, but the majority of high self-esteem people, perhaps a very large majority, aren't bullies. Our way of describing effects is problematic, and certainly the way they're understood by the public is – it relies on a rationalistic, proprietary and often confusing definition of "predict". But here we don't even have that problem. We don't have a minority of 70 out of 176 free marketeers driving a misleading correlation. We have 7. We have nothing, and they kept that from us.

(Note that they also say they allowed minors to participate. "An additional 161 responses were eliminated because the respondent’s age was implausible (< 10 or > 95 years old)..."? Are there 10 and 12 and 16 year olds in this study? They imply it. Misprint? Note also that if 161 participants claimed to be less than 10 or older than 95, this probably tells us a lot about the sketchiness of this study. How many of you have ever had more than 10% of your sample give an out-of-range age? Actually, the fact that 161 people gave <10 or >95 ages suggests some number would've given ages of 10 - 17, and are still in the data. Maybe the 10 was supposed to be 18, but this paper has lots of weird things like that that suggest it wasn't read carefully. Why do political hit pieces get waved through like this?)

Let's go back a bit. Lewandowsky, Oberauer, Gignac titled their paper "NASA faked the moon landing—therefore, (climate) science is a hoax: an anatomy of the motivated rejection of science."

There were 10 participants, in their sample of 1145, who thought the moon landing was a hoax, and most of them did not think climate science was a hoax. The title describes a reasoning process, where a person starts by reminding themselves that the moon landing was a hoax, and "therefore" concludes that climate science is a hoax. It's a directional, even causal claim about the relation between the one belief and the other. There is no evidence of such a phenonemon in their data, even a bare association. This is fraud. Then they link free market views to rejecting the HIV-AIDS link and the smoking-lung cancer link, more bizarre beliefs that virtually no one in the study endorsed.

Back to our list. Sample: 1145 participants, sketchily recruited from politically heated climate change sites.

Oklahoma City bombing conspiracy: 289
JFK assassination conspiracy: 247
Coca-Cola conspiracy (don't ask): 151
Pearl Harbor attack conspiracy: 146
MLK assassination conspiracy: 90
New World Order conspiracy: 70
9-11 attacks conspiracy: 69
Roswell UFO conspiracy: 47
SARS disease conspiracy: 42
Area 51 UFO conspiracy: 35
Princess Diana assassination conspiracy: 25
Reject HIV-AIDS link: 16
Reject smoking-lung cancer link: 11
Moon landing hoax conspiracy: 10


They didn't talk about the conspiracies for which they had usable data, presumaby because they didn't pan out for the drive-by shooting. Some of them are negatively correlated with free market endorsement (e.g. the MLK, 9-11, and the OKC bombing conspiracies), and those correlations might have some hope of being valid, of having meaningful data underneath. Instead, they ignored those and reported completely invalid associations based on 10 or 11 participants. What else did they leave out? The Iraq War conspiracy item – they don't even mention that it was ever collected. Any guesses on how that one bent? When they wrote their title, they had a list of at least 13 conspiracy beliefs in their data, things that might be related to climate skepticism or free market views. They chose to talk about the conspiracy that had the lowest endorsement of everything in their dataset, so low that it's not even there. And most of those not-even-there moon hoaxists, 7 out of the 10, did not endorse a climate hoax (if we even want to talk about such trivial numbers, which we're going to decide we don't.) And it's such a damaging conspiracy to be associated with.

When you have no data, you tell no story. If you're talking about things that are incredibly damaging, beliefs and positions which would marginalize people in public life, you definitely tell no story when you have no data. This is the worst malpractice I've seen. It's especially bad given that these authors did not tell us about their data – they did not tell us they only had 10 moon hoaxists out of 1145 people, or 11 smoking-cancer doubters, or 16 HIV-AIDS doubters. They wrote the entire paper without telling us. And they reported no diagnostics. If fact, they reported no descriptive statistics at all, and their factor loadings are false and uniform – they don't tell us that four items in the conspiracy variable don't load well at all.

Let me touch on something else. When 16 out of 1145 people dispute that HIV causes AIDS, we don't know what that is, as I mentioned before. We have no reason to assume it's rejection/denial. It could be error. It could be the simple error of selecting the wrong response. It could be an error of knowledge. People don't know everything. In any large sample, there will be people who think the sun orbits the earth, who Abe Lincoln was President during WWII, who don't know who America won its independence from. Not everyone is an academic, recurrently exposed to scientific truths. We know from survey data that people have all sorts of quirks like this. Let's even say someone is aware of the link between HIV and AIDS. I would bet that in a large sample, you're going to find some people who think AIDS causes HIV. That's very easy to imagine -- they have the link, but somehow got the direction wrong. Some people might see "HIV causes AIDS" and think it's a trick question with the wrong order, and thus submit their disagreement. That's so easy to imagine. There's no way we can point to 10 or 11 people out of 1145 and say we know what their answers represented.

Surprisingly, climate skeptics got mad about this paper, perhaps because > 97.8% of those who think climate science is a hoax reject the moon hoax idea in Lewandowsky's own data, placing them squarely in the mainstream of humanity. So, Lewandowsky, Cook, Oberauer, and Marriott (2013) wrote a follow-up hit piece that was all about their critics. They wrote a paper that was about the critics of the first paper, the one we've just debunked. It wasn't enough to lie about people and smear them as believing things they definitely do not believe. He needed to take another swipe. The journal, Frontiers in Psychology, wisely ended up retracting that paper, which is exactly what should happen to this fraud here.

A lot of overpolitical social psychologists have rationalized discrimination against conservatives by claiming that they're loons who oppose science. Many of these lazy views are based on "research" like we have evaluated here – scam studies, in other cases rigged in more subtle ways. People need to re-evaluate their beliefs about conservatives, to be sure they don't go around saying things that are false, and should regulate their discriminatory impulses.

If we wanted to identify the people disconnected from reality in this picture, it's the social psychologists, the reviewers, the journal editors who read passages like "Endorsement of free markets also predicted the rejection of other established scientific findings, such as the facts that HIV causes AIDS and that smoking causes lung cancer" and didn't stop and wonder at the plausibility of such a thing, who evidently thought lots of people reject the HIV-AIDS link, or lots of people reject the smoking-cancer link, and that these beliefs go with endorsing free markets. They didn't bother to check. They read this title: "NASA faked the moon landing—therefore, (climate) science is a hoax: an anatomy of the motivated rejection of science", and presumably thought this was plausible, that a lot of people think the moon landing is a hoax, and that this was the springboard for climate skeptics (or hoaxists). They didn't check. If I wanted to talk about disconnect from reality, "denial" and the like, I wouldn't focus on the participants.

If with a sample of 1145 participants, you only have 10 moon hoax endorsers (including fakes; maybe 6 real ones), most of whom are not climate skeptics, and you want to be able to link moon hoax endorsement to your political foes, you have some options:

1. You can go in and fabricate a hundred moon hoax endorsements, make it correlate with climate skepticism or tax cuts or whatever, and then say "NASA faked the moon landing—therefore, (climate) science is a hoax: an anatomy of the motivated rejection of science."

2. You can leave the data alone and just say "NASA faked the moon landing—therefore, (climate) science is a hoax: an anatomy of the motivated rejection of science."

As a field, we define the first as fraud. We don't have a ready classification for the second. I don't think we should distinguish much between the two – we should view them both as fraud. The second is just lazier than the first. The authors should be investigated, obviously the paper retracted – it's a scam paper. It's not real. The title is false. The abstract is false. Many of the claims in the paper are false. The reality of the data is heavily concealed by overly compex statistics and SEM models. The authors made claims they had no right to make, and slandered their participants and the political camps to which they belong. Some of the participants might even have gotten Happy Meals in exchange for their participation. We have no idea what happened in this study. We just know there's no data pertaining to the headline or the key smears in the abstract.

This was an awful thing to do. It was damaging to innocent participants. It's unethical to do this to your participants. It is wildly unethical to invite people to participate in a study, and then do this to them. They are helping us. They are volunteering to participate in scientific research. They've take time out of their lives to help us out. And in return, we slander them? We tell the world that they believe things that they do not believe? What Lewandowsky and colleagues did here was despicable. Why would anyone participate in a social psychology study if this is what we do to them? Why would anyone participate in our research if our goal is to marginalize them in public life, to lie about them, to say that they think the moon landing was a hoax, to say they don't think HIV causes AIDS, to say they don't believe smoking causes lung cancer – when none of those things are true. Do we hate our participants?









Stats Primer:

The authors of this study conducted incorrect analyses for all their effects. I don't know what's going on at Psych Science – the stats here were amateurish and deceptive. First, the data here was go home data. If you want to link moon hoax nonsense to your political foes, and in 1145 participants there are only 10 people who endorse that hoax (fewer after you delete the fakes), only 3 of whom endorse the climate hoax idea (fewer after you delete the fakes), you go home. It's over. If you see similar trivial numbers for the HIV and smoking items, you bail. Go to a show, discover a new restaurant, think about the design of your next study. Those are go home numbers – you definitely don't write it up.

But let's say there was actual data in such a study, let's say we lived in a universe where free marketeers had deep doubts about the link between HIV and AIDS, because you know, free markets, or something.

The key DVs here were wrongly treated as continuous variables. A 4-point scale of disagreement/agreement is not a continuous variable. A scale of disagreement and agreement is special. It's not like temperature or cholesterol. Disagreement and agreement are opposites, and a 4-point scale is virtually binary. For example, agreeing that HIV causes AIDS (3 on the scale) is very, very different from disagreeing that HIV causes AIDS (2 on the scale). But strongly agreeing that HIV causes AIDS (4) is not as different from agreeing as agreeing is from disagreeing. The difference between 2 and 3 is far greater than that between 3 and 4.

It goes even deeper than that. It's not just that the 1 point difference between 2 and 3 is much larger in true quantity than the 1 point difference between 3 and 4. It's not just different in true quantity, it's different in kind. It's substantively different – agreeing with something is fundamentally different from disgreeing with something, in most cases.

At minimum, the responsible practitioner will treat such scales as ordinal, and perform ordinal regression. However, when there are only 4 points, a mere 2 for disagreeement and 2 for agreement, the responsible practitioner will in many cases treat these variables as dichotomous, since there is a profound substantive difference between agree and disagree, there is no midpoint or neutral option, and there is so little texture in a 4 point scale. Failure to treat them as dichotomous risks making wrong inferences from variance between levels of agreement, or levels of disagreement, where there is little variance across the scale. That is, you might have a situation where most or all of the variance on one side of the scale, for example between people who agree and people who strongly agree.

That's exactly what happened here, and it happens a lot. Social scientists use linear correlation invalidly sometimes, and make false inferences. (More on this in an upcoming journal article.) What social scientists sometimes do is like standing on a street corner in St. Louis, seeing a car heading west, and proclaiming "That car is going to Los Angeles". The misuse of linear correlation statistics (including SEM models so structured) is to conflate direction with destination, and it leads to a lot of false inferences and assertions.

(That there are only 4 points is not the central in the above. The central issue is the nature of the scale – it is a disagreement/agreement scale, with 4 points, which in combination suggests we treat it as dichotomous.)

The authors of this paper treated these 4-point scales as continuous, which obscured the fact that there was no significant variance across the scale – it was all on one side for the items they advertised. On the HIV item, all the action was between agree and strongly agree. On the moon item, it was all about those who disagreed and those who strongly disagreed.

When the variables in question are extremely serious matters that would damage the reputation of those who are associated with a particular position on them, the responsible practitioner will have even more reason to treat answers as dichotomous, and not hang people by the noose of their failure to click the "strongly" option on the enlightened side of the issue. No one is under any obligation to say that they strongly agree or disagree with something, and in most cases, with 4-point scales structured as these were, the researcher should be satisfied with simple agreement/disagreement.

When the distribution is so skewed that only 6 or 9 people out of 1145 endorse a view, while 1139 or 1136 reject the view (say, the moon hoax), there is even less justification to treat it as a continuous variable, and more cause to treat it as dichotomous.

The correct analysis here is logistic regression, where you would code disagreement/agreement as dichotomous, here 0 for moon hoax disagreement, and 1 for agreement. (Both levels of disagreement would be coded as 0, and both levels of agreement as 1).

Logistic regression still assumes some things about your predictor. Their free market ideology predictor was extremely non-normally distributed, heavily skewed toward people who rejected it. They did not disclose this either. They told us nothing about their data. I think the SEM assumptions would be a bigger problem – SEM assumes multivariate normality, which they did not satisfy. 85% of their participants rejected free market ideology to some degree (mean scores below the neutral midpoint of 2.5). Their acceptance of other sciences variable (two items apparently, the HIV and smoking items) lies entirely on the acceptance end, with for example only 11 out of 1145 rejecting the smoking link at any level. You would have to transform your data in such circumstances, which they apparently did not do (and again, this convsersation assumes they had meaningful data, which they did not – if after cleaning, you have 6 moon hoax people in over 1100 participants, you don't need to transform your data. You just need to go home.)

It's also unclear why they have only two items in an SEM latent variable. (There are only two items that pertain to other sciences – just the HIV and smoking items. I assume they didn't throw the consensus variables in there, since those are just descriptive assessments of percentage of consensus in various fields, and do not imply acceptance or rejection of any consensus – a number of participants rejected the HIV item for example, while later giving a 9X% figure for the consensus there.)

The SEM analyses here will have to be thrown out. The conspiracy ideation variable will also have to repaired. In the factor analysis, many items don't load well on F1, and would be discard by normal practice. (The paper should be retracted since it flatly says things, incredibly damaging things, that aren't true, in its very title and abstract, and the authors never disclosed the nature of their data, the trivial number of relevant cases, or even bothered to clean out the fake participants. It's incredible that they're so comfortable publishing fake data, and that even with the fakes, they never had anything to talk about.)

Their New World Order item is defective, and they probably lost some movement there because of it. They refer to a secret group called the New World Order. The NWO is an outcome, a state of affairs, a (new) world order, to those who promote the idea. It's not a group of people.

(Every ten minute increment spent looking at this data reveals serious problems. Look at the factor analysis for their conspiracy ideation construct. Try to reproduce their EFA, see what the loadings are, what you'd retain, and what happens to the predictions of that variable when you remove the bad items... Try to reproduce the SEM... None of this ultimately matters, since it needs to retracted for making false and defamatory claims or insinuations in the title and abstract. But they made very simple data much more complicated than it had to be, with bizarre SEM models concealing the fact that there's nothing to talk about. They even say incredible things in the discussion, as though free marketeers actually reject the HIV-AIDS thing and the smoking thing, stuff like this: "The fact that HIV causes AIDS, by contrast, seems of little relevance to one’s views on the free market at first glance. However, the association between ideology and rejection of the link between HIV and AIDS is in good agreement with our finding that perceived consensus and acceptance of science were associated via general factors that transcended pairwise correlations." It's incredible to go on talking like that, like you're actually talking about a thing, a phenomeon, when 95% of those people agree that HIV causes AIDS – there's nothing there, there's nothing to talk about. The data is here.)

(Scene: A dead woman lies on top of a car, having fallen or jumped from a tall building. She's dressed in an angel costume. Two detectives arrive at the scene.)

Charlie Crews: It’s a dead angel.

Dani Reese: It’s a dead woman with a pair of fake wings.

Charlie: How do you know?

Dani: Well, I know the wings are fake because I can see the harness right there.

Charlie: Maybe she’s a real angel with fake wings.



I've been meaning to share this for a long time. It's dialogue from a TV show called Life (Season 1, Episode 5, aired October 24, 2007.) I love it so much. It captures a beautiful sense of life, and for people like me who are absurdly geeky about clean reasoning, fallacies, etc, it's actually an example of good logic, suggestive of how you'd avoid the representativeness heuristic (if we assumed angels exist, which I don't, but I don't care – I love the sense of life in that line, the metaphysical optimism.)

What a summer. I've also called for the retraction of the following paper:

Lewandowsky, S., Oberauer, K., & Gignac, G. E. (2013). NASA faked the moon landing—therefore, (climate) science is a hoax: an anatomy of the motivated rejection of science. Psychological Science, 24(5), 622–633.

The title draws a link between belief in the moon landing hoax and belief that climate science is also a hoax. The title is false. The relationship between those variables is the opposite of that reported. Worse is how the title even suggests a causal relationship here, that climate science hoaxists (which will likely be encoded as skeptics to many readers) got there by starting with the moon hoax belief.

Dr. Lewandowsky has already admitted that there is no link, on his blog here:

"Let's consider the signal vs. noise issue first. We use the item in our title, viz. that NASA faked the moon landing, for illustration. Several commentators have argued that the title was misleading because if one only considers level X of climate "skepticism" and level Y of moon endorsement, then there were none or only very few data points in that cell in the Excel spreadsheet.
Perhaps. But that is drilling into the noise and ignoring the signal. The signal turns out to be there and it is quite unambiguous: computing a Pearson correlation across all data points between the moon-landing item and HIV denial reveals a correlation of -.25. Likewise, for lung cancer, the correlation is -.23. Both are highly significant at p < .0000...0001 (the exact value is 10 -16, which is another way of saying that the probability of those correlations arising by chance is infinitesimally small).
What about climate? The correlation between the Moon item and the "CauseCO2" item is smaller, around -.12, but also highly significant, p < .0001. Now you know why the title of our paper was “NASA faked the moon landing—Therefore (Climate) Science is a Hoax: An Anatomy of the Motivated Rejection of Science.” We put the "(climate)" in parentheses before "science" because the association between conspiracist ideation and rejection of science was greater for the other sciences than for climate science."

That's his explanation for the title? It's the most incredible example of tortured reasoning I've ever seen. I doubt I'll ever see anything like this again. He says he put climate in the title because it was the smallest effect, outweighed by others? The fact that there are parentheses around it is supposed to alert readers that it is the lesser effect, and readers will know that two unmentioned variables – HIV and lung cancer conspiracies – were more significant, because we all know that parentheses means "the enclosed effect is smaller than the effects for HIV and lung cancer research" in modern English. One wonders why, given all this concern for "signals" and noise, the parentheses in the headline didn't say something about HIV or smoking/lung cancer, since those were the effects, not climate hoaxism. 

It gets much worse. There was never any effect. The CauseCO2 item he cites is not the climate hoax variable – it's just one random item out of six items to do with climate, and the correlation is a scam, as we will see below. I was curious why Lewandowsky never mentions the actual climate hoax variable, CYClimateChange, since his title explicitly references climate hoaxism, and CYClimateChange is the item – the only item – that asks participants if they believe climate science is a hoax. So I looked in the dataset (available here). Of 1145 participants, 3 agree with – at any level – both the moon hoax and the climate hoax. To repeat, 3 out of 1145 participants fit the smear in the headline. The items are based on a 1 to 4 scale of disagreement/agreement, from strongly disagree to strongly agree. My count of three qualifying participants includes anyone who gave a 3 or 4 on the moon hoax and a 3 or 4 on the climate hoax. There is no effect driven by mutual endorsement of these hoaxes. The phenomenon stressed in the headline is simply not in this data.

Note that in the entire dataset, only 10 participants agree with the moon hoax at any level (3 or 4), a profile that is of course central to the effect claimed in the headline.

134 of the participants, 11.7% of the sample, agree with the climate hoax idea, on any level. Of those, 131 disagree with the moon hoax, and 122 of those strongly disagree with it (a 1 on the scale). That is, 97.8% of those who agree with the climate hoax idea disagree with the moon hoax, in staggering contradiction of the headline that was snapped up by the media all over the world.

The truth is the exact opposite of the headline. If someone believes that climate science is a hoax, there are extremely likely to reject the idea that the moon landing was a hoax. And of the ten people who believed in the moon hoax, most did not endorse a climate hoax (7 out of 10 rejected it).

Also note that the skew of this data undermines our ability to obtain any meaningful correlations from it. We will not be able to generate valid correlations when only 10 out of 1145 participants endorsed the hoax, only made worse by the fact that the response scale ranges from 1 to 4. Anyone who would report Pearson correlations in that circumstance, on that variable, and tout their "significance", demonstrates a profound incompetence with elementary statistics and/or or fraudulent intentions. His bluster about the "signal and the noise" was complete nonsense, presumably meant to convey the cues of statistical competence and scientific authority, and would make no sense to any qualified practitioner of quantitative psychology, given the characteristics of this data. And regarding that CauseCO2 item he cited, there are only 4 participants who both disagreed that CO2 causes warming and agreed with the moon hoax idea. 97.5% of those who disagreed with the CauseCO2 item, also disagreed with the moon hoax idea. His correlation there was a scam, a type of artifact of linear correlation that I'll go into another time. This requires logistic regression after you've removed all the outlier or high leverage items. 

I offer this definition of fraud: knowingly making a claim that your data do not support, where your (authentic) data do in fact test the claim (i.e. the answer is in your data, whatever the answer is). Fraud can be granular, at the level of Excel spreadsheets and invented statistics, or at the headline level. This is fraud, at the headline level. The headline is false, declaring the exact opposite of the truth. It reverses the relationship given by the data (and the data here will not survive a light breeze, much less things like Cook's distance).

This was an unbelievably awful thing to do, the worst case of malpractice I've seen. It established a link, in the minds of biased journalists and perhaps millions of news consumers, between this moon hoax nonsense and climate skepticism. The apparent malice and pettiness behind such malpractice is deeply disturbing, that anyone would want to do that to millions to people, to marginalize them in public life, by falsely linking them to one of the most insane hoax conspiracies out there (NASA actually landed men on the moon six times – trivia tidbit.)

Something is going on here. Lewandowsky is associated with that deceptively named "SkepticalScience" crew who pushed out the junk 97% study. This is very disturbing. You've got people running a junk study to try to inflate the consensus with the most invalid method we could think up, while their friends in social science are publishing fraudulent studies that smear people who doubt that same (inflated) consensus. Elegant. This is garbage, and we may need a dump truck to clean up social science. I'm looking foward to Chris Mooney's correction/retraction on the Lewandowsky paper, and his future curiosity about actual science.

In the long arc of our civilization, I think the credibility of scientists and journals will likely matter more in influencing public action than whether we can cite "97%" vs. say 91%. The perverse injection of simplistic partisan politics into science needs to come to a speedy end. Journals and scientific bodies, like the politicized AAAS, are doing enormous damage, if we're looking past next week.

UPDATE SEPTEMBER 11, 2014:

(This post was my initial take. For a better and more up-to-date report, go here. For follow-up, go here.)

The paper includes a bunch of psychology studies and public surveys as evidence of scientific endorsement of anthropogenic climate change, even a survey of cooking stove use. This vacates the paper. Lots of things vacate the paper, such as its inherently invalid and untrustworthy method, and the false claim that they used independent raters. But I think we will have broad agreement that the inclusion of psychology papers and public surveys vacate the paper. The world thought this was about climate science. This is ridiculous, but trivially predictable when you have human raters who have an obvious bias with respect to the subject of their ratings, who desire a specific outcome for the study, and who are empowered to deliver that outcome via their ratings. What happened here is exactly why we could never accept a "study" based on such a method.

The following papers were rated as endorsement and included in their 97% figure. Dana Nuccitelli even wanted to include a psychology paper about white males and denial as evidence of scientific endorsement. It's jaw dropping that someone who is supposed to inform the public on science would want to do that to the world, to generate a consensus figure based on studies that have no bearing on the consensus. There will be more such papers for those willing to invest time in this scam paper, and I haven't listed all that I found yet. I'll write this story up in a separate post when I have time, and for a news magazine. The broader ethics disaster here is going into a Nature submission:

Chowdhury, M. S. H., Koike, M., Akther, S., & Miah, D. (2011). Biomass fuel use, burning technique and reasons for the denial of improved cooking stoves by Forest User Groups of Rema-Kalenga Wildlife Sanctuary, Bangladesh. International Journal of Sustainable Development & World Ecology, 18(1), 88–97.  (This is a survey of the public's stove choices in Bangladesh, and discusses their value as status symbols, defects in the improved stoves, the relative popularity of cow dung, wood, and leaves as fuel, etc. They mention climate somewhere in the abstract, or perhaps the word denial in the title sealed their fate.)

Boykoff, M. T. (2008). Lost in translation? United States television news coverage of anthropogenic climate change, 1995–2004. Climatic Change, 86(1-2), 1–11.

De Best-Waldhober, M., Daamen, D., & Faaij, A. (2009). Informed and uninformed public opinions on CO2 capture and storage technologies in the Netherlands. International Journal of Greenhouse Gas Control, 3(3), 322–332. 

Tokushige, K., Akimoto, K., & Tomoda, T. (2007). Public perceptions on the acceptance of geological storage of carbon dioxide and information influencing the acceptance. International Journal of Greenhouse Gas Control, 1(1), 101–112.

Egmond, C., Jonkers, R., & Kok, G. (2006). A strategy and protocol to increase diffusion of energy related innovations into the mainstream of housing associations. Energy Policy, 34(18), 4042–4049.

Gruber, E., & Brand, M. (1991). Promoting energy conservation in small and medium-sized companies. Energy Policy, 19(3), 279–287. 

Şentürk, İ., Erdem, C., Şimşek, T., & Kılınç, N. (2011). Determinants of vehicle fuel-type preference in developing countries: a case of Turkey.  (This was a web survey of the general public in Turkey.)

Grasso, V., Baronti, S., Guarnieri, F., Magno, R., Vaccari, F. P., & Zabini, F. (2011). Climate is changing, can we? A scientific exhibition in schools to understand climate change and raise awareness on sustainability good practices. International Journal of Global Warming, 3(1), 129–141.  (This paper is literally about going to schools in Italy and showing an exhibition.)

Palmgren, C. R., Morgan, M. G., Bruine de Bruin, W., & Keith, D. W. (2004). Initial public perceptions of deep geological and oceanic disposal of carbon dioxide. Environmental Science & Technology, 38(24), 6441–6450.  (Two surveys of the general public.)

Semenza, J. C., Ploubidis, G. B., & George, L. A. (2011). Climate change and climate variability: personal motivation for adaptation and mitigation. Environmental Health, 10(1), 46.  (This was a phone survey of the general public.)

Héguy, L., Garneau, M., Goldberg, M. S., Raphoz, M., Guay, F., & Valois, M.-F. (2008). Associations between grass and weed pollen and emergency department visits for asthma among children in Montreal.Environmental Research, 106(2), 203–211. (They mention in passing that there are some future climate scenarios predicting an increase in pollen, but their paper has nothing to do with that. It's just medical researchers talking about asthma and ER visits in Montreal, in the present. They don't link their findings to past or present climate change at all (in their abstract), and they never mention human-caused climate change – not that it would matter if they did.)

Lewis, S. (1994). An opinion on the global impact of meat consumption. The American Journal of Clinical Nutrition, 59(5), 1099S–1102S.  (Just what it sounds like.)

De Boer, I. J. (2003). Environmental impact assessment of conventional and organic milk production.Livestock Production Science, 80(1), 69–77

Acker, R. H., & Kammen, D. M. (1996). The quiet (energy) revolution: analysing the dissemination of photovoltaic power systems in Kenya. Energy Policy, 24(1), 81–111.  (This is about the "dissemination" of physical objects, presumably PV power systems in Kenya. To illustrate the issue here, if I went out and analyzed the adoption of PV power systems in Arizona, or of LED lighting in Lillehammer, my report would not be scientific evidence of anthropogenic climate change, or admissable into a meaningful climate consensus. Concretize it: Imagine a Mexican walking around counting solar panels, obtaining sales data, typing in MS Word, and e-mailing the result to Energy Policy. What just happened? Nothing relevant to a climate consensus.)

Vandenplas, P. E. (1998). Reflections on the past and future of fusion and plasma physics research.Plasma Physics and Controlled Fusion, 40(8A), A77.  (This is a pitch for public funding of the ITER tokamak reactor, and compares it to the old INTOR.  For example, we learn that the major radius of INTOR was 5.2 m, while ITER is 8.12 m. I've never liked the funding conflict-of-interest argument against the AGW consensus, but this is an obvious case. The abstract closes with "It is our deep moral obligation to convince the public at large of the enormous promise and urgency of controlled thermonuclear fusion as a safe, environmentally friendly and inexhaustible energy source." I love the ITER, but this paper has nothing to do with climate science.)

Gökçek, M., Erdem, H. H., & Bayülken, A. (2007). A techno-economical evaluation for installation of suitable wind energy plants in Western Marmara, Turkey. Energy, Exploration & Exploitation, 25(6), 407–427.  (This is a set of cost estimates for windmill installations in Turkey.)

Gampe, F. (2004). Space technologies for the building sector. Esa Bulletin, 118, 40–46.  (This is magazine article – a magazine published by the European Space Agency. Given that the ESA calls it a magazine, it's unlikely to be peer-reviewed, and it's not a climate paper of any kind – after making the obligatory comments about climate change, it proceeds to its actual topic, which is applying space vehicle technology to building design.)

Ha-Duong, M. (2008). Hierarchical fusion of expert opinions in the Transferable Belief Model, application to climate sensitivity. International Journal of Approximate Reasoning, 49(3), 555–574. (The TBM is a theory of evidence and in some sense a social science theory – JDM applied to situations where the stipulated outcomes are not exhaustive, and thus where the probability of the empty set is not zero. This paper uses a dataset (Morgan & Keith, 1995) that consists of interviews with 16 scientists in 1995, and applies TBM to that data. On the one hand, it's a consensus paper (though dated and small-sampled), and would therefore not count. A consensus paper can't include other consensus papers – circular. On the other hand, it purports to estimate of the plausible range of climate sensitivity, using the TBM, which could make it a substantive climate science paper. This is ultimately moot given everything else that happened here, but I'd exclude it from a valid study, given that it's not primary evidence, and the age of the source data. (I'm not sure if Ha-Duong is talking about TCS or ECS, but I think it's ECS.))

Douglas, J. (1995). Global climate research: Informing the decision process. EPRI Journal. (This is an industry newsletter essay – the Electric Power Research Institute. It has no abstract, which would make it impossible for the Cook crew to rate it. It also pervasively highlights downward revisions of warming and sea level rise estimates, touts Nordhaus' work, and stresses the uncertainties – everything you'd expect from an industry group. For example: "A nagging problem for policy-makers as they consider the potential costs and impacts of climate change is that the predictions of change made by various models often do not agree." In any case, this isn't a climate paper, or peer-reviewed, and it has no abstract. They counted it as Implicit Endorsement – Mitigation. (They didn't have the author listed in their post-publication database, so you won't find it with an author search.))

-----------------------------------------------------------------------

Original post below:

Ignore them completely – that's your safest bet right now. Most of these studies use political activists as the raters, activists who desired a specific outcome for the studies (to report the highest consensus figure possible), and who sometimes collaborated with each other in their rating decisions. All of this makes these studies completely invalid and untrustworthy (and by customary scientific standards, completely unpublishable.) I had no idea this was happening. This is garbage, and a crisis. It needs to stop, and those papers need to be retracted immediately, especially Cook, et al (2013), as we now have evidence of explicit bias and corruption on the part of the raters. (If that evidence emerged during the actual coding period, it would be fraud.)

PAUSE BUTTON: This issue has nothing to do with the reality of the consensus, a reality that was evident before this political operation/study unfolded. I am not a "denier", or even a skeptic. I don't know enough, or have an argument that would lead me to be, even a "lukewarmer". There are 7 billion people on this earth, and we're not all sorted into good people and deniers. I'm quite confident that there's a consensus – a large majority – of climate scientists who endorse both that the earth has warmed over the last 60+ years, and that human activity caused most of it. The warming itself is a descriptive fact, not a theory or inference. I'd be quite surprised, amazed, if the basic theory of anthropogenic forcing as a principal cause turned out to be false, and somewhat surprised if AGW turns out to be mild, like 1° C. (Unfortunately, there is little research on scientists' views on the likely severity of future warming. A consensus only that humans have caused warming, a consensus so vague and broad, is not very useful. The Cook study would be unhelpful even if it were valid, which it is not.)

Back to the program...

In social science, it's not uncommon to use trained human raters to subjectively rate or score some variable — it can be children's behavior on a playground, interviews of all kinds, and often written material, like participants' accounts of a past emotional experience. And we have a number of analytical and statistical tools that go with such rating studies. But we would never use human raters who have an obvious bias with respect to the subject of their ratings, who desire a specific outcome for the study, and who would be able to deliver that outcome via their ratings. That's completely nuts. It's so egregious that I don't think it even occurs to us as something to look out for. It never happens. At least I've never heard of it happening. There would be no point in running such a study, since it would be dismissed out of hand and lead to serious questions about your ethics.

But it's happening in climate science. Sort of. These junk studies are being published in climate science journals, which are probably not well-equipped to evaluate what are ultimately social science studies (in method). And I assume the journals weren't aware that these studies used political activists as raters.

Examples of the unbelievable bias and transparent motives of the raters' in Cook, et al (2013) below. These are excerpts from an online forum where the raters apparently collaborated with each other in their ratings. It's worse than that – the first example is evidence of fraud if this was during the operational rating period. If it was during training, it's practice for fraud.

"BTW, this was the only time I "cheated" by looking at the whole paper. I was mystified by the ambiguity of the abstract, with the author wanting his skeptical cake and eating it too. I thought, "that smells like Lindzen" and had to peek."

Let's look at how the paper described their method: "Abstracts were randomly distributed via a web-based system to raters with only the title and abstract visible. All other information such as author names and affiliations, journal and publishing date were hidden."

Hence the fraud issue. Next example:

"Man, I think you guys are being way too conservative. Papers that talk about other GHGs causing warming are saying that human GHG emissions cause global warming.  How is that not an implicit endorsement?  If CFC emissions cause warming because they're GHGs, then CO2 emissions cause global warming for the same reason.  That's an implicit endorsement."

One wonders if a passing bird counts as implicit evidence of the consensus. This is what we call a nonfalsifiable hypothesis.

If this was the live coding period, this is a joke. A sad, ridiculous joke. And it's exactly what you'd expect from raters who are political activists on the subject they're rating. Who in their right minds would use political climate activists as raters for a serious report on the consensus? This is so nuts that I still have a hard time believing it actually happened, that the famous 97% paper was just a bunch of activists rating abstracts. I've called on the journal – Environmental Research Letters – to retract this paper. I'm deeply, deeply confused how this happened. If this is what we're doing, we should just call it a day and go home – we can't trust journals and science organizations on this topic if they're going to pull stunts like this.

Moreover the raters weren't generally scientists, much less climate scientists. One of the raters is a former bike messenger who founded Timbuk2, a company that makes great bags (Rob Honeycutt.) I've got mad props for him for what he's done with Timbuk2 – for anyone who starts their own business and follows their vision. That's very hard to do. But I'm not going to want luggage entrepreneurs to be rating climate studies or interpreting science for the world. I'll buy you a beer any day of the week Rob, but I just can't sign off on this.

Other raters are just bloggers. I don't mean scientists who blog. I just mean bloggers, who are not scientists. Nothing against bloggers – I'm just not feeling that, don't need bloggers to be rating climate science abstracts. Another rater is only identified by an anonymous username – logicman. Who can argue with logicman? Is there a big L on his uniform? Where's emotionman been lately? What's fallacygirl up to? Anyway, probably no one needs to be subjectively rating climate abstracts, but if anyone did, it would have to be climate scientists. Is this controversial in some cultures?

More importantly, I don't care who you are – even if you're a staunch liberal, deeply concerned about the environment and the risks of future warming, this isn't something you should tolerate. If we're going to have a civilization, if we're going to have science, some things need to be non-political, some basic rules need to apply to everyone. I hope we can all agree that we can't seriously estimate the AGW consensus by having political activists rate climate paper abstracts. It doesn't matter whether the activists come from the Heritage Foundation or the Sierra Club, Timbuk2 or Eagle Creek – people with a vested ideological interest in the outcome simply can't be raters.

Also note that anyone who wants to defend this nonsense, who wants to argue that it's fine for political activists to subjectively rate science abstracts – which they won't be qualified to even understand – on issues central to their political activism, needs to also accept the same method when executed by partisans on the other side. If Heartland gathers a bunch of activists to read abstracts and tell us what they mean, all the Cook defenders need to soberly include the Heartland study. The AAAS needs to include the Heartland study in their reports, including it in their average (they didn't do an average, just cherry-picked junk studies.) If a squad of Mormons reads the abstracts of a bunch of studies on the effects of gay marriage, and sends their ratings to a journal, Cook defenders should be cool with that, and should count it as knowledge about the consensus on gay marriage.

Of course, these scenarios would suck. This method perverts the burden – it allows any group of hacks to present their subjective "data", putting the burden on us, on everyone else, to do a bunch of work to validate their ratings. We should never be interested in studies based on activists reading and rating abstracts – it's a road we don't want to travel. Researchers normally get their data by observation – they don't create it, not normally.

We don't need random people to interpret climate science for us, to infer the meaning of abstracts, to tell us what scientists think. That's an awful method – extremely vulnerable to bias, noise, incompetence, and poor execution. The abstracts for many papers won't even have the information such studies are looking for, and are simply not written at the level of abstraction of "this study provides support for human-caused warming", or "this study rejects human-caused warming". Most climate science papers are written at a more granular and technical level, are appropriately scientifically modest, and are not meant to be political chess pieces.

(Updated paragraph: I had incorrectly suggested that they asked authors to self-rate their abstracts, just as Cook's raters did, when in fact they asked them to rate their papers. The failure to hold that variable constant complicates things, but admittedly it would be very difficult for authors to strictly rate an abstract, as opposed to the whole paper they wrote. None of this matters anymore given the much larger issues that have emerged.) There's a much better method for finding out what scientists think — ask them. Direct surveys of scientists, with more useful and specific questions, is a much more valid method than having ragtag teams of unqualified political activists divine the meanings of thousands of abstracts. Interestingly, but not surprisingly, direct survey studies tend to report smaller consensus figures than the abstract rating studies (I'll have more on that later.) The consensus will be strong regardless, so it's especially confusing why people feel the need to inflate it.

In the second part of their study, Cook et al surveyed authors of the papers in their dataset – that's not at all the way to survey climate scientists, since their paper search seems to have bizarre and unexplained results, e.g. it excluded everything Richard Lindzen published after 1997. Their pool of authors is invalid if we don't know whether the search had some selection biases. It's an arbitrary pool – they'd need to validate that search and its results before we could trust it, and they should've done that at the outset. And the fact that they included psychologists, social scientists, pollsters, engineers and other non-climate science or even non-natural sciences in the 97% (as endorsement) makes their survey of authors moot.

(For subjective ratings of abstracts to be a valid and useful method, it would need to be a carefully selected pool of raters, without ideological agendas, implementing a very specific and innovative method, under strict procedures of independence. I can imagine philosophy of science questions that might be anwerable by such methods, based on things like the usage of certain kinds of words, the way hypotheses are framed and results reported, etc. – but much of that could be done by computers. The studies that have been published are nothing like this, and have no hope of being valid.)

NOTE: The Cook, et al data was leaked or hacked a few months ago – I'm confused by what's going on here. Cook allegedly wouldn't release some of his data, and ultimately a bunch of data was hacked or scraped off a server, and it included the raters' online discussion forum. Climate science features far too many stories of people refusing to release their data, and mysteriously hacked data. Brandon Shollenberger has posted the data online. It's amazing that if it weren't for him, we wouldn't know how sketchy the study truly was. There's much more to report – the issues raised by the leaked dataset extend far beyond the quotes above and rater bias.

The University of Queensland has apparently threatened to sue Shollenberger, on some sort of "intellectual property" grounds. Australia is one of my favorite countries, but we need to stand up for him. To the best of my knowledge, he hasn't done anything wrong – he hasn't posted any sort of sensitive information or anything that would violate our core principles of scientific ethics. The identities of the raters were not confidential to begin with, so there was no new disclosure there. He's exposed the cartoonish bias and corruption of the rating process that underlied this "study", and in so doing, he's served the interests of scientific ethics, not violated them.

Even if those online discussions took place during the training period, it would still be alarming evidence of bias, but other evidence suggests this was not a training period. I've never heard anyone call scientific data "intellectual property" before – that's an interesting legal theory, since this is not about an invention or an original creative work. If scientists were to get into the habit of treating data as IP, or otherwise proprietary, it would impair scientific progress and quality control – it would also violate the basic premise behind peer review. Shollenberger's disclosures took place in a context where the authors apparently refused to release all of their data, so I'm not sure what other options there were for him. In other words, he's a whistleblower. You can contact the research governance people at the University of Queensland here (scroll to the bottom of that page).

Update: In their legal threat letter, the University of Queensland says that the letter itself is intellectual property, and that publication of the letter is cause for separate legal action. What? That's like an NSL. Is this new? What kind of upside-down bizarro world is this? You can send someone a threat letter, copyright the letter, and force them not to disclose it? This is unbelievably creepy.

Update 2: Political activism is not a vice. I'm not saying it's a vice. If you think the left, or right, or libertarian, or Rastafarian perspective is true, do your thing. People have the right to be left-wing activists, conservative activists, environmental activists, wherever their minds and their values have taken them. I'm a pro-immigration activist sometimes. But I will never be a subjective rater of textual material in a study whose outcome would potentially serve my pro-immigration cause, especially if my ratings could deliver that outcome, nor will I ever assemble a team of pro-immigration activists to perform such ratings. Are we being serious right now? This is ridiculous. We can't do that. Do we want to call what we do science? This shouldn't be hard.