Jose L. Duarte
  • Home
  • Blog
  • Media Tips
  • Research
  • Data
  • An example
  • Haiku
  • About

The art of evasion

9/9/2014

45 Comments

 
Here's an update on the Cook 97% scam. They apparently have no answers.

Cook and some of his crew had a chat event at Reddit. Last I checked, they never addressed any substantive issues in the report, even though various people asked them about it. They don't argue against the fraud reports, the validity issues, nothing. Instead, they tried to attack me, or issued vague filibusters that had no bearing on anything happening here. To my knowledge ERL has yet to issue a statement or offer an explanation for the paper's comprehensively and substantively false description of its methods.

To review, in their paper, they described their method as: "Abstracts were randomly distributed via a web-based system to raters with only the title and abstract visible. All other information such as author names and affiliations, journal and publishing date were hidden. Each abstract was categorized by two independent, anonymized raters."

All three substantive features of their method are false. Raters were not blind to authors (or any of the other info.) Raters were not independent. Raters were not anonymized.

They falsely described their methods. That is a very, very serious thing. There is no science without an accurate description of methods, and this paper, like all papers, was published on the assumption that they followed the methods they described.

Normally the way science works, that's the end. Nothing else needs to be done by anyone. There are no results to evaluate if they didn't follow their methods. Why? Because valid results critically depended on those methods, and when people don't follow their stated methods, we don't know what they did and thus can't rely on the results. Climate science, or its journals, can't be an exception to this basic norm and epistemic requirement of valid science. Why would they be an exception? (This has nothing to do with the truth of AGW or the reality of a consensus -- this is about a fraudulent and invalid study.)

This is not a pedantic issue. This was a subjective rating study where human raters read authored works and decided what they mean. Such a study could never be valid without blindness to the authors of the works they were rating, since knowing who the authors are is a multifaceted source of bias. Nor could it be valid without independent ratings – raters discussing their ratings in an online forum contaminates those ratings, exposing one rater's ratings to the views of other raters, and makes it completely impossible to calculate interrater agreement or reliability from that point forward. We have no valid numbers with respect to agreement – the crude percentages they strangely offered instead of proper interrater reliability coefficients don't mean anything given that the raters weren't independent. If we can't calculate interrater reliability, we don't have a study anymore.

Having humans read complex text and decide what it means, what position it's taking on some issue, is a very special kind of research because it enables to researchers to create the data to a degree unparallelled in typical science, and the data is the result of subjective human appraisals of text, which is extremely vulnerable to bias, many sources of error, and in some cases the task is not theoretically possible or coherent. Such research demands great care, and won't be valid if things like blindness and independence are not observed. That these people gave themselves a category of implicit endorsement of AGW only exacerbates this. That they excluded papers they interpreted as taking no position on the issue, which were the majority, and calculated a consensus excluding all the non-polar papers, takes this all further from a universe governed by natural laws. They also thought they could just count papers, which is surprising – that papers, not people, are the unit of consensus – after excluding everything they excluded, and given all the obvious weights and biases that would be applied by a paper-counting method (research interest, funding, maleness, age, connections, whiteness, English-speaking, least publishable unit practices, # of graduate students, position as a reviewer, position as an editor, to name a few...)

Moreover, in their online forum, the third author of the paper said: "We have already gone down the path of trying to reach a consensus through the discussions of particular cases. From the start we would never be able to claim that ratings were done by independent, unbiased, or random people anyhow."

Maybe we should quote their methods section again: "Each abstract was categorized by two independent, anonymized raters."

There appears to be no question that they knew, well before submitting the paper, that they had not implemented independent ratings, since as she mentioned, they were discussing particular papers in the forums the whole time. Yet, they still reported in their article that they used independent raters. What is this?


The percentages for rater agreement that they report in the paper are voided because of this -- those weren't independent ratings, so there's no way to measure agreement between them.

Nor can we assume that the fraud was limited to the forums, since the raters worked at home and could just google the titles of the papers to break blindness and see who wrote them. You can't perform this kind of study in such uncontrolled conditions – there's no way to credibly claim blindness here, which is crucial feature. In any case, they freely revealed authors of papers to each other on the forum, sometimes with malicious mockery if it was someone they had already savaged on their weird website, like climate scientist Richard Lindzen. It's incredible – they exposed the authors of the articles. They did so repeatedly and without censure, lacked any apparent commitment to blindness, and were e-mailing papers to each other, so it would be their burden to show the fraud was limited, if for some reason we cared that it was limited. Since they were e-mailing each other, they violated their claim of anonymized raters – they used their real names in the forum, had each other's e-mails, knew each other, were political activists from the same partisan website.

That last fact also invalidates the study in advance – we can't have political activists rating scientific abstracts on their implications for their political cause. That's an obvious, profound conflict of interest, and empowers people to deliver exactly the results they desire. I want to stress that this is unprecedented. No one ever does this. It's too absurd and invalid on its face. Studies based on subjective human raters are a small fraction of all social science, and many researchers will never need such a design, but using subjective raters who desire a certain outcome and are in a position to deliver that outcome is simply not an option.

For the defenders of this study, I think the absurdity of the design would be obvious in any other domain – e.g. a group of Mormon activists reading scientific abstracts and deciding what they mean concerning the effects of gay marriage, much less falsely claiming blindness to authors, falsely claiming to have used independent ratings, etc. If ERL or IOP want to argue that it's fine to use subjective raters who have an explicit, known-in-advance conflict of interest with respect to the outcome of the study, and thus their ratings, and who we can see in the forums gleefully anticipating their results in advance, bragging to each other that they've hit 100 abstracts without a single "rejection" of AGW, further biasing and contaminating others' ratings, posting articles and op-eds smearing "deniers" while they rated abstracts on the issue, and the leader of this rodeo telling the raters that this study was especially needed after another study found an unacceptably low consensus, again, while they were conducting ratings -- well, I'd really like to see that argument. It would break new theoretical ground. I'd normally say they need to talk to some experts in the subjective rating methodological literature, but in this case, you really don't.

(I'm not aware of any study that has ever used laypeople to read and interpret scientific abstracts. This method is remarkable, since it seems implausible that laypeople would consistently understand specialized scientific abstracts. As scientists, we won't understand every abstract from our own fields, much less other fields. Some of the explicit violations of the claim of independent ratings in the forum were cases where raters didn't understand what an abstract was saying, which is easily predictable. Then of course others offer their interpretations, and the rater's subsequent rating is essentially someone else's rating. There is some research on bias and interrater reliability where scientists or doctors rated abstracts in their fields, as for a conference, as well as research on reviewer agreement in peer-reviewed journal submissions, which is historically quite low. Those researchers did not contemplate the idea of laypeople or non-experts rating scientific abstracts. You'd need a lot of training, I think, and it's unclear what the point of such a study would be, for reasons I detail in my report.)

Relatedly, on the reddit forum all I saw were bizarre ad hominem attacks. One of the raters said I had no climate papers. That's goofy. You'd need deep climate science expertise to rate climate science abstracts, but you don't need it to point that out. The Cook paper isn't a climate paper – it's not about cloud feedbacks or aerosols. It's methodologically a social science paper, a subjective rating study, which is well within my area code. Calling out the fraud and invalid methods has no more to do with climate science than their paper does.

For example, when someone says "Abstracts were randomly distributed via a web-based system to raters with only the title and abstract visible. All other information such as author names and affiliations,
journal and publishing date were hidden. Each abstract was categorized by two independent, anonymized raters."

...you don't need to know anything about climate to read this, read the forums where they disclose the authors of papers and discuss their ratings with each other, refer back to the above text from their Methods section, and observe that they falsely described their methods. It might help to know what blindness means and why it matters for subjective ratings, but that's not hard. Moreover, while I think technical ad hominem is valid in some very restricted cases (like a probabilistic inference that non-experts are unlikely to be able to understand specialized scientific material, such that the burden is on them to show they can), it's never valid to respond to fraud reports by saying "He doesn't have any climate papers." You'd want to respond to the report, to answer it substantively. They did something very serious, and they need to answer for it.

Others said my report needs to be published in a peer-reviewed journal in order to matter. Fraud is not normally debated in journals. I don't want to publish this in a journal (nor do I want any "climate papers".) It might be an example in a future publication, but why kill trees to point out fraud? We can actually see and know reality with our mortal eyes and mortal brains. And since the Cook paper made it through peer review, it's not clear that peer review should carry a lot of epistemic weight for the rational knower, at least in some journals or on some issues. My points are going to be true, false, valid, invalid, or some mix – whether they're published in a journal isn't going to tell us that. You can just read the arguments, consider the evidence, and decide – you don't need other people thinking for you. I could be a welder from Wyoming who just got out for good behavior – it would be remarkable for such a person to blow the whistle on a journal article, maybe worthy of a Lifetime movie, but it wouldn't change a damn thing. It wouldn't change anything about the Cook study, or the validity of my points. Nothing about me alters those realities. Journal publication might not even be a better-than-chance heuristic in such cases. This "consensus" epistemology is taken too far – people forget that consensus is only a heuristic, of variable and context-dependent reliability, not a window into reality.

In any case, I reported it to ERL and IOP, and they should be able to handle it without my needing to write it up for a journal. The authors need to answer some very simple questions. There need to be answers. Without some sort of miracle redemption, some way of the fraud not being fraud, it should be retracted. I don't know if this a wild and crazy idea to some people, but science would mean nothing if we can falsely describe our methods (and use methods guaranteed to produce a desired outcome.)

Cook responded to questioners by saying "The common thread in all criticisms of our consensus paper is that the 97.1% consensus that we measured from abstracts is biased or inaccurate in some way. Every one of these criticisms fails to address the fact that the authors of those climate papers independently provided 97.2% consensus. This is clear evidence that attacks on our paper are not made in good faith."

This is bizarre. They may be counting on people not reading my report. I explicitly address this issue in the report, so his claim is false. Here are a few reasons why we don't care about the author self-ratings anymore:

1) We learned that Cook included psychology, social science, public survey, and engineering papers in their "consensus". This after they explicitly said in their paper that social science and surveys of people's views were not included as climate papers. Chalk up another false claim. But, since they included a bunch of invalid papers, this means their author survey included the authors of those papers. This in turn means we can know longer speak of the authors' self-ratings percentages – those figures have no meaning anymore, given that we don't know how many were from psychologists, sociologists, pollsters, and engineers.

2) I pointed out in the report that their counting method is invalid because they count mitigation and impacts papers that have no obvious disconfirming counterparts. For example, if an engineering paper counts as endorsement because it mentions climate on its way to discussing an engineering project, how would an engineering paper count as rejection? By not mentioning climate? If a paper about TV coverage of climate news counts as endorsement (in contradiction of their stated criteria), what sort of study of TV coverage would count as rejection? One that doesn't mention climate? An analysis of Taco Bell commercials? Where's the opportunity for disconfirmation? There's no natural rejection counterpart to such categorization (it won't matter if you find a mitigation paper that they counted as rejection -- this is about the systematic bias, and the endorsements will be far greater than the rejections here as a result.) This all means we won't care about the authors' self-ratings, because of this systematic selection bias in the method (anything that biases the selection of articles biases the subsequent pool of authors rating those articles.)

I also pointed out that we can't validly measure consensus by excluding the vast majority of actual climate science papers that do not take polar positions of endorsement or rejection, which is what they did. Consensus cannot exclude neutrality. We can't assume that neutrality represents a consensus, as they do. And we probably can't count papers to begin with. This is all in the report.

3) Pointing at squirrels is never good when a study has been rebuked for fraud or invalid methods, both of which are the case here. You cannot redeem the malpractice in the first part of the study -- all their false statements about their methods, the invalidity and meaninglessness of their results -- by talking about a completely different part of the study. That's pure evasion. They need to answer for what they did.

4) There are serious questions about their literature search, such as how it excluded everything Dick Lindzen has done since 1997. There are no results without figuring out what's going on with that search, how it excluded all the modern work of a seminal lukewarm climate scientist. You can't just run a lit search based on a couple of terms and then declare that you've got a valid, representative sample of studies. No way. Science can't be so haphazard – you have to try. There's work involved. New methods need to be validated. We can't begin to talk about percentages and numbers without first establishing that our data is valid, this this literature search is valid (and dealing with all the other issues, the first of which is the fraud.) The search issue will interact with point 2 above. Validating the search will require careful thinking, testing, etc. Some of the methodological meta-analysis literature will have guidelines. How to do a valid search for this purpose is a nontrivial issue – nothing that happens after the search matters if the search isn't valid. You have to figure out if there are selection effects, what you're including, what you're excluding, especially with respect to your hypotheses, what happened to Lindzen's papers, and so forth -- you don't just run a search and start rating papers. This is science, not numerology.

(The equivalent would be if some people rented a couple of hot-air balloons for a few days, took pictures of some clouds, wrote it up for a journal submission on cloud cover in North America, and we simply accepting their method without question. Science can't be that sloppy.)

(And only 14% of authors responded to their survey, and they got about two votes each, which highlights the oddity of simply counting papers. I've already pointed out in my report many of the biases and potential weights that will be applied by a simple paper-counting method.)

Cook's statement is also bizarre because of his conclusion. Saying that people who contest the validity of their ratings never address the purportedly similar results of the authors' self-ratings, and that this "is clear evidence that attacks on our paper are not made in good faith" is so strange. Evidence that people are not acting in good faith? Because they don't address some other part of a study? That's such a strange model of human psychology, and of how science works. They seem to have a remarkable immune system or strategy of not engaging substantive criticism, characterizing it as not in good faith, "denial", misinformation, etc. If genuine, it's a remarkable worldview.

Another Cook tactic was to talk about other studies finding similar results. I was dumbfounded. When someone is referring to your false claims about your methods, and your invalid results, the subject under discussion is your study. Nothing else matters. There could be a thousand 97% papers. It wouldn't matter. That there is a consensus does not matter. The issue is the Cook study, its fraudulent claims, its invalid results. Fraud is fraud. We don't redeem fraud by talking about other people's studies. Some of those studies are almost as bad and unusable, like the one-pager, but that's beside the point. The true, meaningful consensus could be 99.997% – that won't matter here. (We also don't need this study to establish a consensus, just as we don't need a particular hockey stick paper to establish anthropogenic warming. We might consider that the case for the consensus likely weakens if this study is included.)

It's incredibly disturbing to repeatedly see this kind of response from people who were published in a scientific journal. Not only did these people have no idea what they were doing, they betray no sign that they're aware of the basic norms of science, the importance of faithfully describing one's methods, why fraud is a serious thing, what it means to say that results are invalid, or how the existence of other people's studies won't intersect with or override the false claims made about one's own study. If their responses are being validated or accepted by ERL or IOP, we've got a much bigger problem. People might be justified in ignoring climate science if climate journals are fine with what Cook did. We wouldn't have any way of knowing that other research was valid, since climate papers would be much harder for us to evaluate. If ERL is fine with fraud, we wouldn't know what to do with any future ERL articles, or with the field as a whole. That's not anywhere we want to be. Some of the confusion or delay might be due to ERL's unfamiliarity with these methods, the need for blindness, independence, interrater reliability, probably a lack of awareness that they falsely described their methods. But those can't be issues anymore – if people don't get it, don't understand why a subjective rater study necessitates that the ratings be blind and independent, don't understand why this study destroyed our ability to calculate interrater reliability, etc. they can just consult some relevant experts in social science.

Let me tell you something else. Some people are saying this paper won't be retracted because the journal is biased, or because it would look bad for Dr. Kammen (the editor), or because he works with the White House, that science is rigged. We need to cut the crap. I don't care who is who or how much power they have. This was fraud. Falsely describing one's substantive methods is fraud. This paper is invalid in ten different ways, but fraud is fraud. They have not answered for it. They apparently have no answers. If the relevant decisionmakers are thinking that they can just ignore this, or issue some PR blather, they need to think long and hard about what the hell we want science to be. They need to think long and hard about the long-term consequences of failing to retract fraud when the evidence of said fraud is so publicly accessible and straightforward. They might think about how far we have to throttle down our brains in order to believe that this study is remotely valid. They might think about what it will do to the reputation of climate science not only that this study was published in a climate science journal, but that it wasn't retracted when it was revealed to be fraudulent and multiply invalid. There are real consequences here, long-term impacts. We can't be this bad at science, this corrupt. We need to stand for things that aren't in a party platform. These people made a complete mockery of the institutions and safeguards that we take for granted. They torched them.




Afterword:

I think it's worth considering what could happen in the long-term if this nonsense were tolerated, if we couldn't get people to act against fraud if the fraud were politically convenient, in combination with all the conventional fraud and malpractice. Cultures and civilizations evolve in countless ways. The meanings and usages of words can change. There's no a priori hard constraint on this process. Science doesn't have to mean something like rational inquiry, or the systematic, reproducible validation of empirical claims or hypotheses, or any of our other definitions (in capsule form.) The word science could end up meaning something very different. For example, in the worst case, it could end up being used only ironically, as a term for scams. As it stands, we probably use the term too broadly -- science is very diverse, and the issues discussed above will be alien to many scientists. (The idea of gathering some political activists to "rate" scientific abstracts is far removed from most scientists' methods, and the type of fraud we see here is very different from prototypical fraud in cellular biology, for example.)

I don't mean that what we've classically known as science would cease to exist -- that would likely require a large asteroid strike. I mean that science could plausibly evolve into a label for a corrupt and privileged subculture whose divinations are no more tied to reality than a day-old religion. In such a case, actual science as we mean it today would be called something else, perhaps interrogo, ρωτήστε, or bluefresh, and the professionals who practiced it would meet high standards -- the standards people expect of scientists today. I'm not saying this is likely -- I don't quite think it is. But I think it's plausible, and I'd caution you against underestimating the degree of change and evolution that human culture and language can experience, even in our lifetimes. I don't mean to make a trivial point about changes in terminology and phonemes. I mean to illustrate that continuing to generate garbage and fraud under the banner of science will despoil that banner. Circling wagons to protect such fraud and garbage is tautologically incompatible with our prior commitments to scientific integrity. In any case, nothing in life is assured -- not prestige, not funding, not an audience. I don't like being pragmatic about this, since I think idealism should dominate here. We like to say we stand on the shoulders of giants -- we might want to think about whether anyone will be able to stand on ours.
45 Comments
Barry Woods
9/9/2014 07:54:27 am

Hi Joe

Every one would agree with you, in the last paragraph.. but as the journal's and people involved are doing nothing, there seems no possible way I can see to make it happen.. and everyone would agree that is an intolerable position for science.

When I wrote to Psychological Science pointing out the authors of the paper (LOG12) and the University refused to release the raw data (remember it included minors according to the paper) they point blank said no, nothing they could do..

When i said if the authors refuse to provide the data, that the journal could insist or retract it (on basic scientific principle) I had the Chief executive of the APS write back saying...:


Dear Mr. Woods,

You recently lodged a complaint with the Editor of Psychological Science. In keeping with COPE guidelines, we elevated the matter on appeal to the Chair of the APS Publications Committee, who has investigated and supports the editor’s handling of the paper in question and of all subsequent allegations, complaints, and requests. We now consider the matter closed.

Sincerely,

Alan Kraut


--------------------------------------------------------------------------------

Alan Kraut
Executive

Executive Director

ASSOCIATION FOR PSYCHOLOGICAL SCIENCE

1133 15th Street, NW · Suite 1000
WASHINGTON, DC 20005 USA


Reply
ilma
9/9/2014 07:45:04 pm

This goes right to the top. Obama himself continuously and repeatedly cites the 97% consensus, so you can understand the reluctance of Mr Kraut etc. to admit that Cook's paper is basically junk. It would make Obama a laughing stock, but that's going to happen anyway, no matter how much he denies it. As you say, fraud is fraud, and it goes all the way up to the US Presidency, as well as the UN SecGen and EU presidency.

Reply
Barry Woods
9/9/2014 10:21:16 pm

ilma -Kraut's response is not about the Cook paper.
it is about this paper:
http://www.joseduarte.com/blog/more-fraud

Reply
Kristy
9/10/2014 12:53:53 am

It is interesting to note that Peter Gleick is on the review of board of ERL. He is the one who illegally obtained papers from the Heartland Institute. It is also interesting to note that the same journal that accepted the Cook paper rejected Lennart Bengtsson’s paper (the scientist who recently changed his views to a more skeptical viewpoint) on this basis:

Dr. Nicola Gulley, Editorial Director at IOP Publishing, says, “The draft journal paper by Lennart Bengtsson that Environmental Research Letters declined to publish, which was the subject of this morning’s front page story of The Times, contained errors, in our view did not provide a significant advancement in the field, and therefore could not be published in the journal. The decision not to publish had absolutely nothing to do with any ‘activism’ on the part of the reviewers or the journal, as suggested in The Times’ article; the rejection was solely based on the content of the paper not meeting the journal’s high editorial standards, ” she continues.

http://ioppublishing.org/newsDetails/statement-from-iop-publishing-on-story-in-the-times


Reply
Joe Duarte
9/10/2014 10:15:53 am

Illegally obtained papers? Peter Gleick, the clean water guy? I guess there's always google, but is this an established fact? And you're saying he's currently on the board?

Dr. Gulley seems professional to me. I'm sure I haven't endeared myself to IOP for being so public, and probably harsh by their reckoning, but with these cases I'm just not motivated to be smooth and silky. This case is too extreme, too obvious. Fraud should never require a lot of politicking and dancing to report. I feel like I should just be able to give the information needed and investigators should be competent and motivated to do their jobs. I've given much, much more than needed. And good investigators will notice a lot more than I wrote about. As vast as my reports/essays have been, I left out a lot, if you can believe it. I could have had another ten pages of original content and issues, and I can write ten pages very quickly, but I get bored with this stuff and I have work to do. It's not smart stuff. There are no hard problems here, nothing to chew on. The guilty parties aren't serious thinkers and seem unable to say anything logically connected to the issues. I'll be surprised if investigators don't report a bunch of other things that are apparent in the forums, don't dig into some obvious forensic places.

How their editor handled the reports is worthy of a completely separate investigation -- science falls apart if editors are going to behave that way, if there's no process and structure in place to manage the conflicts of interest. The editorial culture so to speak at some journals is simply corrupt. At the very least, it's quite a bit less credible than the whistleblowing provisions of a random corporation. And I was floored to realize that a common newspaper is much more rigorous than some scientific journals -- if they print something that's false, they correct. If you contact the editor at the NYT about a false claim, they will investigate. They will look at it, and think about it, and act as appropriate, in general (the epistemology of journalism and journalistic truth is a separate and potentially rich domain all by itself.) They run corrections all the time. Some of our journals are much less rigorous than the Dallas Morning News, and that's really something. Some journal editors won't even acknowledge or confront their conflicts of interest the way judges do, or newspaper editors do, or even politicians do. No one makes them. They might be outliers though -- I stumbled on a handful of cases and I'm hoping these people are somewhat rare. It's surprisingly easy for scientific bodies to be corrupt. They don't self-police very well. It's strange to me. I'm not used to corruption. I've never confronted it before, not this kind. It blindsided me. I feel like I arrived from another planet and am looking around and asking people if they're serious. It never occurred to me that science could be, in many respects and for some more than others, an *interest group* like any trade group, with all the toxic tribalism, instead of a place for idealistic discoverers. Corruption is so pointless. It's a waste of time, and life is time. The long-term impacts are too obvious, including those for their legacies. It surprises me that they seem to think they could get away with it. That's fascinating. In any case, many of the issues here are going into a journal submission.

Reply
Kristy
9/10/2014 10:44:36 am

Joe, I do want to thank you for your research on this subject.

Here is the information on Peter Gleick:

http://www.theatlantic.com/business/archive/2012/02/peter-gleick-confesses-to-obtaining-heartland-documents-under-false-pretenses/253395/

The science of the climate is very corrupt. If you dug a little deeper into the cult of this science, I think you would be more than blindsided, but also very depressed to see how it seems activism has taken over this science. This is the only science I know of that does not follow the scientific method. Their hypothesis can never be falsified. And if anyone disagrees with the hypothesis, their careers are ruined and the scientists are labeled as working for Big Oil or they are labeled anti-science. That is how they keep getting away with what they do, because most scientists want to keep their job, so they don't speak out.

Not sure if you have heard of Judith Curry, but she is one climate scientist that is standing up to the consensus.

http://judithcurry.com/2014/01/18/mann-on-advocacy-and-responsibility/

Stephen McIntyre
9/12/2014 04:26:34 am

To describe Gleick's offence as "illegally obtained" papers does not adequately describe Gleick's conduct, which had two distinct components. Gleick set up an email account in the name of a Heartland director and, using this account, asked a secretary to copy correspondence to Heartland directors to the dummy account. Similar conduct has led to charges in other situations, but not in Chicago.

In addition, the most inflammatory document disseminated by Gleick - and the one that occasioned most adverse commentary - was forged. The evidence is overwhelming that Gleick was the person who forged the document - that was how Mosher identified Gleick in the first place - but Gleick did not confess to the forgery.

Ironically, Gleick was Chairman of the AGU Ethics Committee at the time, but AGU did not investigate Gleick's conduct and Gleick has been an invited speaker at recent AGU events.

Joe Duarte
9/16/2014 02:39:27 pm

Steve, check your e-mail. From now on I'll use the code "Ethan Fromme" in the subject line :-)

Thanks Kristy. Judith Curry has cited some of my blog posts. This is probably one of the reasons why I'm getting enormous web traffic statistics. This reminds me that in the end I probably can't be the guy some people want me to be. I'll comment on methodological validity in social science, which will sometimes include consensus studies, but I'll never be a climate scientist. People are sending me graphs of outgoing IR -- I'm just not going to be the guy to adjudicate substantive climate science issues. And the politics of climate is too toxic for me long term. I think politics makes people worse in general, and climate politics is like pouring acid on the soul -- there's so much malice, hysteria, and groupthink. I still think there's a consensus, and that climate scientists are right, that they're too smart to get caught with their pants down and be wrong, or mostly wrong, about AGW, but I'm a bit worried by the culture, by journals like ERL, and the ability of incompetent political operatives to publish scam consensus studies.

To wit, I'm stumped on the Gleick thing. No journal would have such a person on their board, except maybe 10-20 years later, after a long period of clear reform and non-recurrence. It's impossible that a scientific journal would have someone on their board when the person just the other day impersonated a principal of a private organization in order to steal their documents, and almost certainly forged a document himself to defame the organization, all as part of an apparent *political activism*. That's impossible. This should make people throw up in their mouths a little bit, like Christine Taylor in Dodgeball. It's like the Boy Scouts retaining a board member who just last year was robbing banks.

Something is misfiring here. How do they get away with it? Do people not know? Nicola Gulley seemed professional to me -- I can't imagine she'd have an impersonator and document-stealer as a board member in one of their journals, but I don't really understand their org chart. Has anyone ever asked Kammen about Gleick? Or IOP? What do they say?

I don't know, maybe I'm wrong. People make mistakes. Gleick's is a pretty damn big mistake, with all kinds of premeditation, but maybe Kammen knows something we don't, maybe it was a moral failure that he knows won't happen again, or that Gleick is a magnificent human being overall who stumbled, or there's some other context here we don't have access to. It's seems unlikely. I can admire people willing to take unpopular stances out of loyalty to a person, but not when the person has dishonored himself. The dishonor would kill me were I Gleick or Kammen -- I'd feel like I'd dishonored my family, my parents, my friends. Did Gleick do anything to repair his honor? I think he'd have to mop the foors at Heartland for a year or something like that.

MikeR
9/17/2014 06:47:05 am

"Did Gleick do anything to repair his honor? I think he'd have to mop the foors at Heartland for a year or something like that." Uh, well.

"Whistleblowers - and that's the role Gleick has played in this instance - deserve respect for having the courage to make important truths known to the public at large. Without condoning or promoting an act of dishonesty, it's fair to say that Gleick took a significant personal risk - and by standing and taking responsibility for his actions, he has shown himself willing to pay the price. For his courage, his honor, and for performing a selfless act of public service, he deserves our gratitude and applause."
http://www.desmogblog.com/whistleblower-authenticates-heartland-documents
Not going to be easy for him to be penitent - there's a cottage industry of AGW bloggers cheering for him. It's obvious to the rest of us that Gleick forged the last document, but places like desmogblog have extensive posts from the time refusing to believe it.

Kristy
9/17/2014 11:21:10 pm

I totally get why you would stay out of the politics of climate. It is toxic. You have written a very thorough analysis and breakdown of the problems with the 97% paper and already you are being labeled a “denier of climate,” and a “conspiracy theorist,” and an “extremist right wing ideologue.”

http://blog.hotwhopper.com/2014/08/irony-alert-more-conspiracy-plots.html

http://blog.hotwhopper.com/2014/08/denier-weirdness-97-irony-deniers-deny.html

The problem with the politics of climate is that it has always been about the politics and not the science. The IPCC was formed in 1988 by the UN at the request of 100 member nations to study how man is causing warming. By 1990, it was decided that we needed international cooperation to tackle the global warming challenge and its consequences. The IPCC therefore played a decisive role in leading to the creation of the United Nations Framework Convention on Climate Change (UNFCCC), the key international treaty to reduce global warming and cope with the consequences of climate change and so billions and billions of dollars were granted to universities and institutions of science to study man’s influence on the climate with little regard to any natural factors that may be playing a more dominant role in the climate. Who is the face of Global Warming? Al Gore, a politician. In 1992, Al Gore wrote in his book Earth in the Balance that there was a 98% consensus. He stated in his book that the 2% should not be given equal weight with the consensus. Gore said that if the remaining unknowns about the environmental challenge enter the public debate, they are presented as signs that the crisis may not be real after all and it undermines the effort to build a solid base of public support for the difficult actions we must soon take. He used Richard Lindzen as an example of someone from the 2% whose views carry too much weight. Today we see leaders of governments around the world calling those who don’t “believe” flat earthers, science deniers, etc. They say these things so that the “2-3%” don’t carry any weight in the debate they say is settled.

So you wonder how Peter Gleick gets away with what he did. When you have the full force of the governments around the world and a protective media, you can get away with anything. You think the scientists don’t want to get caught with their pants down, but they did. It’s all in the Climategate emails. How did the media handle that? They trot out Al Gore who states those emails were taken out of context and the media says okay, moving on. The same media that asked for volunteers to comb through Sarah Palin’s emails. The same media who is saying there is a coverup in the NFL. The same media who are ignoring the coverup in the IRS scandal and the gun running scheme of the Justice Department. Peter Gleick got away with a crime, never even charged. He was investigated by his employer and reinstated. He received an invitation to speak at the 2012 AGU Convention. Michael Mann was investigated by his employer and cleared. They never even interviewed Steven McIntyre. They know they can get away with anything, even a crime, because there is no one holding them responsible for any of their actions. And if anyone does hold them up to any scrutiny or questions the alarmism of the science, those people are immediately labeled denier or being paid by Big Oil or anti-science in order to marginalize them as someone not to be taken seriously. I think there are only a few gatekeepers of the political climate science, those exposed in the Climategate emails. I think these few gatekeepers along with the politicians and the media, keep the scientists with less alarming views of AGW silenced by intimidation, fear of losing their careers.

Brad Keyes link
9/12/2014 11:13:22 am

Stephen,

great to see you here!

You mentioned that, "Gleick did not confess to the forgery."

Gleick has yet to confess to the forgery *in public.* (Unless we all missed something!)

I asked him directly, man-to-man, one-on-one, whether he forged it.

BK

Reply
Brad Keyes link
9/12/2014 04:16:49 pm

Steve forgot (or was too compassionate) to mention the most heartbreakingly, poignantly laughable fact about the critically-endangered Peter Gleick Innocence Society (which is now thought to contain a single breeding couple, three unemployably bimbonic secessionists from the Flat Earth Society mainly there for the donuts, and a very special dog that thinks it's people) is that even Peter Gleick doesn't agree with their psychotic premise.

Gleick has never disputed the fact that he forged the 9th document.

I repeat.

Peter Gleick has never denied forging it.

Anyone who doesn't believe me is welcome to produce the mythical, apocryphal quote.

(This invitation has been open for more than 2 years now.)

Crickets.

And this is how religions die.

Sheri Kimbrough link
9/10/2014 03:07:13 am

Excellent write-up. It is imperative that people understand what is happening with science. The standards for research seem to be getting lower and lower and outright fraud is now published. If politics and deception cannot be extricated from science soon, it does not bode well for the accuracy of future research.

Reply
Fernando Leanme link
9/10/2014 08:12:23 am

On the other hand I see a benefit. Many of you are starting to understand agitprop techniques used to deceive the public.

Agitprop is an old soviet term. It included having articles written in rather obscure journals and newspapers, which would gradually be moved up the line until Soviet propaganda could use it.

The technique used here is fairly simple. They have blizzards of ersatz papers to be mentioned and summarized. The media reports also tend to distort and amplify the papers main points. Then a high level government official quotes a distorted version.

This. Was used by the British (Tony Blair was very good at lying using this technique) during the Kosovo and Iraq affairs. It was used by the Bush administrations during the run up to both Iraq wars, is used by Obama in the climate wars as well as the Ukraine conflict. Like I said, it's common.

Reply
Brad Keyes link
9/11/2014 07:21:43 am

Can anyone name anything—at all—wrong with Cook13 but *not* with Oreskes04?

Has John Cook contributed even *one* new idea to the scam-consensus-study genre? (Has he had an original evil thought in his life, I wonder?)

As far as I can see, Cook13 is a slavishly derivative copycat crime that does nothing to advance pseudoscience and should never been published.

I liked it better when it was called Oreskes04.

Reply
Brad Keyes link
9/11/2014 11:04:56 am

Joe,

"Some people are saying this paper won't be retracted because the journal is biased, or because it would look bad for Dr. Kammen (the editor), or because he works with the White House, that science is rigged. "

The paper won't be retracted, because "climate science" is like "koala bear."
You mustn't believe every word you say, Joe. "Climate scientists" are communicators, not scientists. (A "communicator" is an oral-only whore.) Your retraction demands might gave them a few minutes' amusement. But don't give much more thought to the whole episode; they certainly won't. Chinatown, and all that.

Reply
Vieras
9/12/2014 04:17:33 am

I'm sure that the paper will not be retracted. Not because of some grand conspiracy. The journal is just staffed by people who care way too little about science. It's not that hard to find examples about universities who abuse science the same way. And it sucks just as much that old scientific societies are just as bad.

I hope and pray that the corruption is only in climate science. But as low quality climate science papers are published even in big name journals like Science, it's possible that the cancer runs deeper. It will take a long time to undo the damage.

Reply
Brad Keyes link
9/13/2014 11:16:49 pm

"I'm sure that the paper will not be retracted. Not because of some grand conspiracy."

I didn't suggest any such thing, by the way. There IS no grand conspiracy, as far as I can see.

Unless doing nothing is now a conspiracy.

In which case we're ALL in on it! It's bigger than you can possibly imagine! It goes all the way up to the Oval Office.

"The journal is just staffed by people who care way too little about science."

Fine, but then why hasn't *Oreskes04* been retracted?

Is Science staffed by people who care way too little about science?

(I'm almost afraid to ask.)

Reply
Sheri Kimbrough
9/14/2014 02:35:36 am

Best not to ask.

Reply
Brad Keyes link
9/13/2014 11:32:14 pm

"The journal is just staffed by people who care way too little about science."

The journal staff will be perfectly welcome to try this "explanation" out on the judge at Climate Nuremberg.

(Not that the plea of apathy seems to work very well, historically.)

Reply
steve
9/14/2014 09:04:21 am

What is your responsibility as a social scientist? To point out errors, label it "fraud", and blog about it? Or to seize the opportunity to make an advance in our understanding?

For all of your rhetoric about what "science" is or ought to be, why you refuse to write a formal response piece is still beyond me. Do you want to be taken seriously or not? If you have any interest in advancing what you perceive as the truth, you should elevate your argument beyond the blogosphere (which, like it or not, will always be viewed as a less credible medium) and subject your ideas to the same degree of publicity and scrutiny as the articles you argue against. That's the nature of science's self-correcting process.

With all due respect, it really wouldn't hurt you to be a bit more humble at the same time. You repeatedly scrutinize the authors' objectivity due to their stance on global warming, but never acknowledge your own potential for motivated appraisal and bias. As a social psychologist you should know better.

Your conviction in the word "fraud" is one obvious example. As anyone who has worked with data knows, errors and oversights often occur in the absence of fraud. Your critiques may be perfectly valid. But I've noticed that you have not acknowledged that your critiques do NOT invalidate other analyses by these authors, even though this is what many readers likely take away from your writing. Of course it would be glib to call this oversight an example of fraud -- I'm sure your intentions are good, even if your motivations may lead your focus elsewhere.

Reply
John M
9/15/2014 02:08:21 am

So, exactly what do climate scientists need to do in order for the word "fraud" to be fair game? Truncating data without mentioning it to mislead the audience? Stating methods in a paper that were clearly not followed? Using biased raters with a vested interest in the results? And given that they like to equate their opponents to Holocaust deniers and people who believe the Moon Landing was a hoax, you'd think they'd either show a little more restraint or grow thicker skins concerning criticism.

It amazes me that people seem to have very low quality standards and seem more interested in protecting a political narrative than protecting the credibility of science and scientists.

Reply
Joe Duarte
9/15/2014 06:26:21 am

Hi Steve -- I'll never be willing to respond to a fraudulent paper via a journal submission, at least not until after it's retracted. Fraud should just be retracted. I'm not going to legitimize it by pretending it can be assessed and debated like a normal paper (and in fact, I might be in the majority here -- we don't normally debate fraud in journals.) I'll submit papers in response to some other consensus research, stuff that's not fraudulent.

There are no other findings -- I'm not sure what you mean. This was fraud. They didn't follow their stated methods. We can't do anything with any results. Falsely describing your methods is a doomsday event. Especially since it was so simple and clear. There wouldn't be any results anyway, even if there hadn't been fraud. These were political activists deciding what abstracts mean on their political issue. We were never going to be able to do anything with such data. It's also an invalid study in many other ways, so there are no "analyses" that we'd be able to extract.

(Their survey of authors is invalid since they included a bunch of invalid papers like psychology, polls, social science, engineering, and because of the structural bias of their categories, and because their search was invalid, and many other reasons.)

I don't think social scientists have any sort of uniform responsibilities, other than to not commit fraud, to do valid research, etc. It's perfectly fine to point out fraud. We should never discourage that. We should encourage vigilance, we should want to see a lot of critical review. We need a lot more criticism, a lot more rigor, since social science has profound methodological vulnerabilities. We need a lot more people willing to actually think about studies, to notice things. Science is *not self-correcting* right now, a reality that has been detailed by others in various places.

In my mind, this case was far to obvious. There's no point trying to dance around what they did here. It's so repugnant that it should be gone yesterday -- we need to be arguing about borderline cases, cases that give us something to talk about, not about this.

Reply
steve
9/15/2014 07:13:47 am

"Their survey of authors is invalid since they included a bunch of invalid papers like psychology, polls, social science, engineering"

Obviously, there is no perfect way to measure consensus among "experts", nor is there a perfect consensus on who the experts are. Thus any sampling method will be imperfect. To interpret this reality as "invalidating" the findings is overly dismissive.

Though inevitably imperfect, the researchers took proactive steps to minimize the influence of their imperfect sample by allowing authors (including social scientists, engineers, etc) to report that their studies had no bearing on the evidence for/against anthropogenic climate change. And many did. Might there be a better strategy? Sure, nearly all survey research can be improved. Does this make the whole analysis "invalid"? No. To proclaim these results are "invalid" and without nuance is overly dismissive.

Joe Duarte
9/16/2014 12:06:13 pm

We can't let postmodernism ruin our lives. None of what you said matters if they falsely described their methods. It doesn't matter what we think about including social science papers. It doesn't matter if we put "experts" in quotes or say that it's hard. They said they excluded social science. They didn't. So that will be that. They said lots of things they didn't do.

But logically, there's a problem here. Their method was invalid in multiple ways. The search looks to produce arbitrary results with strange biases and has not been validated. That alone makes everything else pointless. They paper counting method imposes lots of weights and biases, inherently (mostly by giving old white males more votes, assuming a neutral search, along with lots of other weights.) You might not be aware that the respondents got about two votes each, and they were only 14% to begin with. Counting mitigation and impacts papers imposes a serious inflationary bias. All of these things invalidate the study independently, and they all extend to the author self-ratings.

Logically, these things would have to be addressed, refuted, for us to take the study seriously at all. There's no point in just talking about A, if there's also problem B, C, D, E, and F. You have to deal with them, or there's no case. Dealing with them might include saying they don't matter, but that would need to be argued.

This study doesn't have findings. We have no grip on any kind of reality when there are such profound and unaddressed validity issues -- there are so many structural problems with the data that there's nothing we can do with it from where we stand. We should have no interest in talking much about it when there was fraud. No one should get to retain an effect when there's fraud, but there are no effects anyway. (I don't think they've addressed the fraud -- silence is their typical strategy. There's so much evidence, and so many instances, that I don't foresee an answer.)

Your earlier bias point needs some sort of connection to a substantive outcome. We can't just say someone's biased and think something has happened. I'm not a subjective rater of abstracts on some issue I care about. That's where bias would be important, where we know we can't pervert the burden that way. My arguments could be motivated by bias, but if they're wrong, it would be easy to just refute them.

Reality is the kind of place where we can't be this dumb. This study is far too dumb, in multiple layers and dimensions. ERL didn't catch it, but in some secondary analyses they even tried to use explicit and implicit as different quantities, levels of endorsement. They treated explicit and implicit as *quantities*, and tried to run some analyses on that. I don't even want to go into what competent people would have to think about before assuming explicit and implicit endorsement represented different quantities of some stuff, or carried different epistemic information about a consensus, or could be treated as a continuous variable in analyses. This is just way too dumb. This is the kind of dumb that will get people eaten by a tiger at a zoo. We can't survive in the wild being this dumb.

Relatedly, one argument I've seen, I think from Gavin and partisan defenders of the paper (boy, the clock is ticking on you people) is that fraud means criminal fraud and can only mean criminal fraud. We need a hiatus on all the incredibly dumb arguments, a voluntary agreement. Scientific fraud is an extremely well-established and accessible concept, a recurrent phenomenon in the world. Journalistic fraud is very similar, and is also an extremely well established concept with recurrent exemplars. The denial involved in pretending that these realities don't exist, that "scientific fraud" doesn't exist, that "journalistic fraud" doesn't exist, is tantamount to denying the existence of tennis or Fridays.

Humans have long-term memory. We can all remember cases of scientific fraud, Stapel, Hauser, all the rest. We can just remember, and that would be that -- we'd know it wasn't about selling fake rolexes or in that category. We can also remember cases of journalistic fraud, Glass and Blair and so forth. Just from memory we'd know that it wasn't about forging checks.

Or we could just look it up. Like if for some reason we weren't sure if tennis was a real thing, we could just look it up, or drive around and see tennis courts. The fact that fraud is used to mean something different in science and journalism is widely accessible to the layman, to welders, sprinters, bloggers, scientists. We're being asked to be unacceptably dumb. This is not normal -- the denying of basic facts, basic features of civilization, of the ancient reality that words are used differently in different settings or fields. I'm offended by how dumb this all is. This is going to impact our evolution.

Sheri Kimbrough
9/15/2014 08:47:09 am

Steve, you seem oblivious to the reality of how poorly the Cook "study" was done. Pretending that the study had "some" validity because you apparently desperately want it to won't make it so. Consider if we substitute some other blogger out there doing a "study" using only their readers to rate papers and determine "consensus". Would you have been such a supporter if it was Anthony Watts or Monckton who had used exactly the same method? The results of a survey done by skeptic bloggers using skeptic reviewers would have given the opposite result, basically by design. Would you be in there claiming there was no fraud that skeptic bloggers did a survey and found that 97% of research paper writers don't believe AGW is real, using the same methods Cook actually employed? I'm betting not. Please note that had the skeptics done this, exactly the same arguments would apply and the same label of fraud would be applied. This is not about the accuracy of the AGW science, it's about improper and fraudulent methods used in a study and not actually following the methods the researchers said they used. One can cling precariously to the belief that Cook somehow just "sort of made a bit of a mistake", but he committed fraud. Straight up fraud.

Reply
Russ R.
9/16/2014 07:01:19 am

José,

After reading your excellent critiques of the Cook paper, I decided to take them to the forum at Skeptical Science (where I am very much despised), on an appropriate thread discussing the "97% Consensus", to see how people would respond.

Unsurprisingly, my comment was immediately deleted as a violation of their selectively enforced comments policy: http://www.skepticalscience.com/97-v-3-how-much-global-warming-humans-causing.html#106617

I expected this response and fortunately kept a screenshot of my comment:

https://dl.dropboxusercontent.com/u/78507292/Screenshot%202014-09-16%2014.17.43.png

Thought I'd share this with you, but I expect that you already know what sort of people you are dealing with.

Regards,
Russ

Reply
Joe Duarte
9/16/2014 11:28:00 am

Hi Russ. Thanks for the props. They're very creepy -- it's disturbing, because they're going to lose on their paper, but I'm a bit worried about what they'll do with their energies afterward. They're at war on the politics of AGW -- it consumes them, and I don't think they know who they are if they're not fighting this war. It's very hard to do good science when you're at war. I'm worried they'll run more fraudulent or junk studies, kill more trees, waste everyone's time again.

The fraud issue will be elaborated more in an upcoming article -- I refuse to write a paper solely about their study (I'm actually a little bit of a tree hugger / nature lover), but I'm fine with using them as one of many examples. As you probably know, the nature and definition of scientific fraud is itself a focus of inquiry and scholarship. There are lots of journal articles offering different accounts, definitions, causes and consequences. There's some brilliant work out there. It's fundamentally the same as journalistic fraud.

I sort of reorganize and restructure the key elements of most definitions, and focus on the core phenomenon of scientific fraud, which is a sundering of the connection between methods and data, M-->D.

Fraud attacks that connection in various ways. Every scientific paper consists of a description of (M) methods and of (D) data/results, and rests on the assumption that the data/results sprout from the stated methods. Every form of fraud is going to violate that assumption. If you muck around in a spreadsheet and just make up data, well that isn't going to be the method of data collection you described. If you say in a subjective rater study that raters were blind to authors and were independent, the data has to sprout from those methods. If those methods weren't followed, if data was generated by non-blind and non-independent ratings, then M-->D no longer holds. (And it would invalidate such a study.) Here the fraud was superfluous. They were going to get the results they wanted anyway, given their intense biases and the fact that those biases had such a ridiculous outlet as subjective ratings of scientific abstracts on their cause. (Well, maybe they would've gotten 94%, but the numbers have no meaning anyway since the method was invalid in ways that make any conclusions pointless.)

What really surprised me were the forum comments where they explicitly say they're including social science papers. Even Cook said this. And then the paper says they *excluded* social science papers. ?? That's so... And they explicitly and separately claimed that studies on people's views were excluded. Those were so easy to find if they wanted to exclude them, but yet there they are in the endorsement data.

I will say that I was very impressed with Ari J. He's a solid dude. I don't know why he thought social science papers could count as mitigation, or why he didn't speak up after the paper was published (well, maybe he did -- Kammen wasn't operating with integrity, so I don't know that it would've mattered if anyone spoke up). But it's clear he has a lot of integrity, and he's much smarter than the rest. Like the others, he appeared to have no knowledge of interrater reliability as a formal concept, no knowledge that Krippendorff's alpha or Fleiss' kappa would be the analyses here. (It's impossible to estimate reliability when the ratings weren't independent and the coding was so unconrolled an unsupervised, so this becomes moot.) So he tried to invent it. I love that. I love it when people who don't know that there's an existing solution to a problem try to invent or create the solution. He was clearly curious to discover. That's the key test of a scientist for me -- curiosity is crucial. Cook was priming his raters to be even more biased. He actually alerted raters to a new study of the consensus -- a real study, conducted by serious people -- that had an unacceptably low estimate of the consensus. He told them that this result highlighted the importance of this study, the Cook study. He did this while they were engaged in ratings.

What ERL did here is one of the most amazing things I've ever seen. It will be hard to find a worse and more fraudulent study in a normal human lifespan. ERL should be issuing refunds to their subscribers in conjunction with a retraction, and Kammen should issue a long public apology. If I were a subscriber I'd demand a refund. I also wonder about future submissions to ERL. They seem to represent something other than science. I'm confused by what they are. No journal would ever have Gleick on their board right after he did what he did. That's impossible. So I feel like I'm missing something here. If they're just a political outlet, that still wouldn't explain publishing this study, or delaying a retraction. A political newspaper or magazine would retract quickly.

Reply
Russ R.
9/17/2014 05:54:13 am

Hi José,

I'm not an academic, so I don't pay much attention to publications, journals or editorial standards.

I'm a financial analyst and investment manager, so I'm decent with stats and analytical methods, and I have a pretty well-tuned BS detector, since my job relies heavily on being able to parse truth and fact from hype and fiction (all in an arena where people have strong incentives to mislead.)

I only came across the Cook paper because it gained prominence in the mainstream media. One quick look and my first impression was... whoa... they just treated the 66.4% of papers that had no position as if they didn't exist. What they should have reported was that "32.6% of abstracts endorsed global warming" but that wouldn't have made a good headline.

Then I found out it was written by the SkS crew, and it made perfect sense... this was merely a propaganda stunt masquerading as scientific research. So from that point, I pretty much ignored it.

I didn't care too much because I mostly agreed with the reported finding. (By their rating categorization I would have put my own personal position on climate change as Level 2 "explicit endorsement without quantification")

So, for me this isn't a political battle, but a matter of pointing out where people are misrepresenting or overstating the facts (something which I see frequently on both sides of the "debate" and I call out when and where if feel it's appropriate.)

Until I read your critiques (along with those of Brandon Schollenberg) I had no grounds to question their methods and the validity of those "facts".

Anyway, just a word of thanks for the obvious amounts of time and effort you've put into debunking the BS being put out by Cook & company. Hopefully your efforts will raise the bar for what constitutes legitimate scientific research.

BTW, I noticed your request in the other thread. Feel free to email me and I can give you a few examples of where voicing skepticism (even of the luke-warm, middle-of-the-road, fact-based, non-partisan variety) was not well received.

Brad Keyes link
9/16/2014 04:26:22 pm

Russ R.--

nicely done! The deletion, while predictable, can never take away that satisfaction, can it? Unfortunately I've told the inconvenient truth once too often at SkS and they've taken the precaution of blocking my IP. Make sure, if you value your ability to comment there, that you throw in at least ten flatteries for every truth. :-D

Reply
Brad Keyes link
9/16/2014 03:34:58 pm

Joe,

this is incorrect:

"I want to stress that [having political activists triage papers by position] is unprecedented. No one ever does this."

Wrong.

Why are the achievements of women systematically written out of the history books?

Naomi Oreskes, an activist liar who wrote Merchants of Doubt [!!], did all this and more ten years ago.

She deserves priority when giving credit for this new, er, "development" in science.

I'm still not aware of Cook having violated a single ethical or scientific principle that hadn't already been left broken and bleeding by Oreskes. But I'm open to correction on this.

Responding "but Oreskes is a scholar" is meaningless. (So is John Cook, in that sense.)

Reply
Sheri Kimbrough
9/16/2014 11:47:47 pm

Brad: I am unclear why you are so set on giving Naomi credit for her activities as a political activist that obviously affect her interpretation of climate science, and not in a good way. While I have always maintained that anyone who is an activist is not a scientist (the two are mutually exclusive), the point of this blog post seems to be the fraud in Cook's paper. That someone else may be committing fraud would be another post. Oreskes is a scholar, but then again, Cook has a degree in physics, as do many climate scientists. It sounds to me like you're saying "She started it", which I would hope is not your point.

Reply
Brad Keyes link
9/17/2014 12:20:59 am

Sheri,

thanks for the feedback. I was using the word "credit" ironically—as in, "blame." I've promised José I wouldn't write sarcastically anymore, but that line must have slipped past my internal censor. In any case, sorry for the confusion.

" It sounds to me like you're saying "She started it", which I would hope is not your point."

That's precisely my point. Everything Cook did wrong in this paper (to my knowledge), Oreskes had already done wrong in her consensus paper. It's a "separate" fraud, as you correctly point out—but it's also the template by which Cook must have known what kind of shenanigans he could perpetrate with impunity. Remember, Oreskes04 has never been retracted, which sets a precedent for anybody refusing to retract Cook. In fact Oreskes' career has enjoyed a hockey-stick-like trajectory since providing a [one-page, pseudoscholarly joke of a] paper just in time for Al Gore to casually refer to it in AIT. A career that exceeds all expectations one might form on the basis of Oreskes' talents, or lack thereof.

Not that I'm suggesting anything sleazy is going on in the climate movement, of course.

D'oh—sarcasm again—ignore that last line! ;-D

Reply
Sheri Kimbrough
9/17/2014 01:01:55 am

I'm not sure Cook had a template—it looks like he just did whatever would give him the desired result. Oreskes04 may have been the first such paper (I haven't really researched her much) but it in no way excuses what Cook did. Your argument seems to be that is someone does fraudulent research and later on someone uses the same method, it's the fault of the first fraud. Even if Cook did copy Oreskes method, it's still his problem, not hers. The proper methods for research are out there if one wishes to look them up. Before producing "research", it is incubent on the researcher to understand the rules and accepted methods of that research. That would include more than copying the methods of one researcher who happened to be mentioned by Al Gore. More important, Cook appers to have outright lied about what he did in this study. This fraud is on Cook. Oreskes is a separate issue, as far as I can see.

Reply
Brad Keyes link
9/17/2014 01:55:07 am

Sheri, you're cracking me up... in a fun way...

"Oreskes04 may have been the first such paper (I haven't really researched her much) but it in no way excuses what Cook did."

Of course it doesn't.

*Laughing at the thought of myself as a John Cook apologist*

"Your argument seems to be that is someone does fraudulent research and later on someone uses the same method, it's the fault of the first fraud."

Does it really seem to be that? Because it's not! They can both go Science Jail as far as I'm concerned. Life without the possibility of parole.

Reply
Sheri Kimbrough
9/17/2014 02:47:13 am

My apologies. I misread your intent here. Since you brought in a paper not related to the one being discussed, I mistakingly believe you were trying to excuse one using the other. Yes, all fraudulent paper writers should go to science jail—or at least have the paper retracted loudly and clearly. Very loudly and clearly.

Reply
Brad Keyes link
9/17/2014 03:11:06 am

By the power vested in me, Brad, King of Misunderstandings, thou art pardoned!

By the way, it was José who used the word "unprecedented"—so it was important to mention Oreskes04, the prototype of the genre and every bit as bad a paper, if only to highlight that this nonsense goes way, way back.

Also, I forgot a really cunning thing Oreskes did: she published her "paper" as an Essay, whatever that means. What it means, I guess, is exemption from a whole swathe of standards and criticisms. So Al Gore gets to call it a "paper" or "study"—an innocent misunderstanding by a non-scientist!—and use it as one of his infomercial's most convincing punchlines, while social scientists can't even criticise it because it never technically claimed any compliance with their standards.

This invalidates my earlier argument that Cook13 will never be retracted while Oreskes04 hasn't been. Cook13 has a vulnerability the older paper doesn't: he made the mistake of pretending it was a scholarly paper.

Not that it'll make a difference to the outcome.

Vieras
9/17/2014 02:37:21 am

Here's some advice for the sceptics: Do not try to change Jose's opinion about climate change. Don't send any links and information about climate science to him. There is no need to push him!

It's a lot better that Jose reads and finds out about all this by himself. It's also way more entertaining to read how dumbfounded he is about Cook, SKS, Gleick etc. ;-)

Jose, if I would have to find a single word to describe you, it's integrity.

Reply
Brad Keyes link
9/17/2014 02:55:28 am

"Jose, if I would have to find a single word to describe you, it's integrity."

Then we'd make a good Family Feud team because I was thinking that too.

Reply
MikeR
9/17/2014 02:37:38 am

Thought this was interesting:
http://bishophill.squarespace.com/blog/2014/9/17/the-oas-and-replicability.html
"[There is a] unique and important requirement placed up-front for any paper submitted; it must be replicable, with all data, software, formulas, and methods submitted with the paper. Without those elements, the paper will be rejected."

Unique requirement, note. I wonder what Jose thinks about this: does every other journal allow people to publish articles where you cannot actually tell whether they have real data, or at least whether their analysis on the data actually leads to their conclusions?

When I recently took a couple of online (Coursera) courses in data analysis, the basic requirement for turning in an assignment was an RStudio file that when you ran it, it reproduced all your results. All your graphs, all your statistics. It was easy to make.

This is all as opposed to climateaudit, Steve McIntyre's blog, where he and other statisticians spent about a _decade_ trying to reverse engineer various very important results by Mann and others, trying to figure out how he got the results he did from the data he had - and only then were able to disagree with his methods. They are still at it last I looked: http://climateaudit.org/2014/09/03/rule-n-revisited/

Reply
MikeR
9/17/2014 02:49:25 am

Another example, taken in a link from the previous link. There are literally dozens more:
http://climateaudit.org/2008/03/14/mbh-pc-retention-rules/
This is not a good use of people's time and energy. My first reaction to anyone who published a paper but refuses to include data and complete reproducible code: Come back when you are ready to provide proper information. Till then, don't tell me about your results.

Reply
Sheri Kimbrough
9/17/2014 02:58:41 am

MikeR: When I have written about the need for reproducibility in climate science, I have been told that it's too expensive to rerun the data so they only run it once. I can't say I have independently verified it, but it is a very common response.

As for McIntyre, if the Mann data were publicly availvable, or at least available in the paper itself, reverse engineering would not be necessary. (It is not clear to me if the data is or ever was available to the extent the resuIts could be replicated by others. I get conflicting responses when I ask.) It is a tenet of science that results must be reproduced and checked by other scientists. The data should be complete and the software available to check for errors. Without it, the study means nothing. It could be a one-time thing fluke or it could be a very important discovery. However, without it being reproduced and checked thoroughly, it's not really more than an unproven hypothesis.

Reply
MikeR
9/17/2014 03:51:15 am

"it's too expensive to rerun the data" Climate models are run on supercomputers. But we're not talking about that; Mann's paleoclimate reconstructions are little data sets involving a few hundred or a few thousand trees. Nothing to it.

As for what was available, that's been a saga of many years. For a long time, the fight was over the data itself. Even now, one thing that make paleoclimate a queasy subject IMHO is that a lot of the relevant data has never been published at all: http://climateaudit.org/2012/07/01/lonnie-and-ellen-serial-non-archivers/
I have no idea how anyone is supposed to judge the reliability of ice cores or tree rings or anything, when they only publish the sets they like. Say one went up and one went down and they got to choose... No one knows or can find out.

Anyhow, by now most everyone understands that you need to publish your data. But the second battle is over the methods and the code. Mann claims to get _this_ hockey stick. McIntyre and McKitrick ask him how he got it. He refuses to say, says that any idiot knows how to do elementary statistical methods. They try them and get a different result. He mocks them, saying that of course he used a different method. They try his method and get a different result. He mocks them, saying they did it wrong. Eventually it turns out that he did it wrong - but it doesn't matter because his result is right. Eventually it turns out that it does matter and his result was dependent on his method. No, that doesn't matter because there have been more recent studies that also make hockey sticks and we've all moved on from your silly quibbling. What methods did the new studies use...
This nonsense has been going on for more than a decade. Don't be surprised if those of us who were watching don't have much respect for the pronouncements of that group (who basically have done all the _independent_ studies), nor for their description of the issues, nor for their claim that all problems have long since been handled.

MikeR
9/21/2014 01:28:56 am

As far as consensus goes, this seems important:
http://online.wsj.com/articles/climate-science-is-not-settled-1411143565
In case most of you don’t know Steve Koonin – I do. He was a professor at Cal Tech when I was a student there, known as one of the brilliant young physics professors. He was scary smart. His book on Computational Physics is one of the most important in the field.

Why is this important?
http://judithcurry.com/2014/02/19/aps-reviews-its-climate-change-statement/
That was quite a few months ago, and I’d wondered what the APS was going to decide to do. The head of the APS committee: Steve Koonin.

Obviously, we want to know what climate scientists think. But we also have to keep tabs on the field as a whole. Has it become overwhelmed by politics? Is it doing shoddy work? Plenty of people are skeptical of social science results (sorry, Jose!) for that reason. If physicists start saying that about climate science, a lot of people are going to reassess.

Reply
Peter D. Tillman link
11/18/2014 04:43:49 am

Mr. Duarte:

Thanks for taking the time to look into this mess. And for calling a spade a spade:

"It will be hard to find a worse and more fraudulent study in a normal human lifespan. "

Do you know if ERL is actually looking into retracting this paper? It is remarkable, and disheartening, that stuff like this and Lewandowsky's gets published in respectable journals -- and then wins awards!

Best wishes,
Peter D. Tillman
Professional geologist, amateur climatologist

Reply



Leave a Reply.

    José L. Duarte

    Social Psychology, Scientific Validity, and Research Methods.

    Archives

    July 2017
    December 2015
    October 2015
    September 2015
    August 2015
    June 2015
    May 2015
    April 2015
    March 2015
    February 2015
    January 2015
    November 2014
    October 2014
    September 2014
    August 2014
    July 2014
    June 2014

    Categories

    All

    RSS Feed

Powered by Create your own unique website with customizable templates.
  • Home
  • Blog
  • Media Tips
  • Research
  • Data
  • An example
  • Haiku
  • About