|
Lady Gaga
Mexicans Mexican food Social Science Central Park 5 Adam Cohen Steve Neuberg Jonathan Haidt Carol Dweck Peyton Manning Phil Tetlock Lee Jussim Charlotta Stern Aly Light Noah Goldstein John Cacioppo Acacia Parks Jarret Crawford Kris Preacher Andrew Hayes Martin Seligman R Other Science Dick Feynman Space.com Phys.org Eric Drexler Centauri Dreams Kurzweil AI Nature Nano Nature News Nature Physics Crop Biotech Sam Harris Retraction Watch Climate RealClimate Gavin Schmidt Climate Etc. Climate Audit Climate Dialogue NOAA Data |
Good story in Pacific Standard
Angela Chen wrote an interesting story on my and my colleagues' work. The opening about how the University of Arizona interrogated me about my views of Jimmy Carter brings back memories... I don't think I had much of a view of Carter either way. I had objected to what he did in one instance, and it became a basis for discriminating against me. That's how insanely biased and discriminatory academic social psychology is... Anyway, Pacific Standard was a nice discovery – it's a good publication. Highly recommended. Another nice story in the Chronicle of Higher Education Nice story about our the BBS paper. I like that Bartlett also went back to my old 2011 essay, and quoted my favorite part: "If a research question requires that one assume that a particular ideology or value system is factually true, then that research question is invalid." I'll develop the methodological validity issues more thoroughly in a follow-up paper, eventually a textbook. We're in press at BBS! Duarte, J. L., Crawford, J. T., Stern, C., Haidt, J., Jussim, L., & Tetlock, P. E. (in press). Political diversity will improve social psychological science. Behavioral and Brain Sciences. Update: Cambridge University Press has now posted a preprint of the article. Some coverage of my chapter on well-being measurement
The Chronicle of Higher Education has a story that discusses my chapter on well-being measurement. The chapter appears in the Handbook of Positive Psychological Interventions (Parks & Schueller, Eds., 2014). My chapter explores the need to go beyond subjective well-being (usually measured as positive affect and life satisfaction, sometimes just life satisfaction) in how we measure well-being as such. Ignore climate consensus studies based on random people rating journal article abstracts
Ignore them completely – that's your safest bet right now. Most of these studies use political activists as the raters, activists who desired a specific outcome for the studies (to report the highest consensus figure possible), and who sometimes collaborated with each other in their rating decisions. All of this makes these studies completely invalid and untrustworthy (and by customary scientific standards, completely unpublishable.) I had no idea this was happening. This is a scam and a crisis. It needs to stop, and those papers need to be retracted immediately, especially Cook, et al (2013), given that we now have evidence of explicit bias and corruption on the part of the raters. (It's crazy that people think the consensus needs to be artificially inflated to absurd heights – do they think 84% or 90% isn't good enough?) In social science, it's common to use trained human raters to subjectively rate or score some variable — it can be children's behavior on a playground, interviews of all kinds, and often written material, like participants' accounts of a past emotional experience. And we have a number of analytical and statistical tools that go with such rating studies. But we would never use human raters who have an obvious bias with respect to the subject of their ratings, who desire a specific outcome for the study, and who would be able to deliver that outcome via their ratings. That's completely nuts. It's so egregious that I don't think it even occurs to us as something to look out for. It never happens. At least I've never heard of it happening. There would be no point in running such a study, since it would be dismissed out of hand and lead to serious questions about your ethics. But it's happening in climate science. Sort of. These junk studies are being published in climate science journals, which are probably not well-equipped to evaluate what are ultimately social science studies (in method). And I assume the journals weren't aware that these studies used political activists as raters. Examples of the unbelievable bias and transparent motives of the raters' in Cook, et al (2013) below. These are excerpts from an online forum where the raters collaborated with each other in their ratings: "BTW, this was the only time I "cheated" by looking at the whole paper. I was mystified by the ambiguity of the abstract, with the author wanting his skeptical cake and eating it too. I thought, "that smells like Lindzen" and had to peek." "Man, I think you guys are being way too conservative. Papers that talk about other GHGs causing warming are saying that human GHG emissions cause global warming. How is that not an implicit endorsement? If CFC emissions cause warming because they're GHGs, then CO2 emissions cause global warming for the same reason. That's an implicit endorsement." Jesus. This is a joke. A sad, ridiculous, confusing joke. And it's exactly what you'd expect from raters who are political activists on the subject they're rating. Who in their right minds would use political climate activists as raters for a serious report on the consensus? This is so nuts that I still have a hard time believing it actually happened, that the famous 97% paper was just a bunch of activists rating abstracts. I've called on the journal – Environmental Research Letters – to retract this paper. I'm deeply, deeply confused how this happened. If this is what we're doing, we should just call it a day and go home – we can't trust journals and science organizations on this topic if they're going to pull stunts like this. I don't care who you are – even if you're a staunch liberal, deeply concerned about the environment and the effects of future warming, this isn't something you should tolerate. If we're going to have a civilization, if we're going to have science, some things need to be non-political, some basic rules need to apply to everyone. I hope we can all agree that we can't seriously estimate the AGW consensus by having political activists rate climate paper abstracts. It doesn't matter whether the activists come from the Heritage Foundation or the Sierra Club – people with a vested interest in the outcome simply can't be raters. We don't need random people to interpret climate science for us, to infer the meaning of abstracts, to tell us what scientists think. That's an awful method – extremely vulnerable to bias, noise, incompetence, and poor execution. The abstracts for many papers won't even have the information such studies are looking for, and are simply not written at the level of abstraction of "this study provides support for human-caused warming", or "this study rejects human-caused warming". Most climate science papers are written at a more granular and technical level, are appropriately scientifically modest, and are not meant to be political chess pieces. There's a much better method for finding out what scientists think — ask them. Direct surveys of scientists is a much more valid method than having ragtag teams of unqualified political activists divine the meanings of thousands of abstracts. I don't mean ask about them their abstracts, as Cook, et al did – that inserts an unnecessary layer and potential selection bias. I mean ask them directly what they think about the principal questions. Interestingly, but not surprisingly, survey studies tend to report smaller consensus figures than the abstract rating studies (I'll have more on that later, see the Bray and von Storch series for now) The consensus will be strong regardless, so it's especially confusing why people feel the need to rig it. (For subjective ratings of abstracts to be a valid and useful method, it would need to be a carefully selected pool of raters, without ideological agendas, implementing a very specific and innovative method, under strict procedures of independence. I can imagine deep philosophy of science questions that might be anwerable by such methods, things like the usage of certain kinds of words, the way hypotheses are framed and results reported, etc. – but much of that could be done by computers. The studies that have been published are nothing like this, and have no hope of being valid.) NOTE: The Cook, et al data was leaked or hacked a few months ago – I'm confused by what's going on here. Cook wouldn't release some of his data, and ultimately a bunch of data was hacked or scraped off a server, and it included the raters' online discussion forum. Climate science features far too many stories of people refusing to release their data, and mysteriously hacked data. The person who posted this data, Brandon Shollenberger, is a complete unknown. It's amazing that if it weren't for him, we wouldn't know how rigged the study truly was. There's much more to report – the issues raised by the leaked dataset extend far beyond the quotes above. The University of Queensland has apparently threatened to sue Schollenberger, on some sort of "intellectual property" grounds. Australia is one of my favorite countries, but we need to stand up for him. To the best of my knowledge, he hasn't done anything wrong – he hasn't posted any sort of sensitive information or anything that would violate our core principles of scientific ethics. The identities of the raters were not confidential to begin with, so there was no new disclosure there. He's exposed the cartoonish bias and corruption of the rating process that underlied this "study", and in so doing, he's served the interests of scientific ethics, not violated them. Even if those online discussions took place during the training period, it would still be alarming evidence of bias, but other evidence suggests this was not a training period. I've never heard anyone call scientific data "intellectual property" before – that's an interesting legal theory, since this is not about an invention or an original creative work. Obviously, if scientists were to get into the habit of treating data as IP, or otherwise proprietary, it would seriously impair scientific progress and quality control – it would also violate the basic premise behind peer review. Shollenberger's disclosures took place in a context where the authors refused to release all of their data, so I'm not sure what other options there were for him. In other words, he's a whistleblower. You can contact the research governance people at the University of Queensland here (scroll to the bottom of that page). Update: In their legal threat letter to Shollenberger, the University of Queensland says that the letter itself is intellectual property, and that publication of the letter is cause for separate legal action. What? That's like an NSL. Is this new? What kind of upside-down bizarro world is this? You can send someone a threat letter, copyright the letter, and force them not to disclose it? This is unbelievably creepy. False Confession
The video below captures the reality and features of false confessions better than any case I've seen. False confessions happen all the time. It's counterintuitive – why would anyone confess to a crime they didn't commit? For lots of reasons actually, and Christopher Ochoa's account below is a vivid and absolutely heartbreaking illustration of how a completely innocent man can be induced to confess to murder and rape. Even more shocking is how five innocent boys could be induced into confessing to an unspeakably brutal rape. Police should be required to video record every second of interrogations, and both the accused and the interrogator(s) should be visible at all times. In my view, one of social psychology's greatest contributions to the world is exposing and exploring the reality of false confessions. For an experimental investigation of false confession, see this paper by Kassin and Kiechel. For a comprehensive treatment, see Gudjonsson's book below. Media notes on climate consensus story below:
Media can contact me here. Thanks for asking, but feel free to quote anything I've written on this website – you don't need to clear it with me. You also have my default permission to quote anything I tell you by e-mail (unless I say otherwise in the e-mail.) Dates: I called for the retraction of the Cook, et al (2013) paper on June 3, 2014 by e-mailing Environmental Research Letters editor Daniel Kammen. Yes, ERL is the same journal that allegedly rejected a climate science paper earlier this year because of how it would play politically if published, but that is a completely separate issue and I'm not the best person to ask about it. The paper in question was authored by Swedish climatologist Lennart Bengtsson, and some collaborators I believe, but I know very little about it. The journal's publisher issued a statement on that issue here. The 97% paper was authored by John Cook, et al. The Bengtsson paper is a climate science paper. The Cook paper is not – it's a paper purporting to describe the consensus among climate scientists regarding AGW, basically a social science or survey type of paper. ERL doesn't normally publish social science type papers, and that might partly explain how the paper got through, but we'd need to see the reviews to know whether the reviewers were knowledgeable about the appropriate methods for this kind of rating study – for all I know, they tapped famous social scientists and interrater reliability experts as reviewers. I have no idea. But you don't need experts to know that we can't use raters who are ideologically motivated to see a particular outcome from their ratings... This has probably never happened in such a brazen way before. Examples of valid content analysis of written material can be found here and here. Climate science is biased, but right
(This one of many treatments of specific issues in science that I'll post here. There's a narrower post on a related topic in the blog section.)
Over the last few months, I've been alarmed by what I've discovered in looking into the research on the climate science consensus. There's clearly a consensus on AGW, but many of the research reports on the consensus are remarkably shoddy, clearly biased, and would not survive a social science review process. In some cases, the researchers seem to have no training in how to conduct such studies, because they're political activists, not researchers. Since the consensus will be there no matter what, it's amazing that people feel the need to inflate it, to rig it. The American Association for the Advancement of Science (2014) broke my heart, by releasing a wildly unscientific report that cherry-picked only the studies that gave it the inflated consensus figures it wanted -- many of which are so bad as to be inadmissable. When scientists want to review a body of research, they conduct a meta-analysis that includes all the research that meets certain criteria of rigor and validity. The AAAS strangely chose not to perform a meta-analysis -- they simply ignored most studies, and cherry-picked four studies that gave them the inflated, shock-value numbers they wanted. Among the four was an obsolete one-page study from 2004 that doesn't clearly describe its methods (Oreskes, 2004, yes, really, one page long). That is, they skipped past all the more recent and credible studies from the intervening decade (e.g. Harris (2007), Bray and van Storch (2008), and others) to reach all the way back to a junk study from 2004. I've never seen such behavior – we clearly can't do anything with mysterious one-pagers from 2004. This isn't what I expected. It's clear that some climate scientists bring their politics into this. They leap to policy prescriptions and seem unaware of their ideological assumptions. Scientists are surprisingly not well-trained in separating ideological assumptions from descriptive facts, and don't seem to run bias-correction algorithms on themselves. Climate science displays many of the classic signs of groupthink, and the tenor of the debate is disturbingly hostile and malicious as a result. That said, I think the consensus of climate scientists regarding the reality of human-caused warming is both real and correct. A few reasons: 1. Bayesian inference is helpful here. When we see bias or groupthink in a field, we often discount the credibility of that field or their consensus. I think this is sometimes a mistake. There are two kinds of biased scientists -- right ones and wrong ones. If the climate consensus is correct, the climate science community would be biased and tribal. If the climate consensus is incorrect, the climate science community would be biased and tribal. In other words, we'll see this kind of bias, malice, and tribalism whenever there's a consensus, whether it's correct or not. The existence of bias doesn't refute the consensus...yet. 2. There really does seem to be a consensus. I have a lot of respect for Judith Curry, and I'm intrigued by Richard Lindzen and Roger Pielke Jr. -- three prominent climate scientists who are somewhat skeptical of the stronger versions of the AGW consensus. There are a number of other scientists in their camp. But they're seriously outnumbered, and it's not apparent to me that the dissenters are smarter than the much larger number of climate scientists who disagree with them (nor is it apparent that they're dumber). 3. I have a hunch. I think I know a little bit about science and scientists, and what good scientists look like. When I read some of the "mainstream" climate scientists, I often sense... real intelligence and competence. For example, when I read Gavin Schmidt, I'm pretty sure I'm dealing with a very intelligent scientist who uses valid methods, and who thoroughly understands those methods. I can only go on a hunch, because I'm not a climate scientist and can't validate their methods or models. (I can do PCA and other cross-field statistical methods.) You obviously don't have to care about my hunches. A story: When I needed surgery, a friend put me in touch with his uncle, a retired physician. His uncle gave me advice about what kind of surgeon to look for: Find someone youngish, but not too young, because you want someone who's up to speed on the latest methods and advances, with maybe a decade of experience, ideally someone who has made a name for herself. So, not someone fresh out of residency or internship, and not someone on the downslope toward retirement. I think this heuristic is probably good for a lot of scientific fields, especially those with complex new methods. I know this heuristic is somewhat ageist, or has the potential to be, but reality doesn't have to conform to our ideals. I've noticed that skeptical climate scientists tend to be much older than many of their detractors, and I wonder if it sometimes has to do with new methods that older scientists aren't well-versed in. In any case, if a 100-year-old scientist is right, she's right – her age won't override that reality at all. So, use wisely. (Alternatively, with age comes experience, and perhaps a certain kind of wisdom – older scientists might see right through an error or overconfidence in a model. My heuristic could be completely wrong on this one.) Broader considerations: That AGW is true has no inherent implications for policy. For one thing, severity or magnitude will matter. If the warming is only 1° C, that's a very different scenario than a 6° C change. Global warming is not a dichotomous or binary thing – it's a matter of degree, in every sense. You need to do some serious work to get from 1) AGW is true, to 2) Do something! We might value economic prosperity more than some increment of climate stasis. We'd also have to establish whether we owe the people of 2100 a very specific band of temperatures, and a very specific range of sea levels -- that's not obvious. We'd have to decide whether government should be an open-ended, unconstrained, intergenerational welfare-maximization engine, or a protector of individual rights on human lifespan timescales. There is a substantial body of evidence detailing the harms of giving government a coercive role in economic life -- see public choice theory, rent-seeking, regulatory capture, the knowledge problem, general economics, Hayek, Buchanan, Easterly, Cowen, Mankiw, Caplan, Epstein, the history of the 20th century, etc. (and many economists disagree with them -- I'm puzzled why economics isn't more unified.) There will be deep philosophical and ethical differences on whether we have the right to coerce billions of people for an unclear likelihood of preventing a 2-4 C increase in global mean surface temperatures by 2100. None of this is self-evident -- people will disagree. Beware centralized authorities and lofty scientific organizations. Climate science is going through a pompous phase right now, where they think that if they issue a report under the banner of the AAAS or the Royal Society or the IPCC, laypeople should just kneel before them. That's unscientific, un-American, and terrible epistemology. Authority and officialdom are not good heuristics for scientific truth, and clearly, organizations like the AAAS can no longer be trusted. I don't think climate scientists fully appreciate the fact that lots of people simply do not trust them -- and behavior like the AAAS' scam report will only further erode the public trust, and deservedly so. They need to have much higher standards, make it trivially easy to obtain their data, and always, always, always tell the truth. AAAS grossly misled the public about the quality of the evidence for their 97% consensus figure, and I can't tell you how much that crushed me -- they're a left-wing political advocacy organization at this point, not a scientific body. A scientific body would use robust scientific methods like meta-analysis, and carefully control for the political biases of its membership – cherry-picking junk studies is the well-worn tactic of mediocre political advocacy think-tanks. Finally, governments can have profound conflicts-of-interest in disseminating scientific findings, or any findings really, that have implications for their power -- politicians and agencies generally seek to expand their power, and jump on any rationale to do so. People can also be very motivated to see themselves as saviors. An example of errant government science: the US government might have shortened a lot of people's lives and increased the incidence of Type-2 diabetes with the scientific nutrition guidance it issued for several decades -- guidance that led millions of people to overconsume carbs in their efforts to avoid fat (nothing captures the culture this produced better than the non-fat yogurt episode of Seinfeld). I think government conflicts-of-interest on climate science are as serious as oil industry conflicts. So I'm not confident in government bodies that issue reports on climate – governments love the politics of fear, and they exploit it all the time, in the war on terror, with every financial crisis, immigration, and climate. The recent White House report made false claims about AGW being the cause of certain recent weather patterns -- claims that were rebuked in peer-reviewed journals. If you want a clean read on the true state of climate science, I'd be cautious with government products, like the IPCC, and steer clear of reports by committees hastily formed by politicians. Government in our era is heavily involved in science, and lots of good scientists receive government funding – people have to work within the system they're given (and I'm an NSF Graduate Research Fellow). That's not what I'm talking about – I mean what happens to science when politicians and lobbyists get involved, when people start forming committees and task forces and jockeying for position. I recommend you read the blogs of climate scientists, where they comment on newly published research, like RealClimate and Climate Etc., and the journal articles if you can. (Most of the raw data comes from government agencies, and I link to the NOAA in my sidebar. I don't have significant concerns about the data, and I don't buy the conspiracy theories about government-employed scientists doctoring data. It's important to live in Realityland, a place where evil is quite rare. These are good people, going to work every day to do a job, and do it well.) -- José Duarte June 4, 2014 Update on July 18, 2014: I've asked the AAAS to correct or retract their report with respect to the 97% figure, for the reasons noted up top, and because the principal paper behind that figure turns out to be a scam. A formal and long-lived scientific body like the AAAS should be a benchmark, should set the standard for rigor and objectivity – cherry-picking scam studies in order to (needlessly) bolster a political narrative is far afield from the role AAAS should occupy. I assume they'll retract or correct. If they don't, that report will likely become an example dissected by social scientists, philosophers of science, and future 8th graders when they learn about the politically corrupted science of the early 21st century, and the organizations that perpetrated it. (It might even be dissected by some future iteration of the AAAS itself, as a reflection on a shameful chapter.) I've also contacted the editor of Environmental Research Letters and called for the retraction of Cook, et al (2013), the "97%" paper. It was coded by political activists – they used political activists to rate the abstracts, activists who were passionately committed to the cause of global warming mitigation and wanted to be able to stress the size of the scientific consensus on AGW. They passionately desired a particular outcome for their "study", an outcome that, by virture of their role as raters, they had the power to deliver. That's an amazing sentence to have written, an amazing reality to encounter, in the year 2014. I've never heard of such a corrupt rating study. They also seemed to be completely unaware of the scientific methods of conducting a subjective rating study, provided none of the relevant analyses, etc. To get an explicit flavor of the corruption, see quotes like these: "BTW, this was the only time I "cheated" by looking at the whole paper. I was mystified by the ambiguity of the abstract, with the author wanting his skeptical cake and eating it too. I thought, "that smells like Lindzen" and had to peek." That's from one of the raters discussing his ratings with other raters on an online forum, and how he decided to cheat by looking past the abstract. The fact that the raters discussed their ratings with each other also means that rater independence was violated (the study claimed to use independent raters, which was both false and is ultimately meaningless since the raters were political activists with the same aims.) I'll have much more to say next week, and possibly a standalone website, about this paper, and the AAAS' myserious citation of it. I assume the paper will be retracted. The standard for retraction is normally much lower than this – it would be amazing for a journal to not retract a paper that used political activists as subjective raters where their ratings would serve their political aims. It would be more fodder for future 8th graders, but in the meantime would certify that science is in a very bad way right now, and that the public might be best advised to ignore us completely. Such a situation would have broad and harmful impacts far exceeding the pedagogical benefits to future 8th graders. NOTE: Some scientists I've spoken to have a useful heuristic. They never believed the 97% paper because they said it would be shocking to get 97% of scientists to agree on anything, much less a claim resting on such a young and dynamic field as climate science. This is probably right. You might be better off ignoring claims of 9X% in science, and science isn't really about consensus anyway. Moreover, a study based on random people, much less political activists, rating science paper abstracts, is a terrible way to find out what scientists think – it introduces broad new opportunities for bias, noise, and nonsense, when you could've just asked scientists directly what they think about something. Lastly, almost no one cares about "the consensus" on the question of whether human activity was chiefly responsible for the observed 20th century warming. That's a very trivial consensus. More important would be the expected magnitude of future warming, the degree of confidence in projections, the amount of unexplained variance, the likely consequences of given levels of warming, the economic tradeoffs of mitigation vs. adaptation (which would be the domain of economists, not climate scientists), etc. |
