Last year there was a guy who started posting to effective altruism forums promoting his own site, which he thought ought to fall under the EA umbrella. These links identify the person and the organization; they’re not secret or anything, I just don’t want to give them additional attention.
Anyway, this person claimed to be doing outreach for EA, but his outreach practices were misleading and the quality of the work produced was honestly pretty terrible. Everyone kept giving him feedback, even pretty harsh feedback.
And he was really receptive! “That’s an excellent point,” he’d say, and “I’m so grateful that you took the time to tell me this,” and “I’ve learned so much from this engagement and will take this into account” and “I think it’s really reasonable of you to have this concern, and I’m so grateful you raised it with me”.
And then nothing would change.
This sort of impeded dealing with the guy. If you tell someone “this is dishonest and incompetent and has no place in this community” and they say “wow, thanks for the criticism, I’ll really take it to heart, it’s great to have so many people looking out for mistakes”, it’s hard to know what to do with that. It took months and comprehensive effort from a lot of people to finally describe the pattern clearly enough that the criticism couldn’t just be deflected by wholeheartedly accepting it.
I was reminded of this in the latest flareup of the social science replication crisis debates, which caught fire when the director of Cornell’s diet-science lab posted a blog post detailing (apparently obliviously) very worrying statistical practices:
I said, “This cost us a lot of time and our own money to collect. There’s got to be something here we can salvage because it’s a cool (rich & unique) data set.” I had three ideas for potential Plan B, C, & D directions (since Plan A had failed). I told her what the analyses should be and what the tables should look like. I then asked her if she wanted to do them.
Every day she came back with puzzling new results, and every day we would scratch our heads, ask “Why,” and come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually we started discovering solutions that held up regardless of how we pressure-tested them. I outlined the first paper, and she wrote it up, and every day for a month I told her how to rewrite it and she did. This happened with a second paper, and then a third paper (which was one that was based on her own discovery while digging through the data).
This approach to a data set is very likely to turn up spurious correlations, and readers of the blog post were surprised that Wansink would say he did that so casually. So they started digging and found a hundred fifty statistical errors in the four papers produced from that data set. (They published this result as “Statistical heartburn: An attempt to digest four pizza publications from the Cornell Food and Brand Lab”).
And Andrew Gelman writes:
One of the weird things about the Brian Wansink affair (“Pizzagate”) was how he responded to such severe criticism (serious in its content and harsh in its tone) in such an equanimous way.
Critic A: You pushing an unpaid PhD-student into salami slicing null-results into 5 p-hacked papers … Because more worthless, p-hacked publications = obviously better….? … I really hope this story is a joke. If not, your behaviour is one of the biggest causes of the proliferation of junk science in psychology and you are the one who should be shamed, not the postdoc.
Wansink’s reply: I understand the good points you make… .
Critic B: I’m very grateful to you for exposing how these valueless and misleading studies are generated… .
Wansink’s reply: Hi George, You are right on target… .
Critic C: There is something catastrophically wrong in the genesis of the papers you describe … the moment you start exploring your data, the p-value becomes worse than worthless: it is positively misleading. It misled you, the entire field of psychology and many more. To be honest, it’s even worse than this …
Wansink’s reply: Outstanding! Thank you so much for point out those two paper. (I downloaded the draft form and will be making it required reading for my team). You make outstanding points… .
This is exactly the thing our community experienced, practically to the word. It was eerie to see it recorded somewhere else. And now I’m wondering if it’s a common thing. You can’t be defensive about criticism, that’s quotable and easily used against you and widely understood to be a bad sign in an intellectual environment. But you can certainly be cheerfully grateful for (and then completely unresponsive to) criticism! And people who want to keep up their intellectually dishonest practices uninterrupted might sort of converge on this bizarre, vaguely surreal, way of handling complaints.