Can We Trust Peter Turchin?31 Dec 2015
This is a short-ish critique of his book Ultrasociety. I’ll only cover its one paragraph, which, I hope to show you, is absolutely enough. The paragraph was taken from the 4th chapter titled “Cooperate to Compete”. Here’s what Turchin writes:
Frederick Wiseman and Sangit Chatterjee sorted the Major League Baseball teams into four payroll classes, ranging from those with the biggest disparities to those with the smallest. Between 1992 and 2001, teams in the most equal class won an average of eight more games per season than those in the most unequal class.
Here’s what the paper he refers to says:
Table 3 shows that while there was no relationship between the Gini coefficient and team performance from 1985-1990, a relationship has existed since 1990. This relationship was strongest in the 1991-1997 time period when teams that had the greatest degree of equality of salaries won, on average, 9.3 more games than those teams whose individual salary distribution had the greatest amount of inequality. [emphasis mine]
There are three data points: one that contradicts him (1985-1990), and two that support him (1991-1997 and 1998-2002). Turchin just ignores the first one and goes on to report the latter ones.
The fact that the paper has zero mentions of the word “cooperation” doesn’t even seem like such a big problem anymore. (There are at least two more issues with Turchin’s reporting of this study, can you find them?). But let’s get back to the book:
The corrosive effect of inequality on cooperation is not limited to baseball. The same effect was observed when researchers analyzed the performance records of soccer teams in Italy and Japan.
Here he refers to two different papers. The first one is Pay Dispersion and Performance in Teams and it says:
Our results show that higher pay dispersion has a detrimental impact on individual performances, but has no significant effect on cooperation.</blockquote>
The paper Turchin cites directly contradicts the claim he makes referring to it! Amazing, isn’t it?
I can’t quote anything contradicting him from the second paper – if you read it, you’ll notice it does not analyze effect of wage disparity on cooperation. There’s no relation between the Turchin’s claim and his reference at all.
Do I need to check literally every reference in Ultrasociety to see if they were reported faithfully by Turchin? I might be overblowing it but how can I trust anything he writes after seeing three papers in succession being falsely reported by him? How can anybody trust him?
Oh, I lied, sorry! I’m not going to limit myself to that one paragraph. Here’s an insightful tweet:
Every time someone inappropriately draws conclusions about causality from a correlational study, a kitten 'coincidentally' dies.— Neuroskeptic (@Neuro_Skeptic) September 25, 2014
And here’s Ultrasociety again, try to find a connection:
[…] the performance of most MLB teams could be improved by making the pay within them more equal.
We know this because the quarter of MLB teams that have the most equal distribution of salaries win more games than the next quarter, which is still more level than average, but not as egalitarian as the first quarter. The second quarter wins more games than the third quarter, which in turn does better than the fourth, least equal quarter. In other words, performance of at least three-quarters of Major League Baseball teams could be improved by a more equitable distribution of player rewards!
You can buy Ultrasociety on Amazon, where it has a 4.5 star rating :)
Edit: This is my response to Peter Turchin:
Thank you for your response! I do not believe that you did a fair job of replying to my critique, though.
- 1985-1990 results do contradict your hypothesis. By ignoring entries that don’t reach statistical significance you inflate the chance of false positives and your analysis ceases to be valid.
- 1985 is an outlier. Excluding it, average Gini coefficent fluctuates around .55 and does not show a steep upward trajectory.
- It is still unclear where did you get 1992-2001 interval – it’s not mentioned anywhere in the paper. You can’t just bundle 1991-1997 and 1998-2002 and report the average of them.
- Your causal claims are still based on purely correlation data.
- You completely ignored my concerns about unfaithful representation of the other two studies, which contradict your hypothesis.
Feel free to share your thoughts on this post with me!