Can We Trust Peter Turchin?

Summary: in one paragraph of Ultrasociety, Peter Turchin cites three papers. He misrepresents all of them and lies about the things these papers say.

This is a short-ish critique of Peter Turchin’s book Ultrasociety (perma). I’ll only cover 4 sentences from the book. They were taken from a single paragraph in the 4th chapter titled “Cooperate to Compete”.

Turchin:

Frederick Wiseman and Sangit Chatterjee sorted the Major League Baseball teams into four payroll classes, ranging from those with the biggest disparities to those with the smallest. Between 1992 and 2001, teams in the most equal class won an average of eight more games per season than those in the most unequal class. The corrosive effect of inequality on cooperation is not limited to baseball. The same effect was observed when researchers analyzed the performance records of soccer teams in Italy and Japan.

Here’s what the paper he refers to in the first two sentences (perma) says:

Table 3 shows that while there was no relationship between the Gini coefficient and team performance from 1985-1990, a relationship has existed since 1990. This relationship was strongest in the 1991-1997 time period when teams that had the greatest degree of equality of salaries won, on average, 9.3 more games than those teams whose individual salary distribution had the greatest amount of inequality. [emphasis mine]

The problem here is that the paper does not tell us anything about the time period between 1992 and 2001. The periods it has data on are 1985-1990, 1991-1997, and 1998-2008. Turchin’s dates are literally made up. Let’s move on to the last two sentences I quote from Ultrasociety:

The corrosive effect of inequality on cooperation is not limited to baseball. The same effect was observed when researchers analyzed the performance records of soccer teams in Italy and Japan.

Here he refers to two different papers, but before looking at them note that he says that the paper we just looked at somehow tells us something about causal effect of inequality of cooperation. The problem is of course that that paper only looked at the association.

Anyway, in the last quote Turchin refers to: Pay Dispersion and Performance in Teams (perma) and it says:

Our results show that higher pay dispersion has a detrimental impact on individual performances, but has no significant effect on cooperation.

This paper directly contradicts what Turchin says it says.

I can’t quote anything contradicting him from the last paper he cites here (sci-hub) for one simple reason: it does not analyze effect of wage disparity on cooperation. Turchin literally makes up stuff that is not in the paper. There’s no relation between the Turchin’s claim and his reference at all.

Do I need to check literally every reference in Ultrasociety to see if they were reported faithfully by Turchin? I might be overblowing it but how can I trust anything he writes after seeing three papers in succession being falsely reported by him? How can anybody trust him?

You can buy Ultrasociety on Amazon, where it has a 4.5 star rating :)

Edit: Peter Turchin wrote a response to this on his blog. Here is my response:

Thank you for your response! I do not believe that you did a fair job of replying to my critique, though.

1. 1985-1990 results do contradict your hypothesis. By ignoring entries that don’t reach statistical significance you inflate the chance of false positives and your analysis ceases to be valid.
2. 1985 is an outlier. Excluding it, average Gini coefficient fluctuates around .55 and does not show a steep upward trajectory.
3. It is still unclear where did you get 1992-2001 interval – it’s not mentioned anywhere in the paper. You can’t just bundle 1991-1997 and 1998-2002 and report the average of them.
4. Your causal claims are still based on purely correlation data.
5. You completely ignored my concerns about unfaithful representation of the other two studies, which contradict your hypothesis.