Monday, August 25, 2008

Fukuyama: Authoritarianism is still against history

The latest on the world ideologies front --

In the light of Russia's Georgia adventures, there's been lots of talk whether this represents a new rise of authoritarian Russia, which is presumably another nail in the coffin for U.S.-led liberal democratic hegemony in the world. Our "end of history" friend Francis Fukuyama just wrote an op-ed arguing that Russia and China are still not big threats to liberal democracy. There are some good points: Russia is behaving as an aggressive imperial power, but does not embrace a grand, exportable ideology with universal appeal. Similarly with China. They both still feel the need to pay lip service to democratic rituals and norms. Even Nicholas Kristof's hilarious column chronicling his experience with China's dubious protest registration system concludes that even a pale mockery of democracy is progress.

I still like Azar Gat's article which I wrote about last year, that Russia and China represent authoritarian capitalism, which will be an effective alternative to liberal democracy. Sure, it's not a war of ideologies, he argues, but now it looks like big successful nations can economically succeed without being very democratic. Furthermore, this should encourage others to not bother with democracy.


Saturday, August 16, 2008

A better Obama vs McCain poll aggregation

Update: Charles Franklin (of kindly emailed me with many interesting points on this post. One important note is that my technique isn't really "no smoothing" -- rather, there is now implicit smoothing within the polling houses, by assuming that responses are evenly distributed across the time interval of the poll.

I was looking at's page that aggregates many opinion polls on the Presidential race. Here, they have a chart that shows the many polls plus lowess fits:

So there's a trend of Obama recently declining. But it wasn't clear to me that the fitted curve was correct. I downloaded the data and started playing around with it.

Here are several more graphs I made, with different smoothing parameters for the lowess fit. Your interpretation completely changes depending which smoothing parameter you like best!

Well, maybe this is an argument to use rolling averages over a fixed number of days or something. But it would be nice to directly look at the data with a minimum of extrapolation or smoothing, since they can destroy or mask effects.

Turns out this is possible with this data set by using single-day smoothing. Every poll is taken over a range of days and the table says how many respondents there were, so I did a day-by-day calculation: take the weighted average of all polls on that day, assuming that a poll's responses were evenly distributed over its date range given. (The weighted average across the day's polls just means, it's as if on that day there was one big poll and we're just calculating the Obama vs McCain percentages for it.) And to further clean things up, I omitted polls that had "Don't plan to vote" numbers and only included ones with undecideds. ("NV" responses wildly differ over time, so I figured if you ask it on a poll it must skew things. Not sure though. This cleanup step might be superfluous.) I know I just argued smoothing is bad and now I'm doing it, but note that the time resolution for this polling data is only at the day-by-day level, so I'm not doing any smoothing across time windows. What's significant is aggregating across polls, binning per-day.

Anyway, I can now see some trends on the direct scatterplot. I'm only plotting the Obama percentage:

The amount of noise significantly goes down. It makes clearer there's been a negative trend for Obama recently.

On the perils of smoothing or not -- below is Gallup's graph from their daily tracking poll, which uses a 3-day rolling average. But below that are two graphs from Alan Abramowitz, attacking the day-by-day interpretations out there -- the left shows daily with no smoothing, but the right is with a 10-day rolling average.

The Huffington Post article claims the 10-day one is the safest to interpret -- surely the single-day one seems too noisy -- but honestly I think it's hard to say what amount of smoothing is acceptable. (Be careful comparing those new graphs vs the official Gallup one, as they're for different dates, I believe a week and a half off or so. The 3-day vs 10-day is somewhat similar over the same date ranges, I think?) It's so much nicer if you don't have to do smoothing across time windows, which is why I like my graph :). Of course, it would be best if political polling had better overall methodology and there was less noise, but that's another (big) post...

If you're interested in my code and the data, I threw it up at

Finally, there are interesting tie-ins between this poll methodology stuff and Mechanical Turk statistical aggregation. You can think of polling as a similar problem to data annotation: there is a true quantity out there in the world (the percentage of vote for Obama, or whether a certain email is spam or not) and annotators/polls are noisy signals somewhat reflecting that true value. Under certain assumptions, simple averaging is the best technique to estimate the true value. A new Dolores Labs blog post on this is coming ...

Thursday, August 14, 2008

East vs West cultural psychology!

Great anti-pop-science article of the moment -- Mark Liberman does a take-down of David Brooks' apparently careless column on cultural psych experiments that purport to show that East Asians are collectivist while Westerns are individualist.

From Liberman:

Question to Language Log: Is it correct that if you show an American an image of a fish tank, the American will usually describe the biggest fish in the tank and what it is doing, while if you ask a Chinese person to describe a fish tank, the Chinese will usually describe the context in which the fish swim?

Answer: In principle, yes. But first of all, it wasn't a representative sample of Americans, it was undergraduates in a psychology course at the University of Michigan; and second, it wasn't Chinese, it was undergraduates in a psychology course at Kyoto University in Japan; and third, it wasn't a fish tank, it was 10 20-second animated vignettes of underwater scenes; and fourth, the Americans didn't mention the "focal fish" more often than the Japanese, they mentioned them less often.

In fairness, I have sympathy for Brooks' style of presenting interesting results in a provocative way. In a short opinion piece you have to aggressively summarize. But I also like science that's not totally intellectually irresponsible. Not yet sure where this work stands...