Using the crowd to predict the future

I just finished Crowdsourcing by Jeff Howe, the definitive book on crowdsourcing, and although it was written nearly 4 years ago, I was really bowled over by key insights throughout the book. Crowdsourcing is more than just the design work or iStockphoto, there’s also an offshoot into the world of Crowd-predicting. Utilizing the wisdom of the crowds to predict anything from sports results, Hollywood sales or even Presidential elections, and it appears these prediction markets actually do a pretty good job of predicting the correct outcome. They’re not right 100% of the time (then again neither are the experts), but overall the Crowds can –and do–predict with great accuracy.

The premise of crowd-predicting is simple. Get a whole bunch of people together and then ask them to predict the outcome of a particular event, once each individual prediction comes through you then aggregate that information to get the final result. Proponents of crowd-predicting say this result often beats the ‘experts’, and they have some data to back it up.

Think of it this way, in ‘Who Wants to be a millionaire’ , one of the lifelines is asking the audience, which is basically asking a crowd of diverse people. There is also the ‘ask the expert’ lifeline in which you can ask an expert or a friend to help you answer the question. In his book wisdom of the crowds (which I have still to read), James Surowiecki presents a compelling piece of data:

The ‘experts’ did okay, offering the right answer – under pressure – almost 65 percent of the time. But they paled into comparison to the audiences. Those random crowds of people with nothing better to do on a weekday afternoon than sit in a TV studio picked the right answer 91 percent of the time.

The audience which is nothing more than an average collection of very average people score 91 percent of them. Which means the crowd is correct 91% of the time. Of course this isn’t strictly ‘predicting’ but it does suggest we might be onto something in using crowds to predict answers.

Wisdom of the crowds

There is something powerful in the crowd, a collective wisdom that when correctly tapped could yield insights completely invisible to the experts. Consider for the a moment how a group of amateur bloggers managed to predict Apples 2010 earnings far more accurately than the professional bankers whose sole job it was to analyze Apple. According to a report from the brilliant Cult of the Mac Blog:

It’s not even close, either. Once ranked, the bottom twenty spots in scorecard accuracy all go to professionals being paid for their insight and accuracy. On average, professionals were off by a 9.04% margin. Meanwhile, nine out of the ten amateurs made the top ten, and overall were only over by a little under 4%.

Two things make this remarkably different from the millionaire question, firstly, this was a matter of predicting, no one had the right answer but amateurs were closer than the professionals. The second difference is even more interesting, predicting Apples earnings requires analyzing data, and then extrapolating that data to get a number. It’s not a matter of searching Google or just looking it up, it’s actually analysis of data–and for a company as secretive as Apple that’s a lot of analysis to do, yet even with all the advantages a full-time professional analyst would have over blogging amateurs–the amateurs won.

The question then becomes could we get the amateurs to predict other things, like the presidential elections or Hollywood blockbuster sales?

In-Trade & Iowa Electronic Markets

The answer is obviously Yes we Can!

In 2008, users from website called In-Trade not only predicted that Obama would be president in 2008, it correctly predicted the outcome of the 48 US states missing only Indiana and Missouri. That’s a pretty good prediction, something no expert would have come close to.

In-Trade is a rather unique site in the crowd-predicting market. It trades futures on outcomes like Presidential Elections to the weather, to the completion of certain landmarks (including the Freedom Tower in New York). Each trade is essentially a bet you make with another In-Trade member and the bet is worth real-money.

What the website does, is it allows you to buy or sell predictions. A prediction is something that will either come true or fail to come true and by certain date. You can either bet for a prediction or against a prediction and if you guessed correctly you get $10.00, otherwise you get $0.00.

That’s pretty similar to any other bet with a friend. Where in-trade really gets interesting, is that the you can buy these predictions at various prices depending on the date and time. So for instance right now, you can buy predictions that Barack Obama would win the 2012 Presidential Election for $5.77, meaning that if the prediction came true you’d get $4.23 (which is the the $10.00 payout minus your initial outlay of $5.77). If Obama lost you’d lose $5.77 (because you get $0.00 in return).

The price of the predictions is based on market forces and depending on who wishes to sell or buy, it’s just like the stock market. It even reacts to news like the stock market, so for instance if Obama were to announce that Hillary would be vice-President and Joe Biden would be Secretary of state, you’re likely to see that number fluctuate dramatically. The price of the predictions fluctuates as news filters in and the more  likely the prediction is to occur the pricier it becomes and vice-versa.

Another way of looking at the price of $5.77  for Obama winning is as a 57.7% chance of Obama winning the 2012 election, or a 42.3% chance he’d lose the next election. Using these rather interesting statistic, in-trade also acts as a giant prediction engine where the crowds (not just a handful of experts) are predicting the outcomes.

So how did they do in 2008 elections? Well according to electoralmap.net:

Intrade had the electoral vote count virtually dead on.  Traders gave Obama a razor thin lead in Missouri, but it was truly a coin toss.  The only state that was objectively wrong was Indiana.  Indiana has not gone for the Democrats in decades, and the polls indicated that it would go for McCain as well.  Both of those states being worth 11 electoral votes means that Intrade ended up with the totals being correct – ignoring the one vote tweak in Nebraska.

For an brief analysis on how In-Trade did in the 2008 Elections, head over here

While In-Trade isn’t exactly bustling with activity, the idea behind it originates from the academic circles. The Iowa Electronic Market run by University of Iowa was the forerunner is these predictive markets. Unlike In-Trade the Iowa Electronic Market (IEM) is run purely for academic purposes and with a cap of $500 per user. It’s still real money, but with much less fanfare.

So what’s the prediction for Obama winning 2012, well the DEM12_WTA which is the a share in a ‘winner-take-all’ model of the democratic president winning is $0.58 to the dollar, roughly translating to 58.8% chance, very similar to what In-Trade has. The difference is that the IEM proposes an unnamed democratic nominee while In-trade proposes Obama, and that difference is probably what accounts for the 1.1% differential.

Hollywood Stock Exchange & Betfair

However, it doesn’t just stop with Presidential elections. The Hollywood stock exchange(HSX), lets you predict the outcome of blockbusters and Academy award winners. In similar fashion users buy and sell bonds in movies and those bonds increase in value as the movie sales increase, so if the movie is a great success the bonds can be cashed in for a higher value than if the movie tanks.

There is one difference between HSX and In-Trade. HSX trades in fictitious Hollywood dollars as opposed to In-Trades real money. That being said the predictions are pretty solid.

Pretty solid though still comes with its surprises. Some predictions are nearly spot on, for instance just 0.04% off the mark for the film Forgetting Sarah Marshall. Others were completely off the mark, HSX users predicted My Super Ex Girlfriend would gross $8.603 million, as opposed to the $15.925 million it actually pulled, the difference of a whooping 84%.

HSX it seems does better for the mega blockbusters like Avatar and Pirates of the Carribean. For films whose budget exceeded $100million, HSX users manage to predict on average to with 24% of the actual gross, that increases to nearly 34% for films whose budgets are under $25million.

Still a lot to be done one suspects, however a recent article in FastCompany does hold promise for the future of crowd-predicting in the movie arena. The article illustrates two models of crowd-predicting, using HSX vs. a twitter algorithm, that scours the tweets of all the twitter-verse and compiles that data for specific movies:

To predict first weekend performance, they built a computer model, which factored in two variables: the rate of tweets around the release date and the number of theaters its released in. Lo and behold, that model was 97.3% accurate in predicting opening weekend box office. By contrast, the Hollywood Stock Exchange, which has been the gold standard for opening box-office predictions, had a 96.5% accuracy.

That’s pretty stunning. The article goes on to read that most Hollywood producers aren’t exactly lining up for twitter analytic data, but the promise of crowd-predicting still exist. Whether HSX would be the final solution remains to be seen.

A side note obviously is that Hollywood has been pretty adamant against HSX or any other form of ‘movie-betting’. The MPAA (not to be confused with their equally evil brother the RIAA), has lobbied against the use of HSX claiming among others that:

While the industry’s opposing comments were not yet final on Wednesday afternoon, Mr. Pisano and others said they were expected to cite a host of potential problems. Those include the risk of market manipulation in the rumor-fueled film world, conflicts of interest among studio employees and myriad contractors who might bet with or against their own films, the possibility that box-office performance would be hurt by short-sellers, difficulty in getting or holding screens for films if trading activity indicated weakness and the need for costly internal monitoring to block insider trades.

Betting vs. Trading

In all this predicting, where do draw the line between betting and trading. I guess it’s the same whether you’re trading in oil or betting on oil, whether you trade gold futures of bet gold-futures, whether you trade in Obama winning the election or losing the election. It’s all a matter of pitting your guesstimate against everyone else and see how things play out, that guesstimate can either be backed in real dollars or just plain ol’ game tokens.

Conclusion

The point isn’t about betting or trading, it’s about whether we can use the collective intelligence of the crowd to more accurately predict the outcomes. It’s about whether we can take data usually privy to only a few select ‘experts’ and put it out for the general public to digest, analyze and then predict, and whether those predictions would match or better the expert predictions?

Would we be able to predict more accurately earthquakes and tsunami if seismic data were made available to the masses and then trade the chances of earthquakes occurring in regions throughout the world to obtain an accurate prediction of Earthquakes in New Zealand or Tsunamis in Japan?

Crowd-predicting isn’t anywhere near prefect yet, but it does offer a glimpse of hope that as data becomes more freely available and we begin to tap the surplus potential of the billions already online, we could–in theory–improve out ability to predict future events without the need for experts.

Hollywood sign courtesy of :http://www.flickr.com/photos/photographerglen/5921490858/sizes/s/in/photostream/
Yes we can courtesy of : http://www.flickr.com/photos/20106852@N00/2239098608/sizes/m/in/photostream/
Crowdsourcing courtesy of: http://www.flickr.com/photos/adesigna/4983863106/