How Microsoft is using prediction and polling tools to forecast the election

Who will win the U.S. presidential election?

For David Rothschild, an economist with Microsoft’s research organization in New York, providing continuously updated odds for the major party candidates is a prominent part of his daily job.

Rothschild runs his own website, PredictWise, that hosts his probability forecasts about the outcomes of public contests including political elections, reality shows and sporting events. The predictions stem from his academic research on digital betting markets as well as an evolving and growing trove of online polling, web search and social media data.

“These are different ways that individuals engage with an online system and either implicitly or explicitly answer a question and provide a data point,” Rothschild explains. For PredictWise, he employs a “method of aggregating that data to create forecasts.”

TWEET THIS
“This is the tip of the iceberg for market intelligence,” Rothschild notes.
“How people act on the web correlates to how they vote,” explains Walter Sun, team lead for Bing Predicts.

The method includes publishing the forecasts in real time to help ensure that he gathers the most relevant data available. In addition, he says, the website keeps him honest by showing he can do what he says he can do.

“And, of course, a lot of people enjoy it,” he adds. “It is a good way to spread what I am doing.”

Market and business intelligence
While Rothschild’s PredictWise is a personal website, his research helps inform several other efforts across Microsoft product groups. All the efforts analyze aggregated, anonymized data based on online searches, polls and other publicly available information to gauge public opinion and predict the outcome of future events. They are on full display this election season.

When Bing users search for information related to political campaigns, for instance, Bing Predicts, a prediction engine embedded in the search tool, drives several features associated with Bing’s election experience.

For example, the prediction engine uses anonymized, aggregated web search data and other information to predict the outcome of the presidential race. A search on “Ohio state prediction” reveals what candidate is predicted to win the battleground state and how the prediction has changed over time.

The same data is also used to make forecasts about everything from professional sporting events to reality television shows.

“How people act on the web correlates to how they vote,” explains Walter Sun, team lead for Bing Predicts.

The prediction platform leans heavily on signals coming from aggregations of web data, known as the wisdom of the crowd, to capture and incorporate real-time information in the predictions, such as the impact of a candidate dropping out of a race or an injury to a star player on an NFL team.

Walter Sun

Walter Sun

Over on MSN, interactive polling widgets are being used to engage visitors with the website’s election coverage.

And Microsoft Pulse, an audience-response technology, is being used during the presidential debates by major news broadcasters to engage their audiences on second screens with real-time questions that probe sentiment and feelings.

“From the broadcasters’ perspective, it forces people to actually pay attention to what is being said versus being drawn out into a whole other conversation,” says Lee Brenner, who leads market development for Microsoft’s Technology and Civic Engagement team. Audiences, he adds, “feel like they can respond and have their voice heard.”

All of these initiatives come at a time when traditional polling methods are becoming less effective because people have abandoned, or don’t answer, landline telephones. Rothschild says Microsoft, with its built-in expertise and background online, can help fill that gap with market and business intelligence tools and services.

“This is a Wild West field right now,” he says. “You are talking about a massive industry that has reached a great disruption with the demise of its standard tool in the telephone and the rise of a new and innovative one in online, a place that we happen to be in.”

Intelligence from betting markets
Rothschild came to the field via a fascination with prediction markets, which are typically online exchanges where people buy and sell contracts on upcoming events. Contract values rise and fall in line with the market’s expectation of a certain outcome, such as whether a candidate will win an election.

For example, on PredictIt, a legal real-money prediction market that Rothschild closely monitors, traders buy and sell contracts on events such as the upcoming presidential election. Here’s how it works:

Trader “A” thinks Hillary Clinton has a 70 percent chance of winning so offers to buy a “yes” contract for 70 cents. The website matches the offer with a seller who thinks Clinton’s odds of winning are lower. Trader “A” can sell the “yes” contract up until the election results are tallied. If Clinton wins, the “yes” contract will pay out $1. If Trump wins, the “yes” contract is worthless.

“Ultimately, prediction markets are a more transparent futures market where people are buying and selling contracts on whether or not a company will be put at a certain valuation at some point,” Rothschild says.

Like other markets and exchanges, each prediction market has quirks. So, for PredictWise, Rothschild aggregates data streams from multiple markets in a way that provides a consistent probability forecast. His model also pulls in polling data and other information, weighting the data streams according to his research on how the individual markets and polls function and interact with each other.

Making the fruits of the academic research publicly available in real time, he notes, also provides a window onto how real-time data flows influence real-time decisions and feed off each other.

“This is the tip of the iceberg for market intelligence,” he notes, “and where market intelligence is going.”

Research flows across Microsoft
The teams that do prediction modeling work together closely. For example, Rothschild collaborates with the MSN team to develop the questions asked in its polling widget. The goal is to generate a high response rate and, in turn, translate the response data into projections of the voting population.

Microsoft Pulse, which is already being implemented by several television networks to retain audience attention during the presidential debates, also has been adopted by market researchers as a low-cost replacement for hardware commonly used to collect preference data from focus groups, notes Brenner.

Lee Brenner

Lee Brenner

The Bing Predicts team recently partnered with Cortana Intelligence to create an enterprise offering of predictive analytics uniquely bundled with web, search and social data anonymously collected and aggregated in the cloud.

“For example, we can look at how people are searching for Xbox around announcements of new release dates, new games, and correlate that to how well they will sell in a given quarter,” explains Sun. “We use that signal for sales forecasting.”

Validation for the enterprise offering, Sun adds, comes from Bing Predicts’ record on forecasting everything from reality shows and NFL matchups to the primary races in the current election cycle.

Elections, the ultimate advertising campaign
“Ultimately,” notes Rothschild, “a political election is a large-scale advertising campaign for a product.”

The current U.S. presidential election, he adds, is providing researchers an unusual look at the impact of advertising and get-out-the-vote efforts. Leading up to the televised debates, the Clinton team’s budget earmarked for both was much larger than the Trump team’s budget.

“There is a real possibility that it is going to be very lopsided in spending on the ground,” Rothschild says. “It will be very interesting to see what happens.”

Related:

John Roach writes about Microsoft research and innovation. Follow him on Twitter.