Microsoft researchers bring computer power to social science

It’s called the prisoner’s dilemma and it goes like this: You and a fellow gang member are in jail. One of you committed a heinous crime, but the prosecutor doesn’t know who. Instead, you are both booked on a lesser charge and serving a one-year sentence in solitary confinement.

But here’s the deal: The prosecutor has given you both an opportunity to rat. If you betray your fellow gang member but are not ratted on, you walk free while your buddy serves a three-year term. If you both rat, you both serve two years. The prosecutor knocks on your cell door. What do you do?

For decades, social scientists have studied how cooperation among humans evolves by observing people play multiple games of prisoner’s dilemma, each game lasting for a fixed number of rounds. Over the course of these games, the players balance the benefit of cooperation with the risk of exploitation and the temptation to rat with the risk of retaliation in subsequent games.

Findings from these prisoner’s dilemma studies suggest that players at first are likely to cooperate for several rounds, but eventually rat, hoping to get out of jail before they are exploited. What’s more, as players learn the game, they realize that the rational choice is to rat, which they do in earlier and earlier rounds in each subsequent game.

Cooperation, the studies suggest, eventually unravels.

That’s exactly what a team of computational social scientists from Microsoft’s research organization in New York expected to prove when they did an experiment that enabled 94 participants to play 400 10-round games of prisoner’s dilemma in a virtual lab over the course of 20 consecutive weekdays. That’s much longer than previous studies, which are typically based on observations of undergraduates playing in a room for an hour.

Duncan Watts, principal researcher at Microsoft’s New York lab.

In the new study, game play proceeded as expected for the first week. Players defected earlier and earlier in each 10-round game. But then the unraveling stopped; game play stabilized for the remainder of the month. When the researchers investigated why, they discovered that 40 percent of the players were so-called resilient cooperators who avoid unravelling even at significant personal cost.

“Thirty years ago, people were doing experiments that were almost exactly the same as this one, but the big difference is now we are able to do it for a month rather than an hour,” said Duncan Watts, a principal researcher in Microsoft’s New York lab and a pioneer in applying computational techniques to traditional social science problems. “The payoff is that we see behavior happening on very long timescales, and that behavior turns out to be really important for the long-run evolution of cooperation.”

Computational social science
The study, published Jan. 13 in the journal Nature Communications, was written by a team of researchers including post-doctoral researcher Andrew Mao and software developer Lili Dworkin. It’s the latest example out of Microsoft’s New York lab to illustrate how digital technologies are transforming the social sciences by providing new ways to gather, process and analyze data, which has traditionally been the greatest limiting resource in the social sciences.

“A lot of human and social activities that used to happen offline in an analog environment are now happening online in a digital environment, and that is generating a lot of data about interactions and behavior,” said Watts, who also is a professor at large at Cornell University. “It has opened a door to a new set of methods that generally come from computer science that can now be applied to social science.”

The prisoner’s dilemma research took advantage of virtual labs, which allow study participants to log on from anywhere. This longer time period brought to light human behavior that hadn’t been found in earlier prisoner’s dilemma studies, said Siddharth Suri, a senior researcher in computational social science in Microsoft’s New York lab.

“We previously did not have a scientific instrument that was capable of detecting this kind of behavior,” he said. “If you think back to the old days, the first person who invented the telescope could see things that no one else could see. Well, now that we have virtual labs, we can see things that we couldn’t see before.”

Explanation versus prediction
These advances present computer and social scientists an opportunity to better understand human and social phenomena by combining the social scientists’ knack for trying to understand causal behavior with the computer scientists’ desire to do predictive analysis, said Jake Hofman, a senior researcher in Microsoft’s New York lab, who applies statistics and machine learning to large-scale social data.

For example, the social science literature is full of studies that try to tease out the causal relationship between the level of education a person attains and their lifetime income. But these types of studies rarely focus on how accurately combinations of factors such as school type, class size and family background in addition to education level predict income.

Jake Hofman, a senior researcher in Microsofty’s New York lab, applies statistics and machine learning to large-scale social data.

Jake Hofman, a senior researcher in Microsoft’s New York lab, applies statistics and machine learning to large-scale social data.

By adopting methods from computer science such as machine learning, experts could better understand how much observed differences in incomes can be explained by other factors besides education, and therefore which interventions are likely to have the most impact, explained Watts.

“This gives another way to quantify how much you understand the phenomena,” added Hofman. “Namely, if you can do a reasonable job predicting future earnings under many settings, then you have some understanding of what’s going on. If you can’t, you’re missing something.”

Hofman and Watts, together with colleague Amit Sharma, published an essay in Science this February arguing that a greater emphasis on prediction in the social sciences would make the discipline “better, more replicable, and more useful.”

Incoherency problem
Another concern among social scientists is that, too often, individual theories are tested in isolation rather than in the context of other competing and sometimes contradictory theories.

“In real-world problems, lots of theories apply and we need to know how they all interact. And we can’t figure that out if we only ever consider each theory in isolation,” Watts said.

In the Jan. 10 issue of Nature Human Behavior, Watts argued that one approach to addressing this so-called “incoherency problem” is for social scientists to focus more on solving real-world problems.

“The problem-solving framework is useful because it takes a lot of these abstract ideas like replication, reproducibility and prediction and makes them very concrete,” he explained. “Did you solve the problem? If yes, great, continue to go. If not, go back and try again, you have not done something right.”

The approach echoes the framework that enabled computer scientists to achieve breakthroughs in artificial intelligence by focusing research on common tasks such as reducing word-error rate in speech recognition and the development of driverless cars. Advances on these tasks are tested on standard datasets agreed to by the computer science community.

Microsoft’s recent achievement of speech recognition technology that is as good as a professional human transcriptionist, for example, was based on an industry standard benchmark test.

Crisis mapping
To illustrate how solution-oriented social science can work, Watts pointed to a study he, Mao, Suri and other colleagues published in PLoS One in April 2016. It used a virtual lab to gain a deeper understanding of competing theories about team size and productivity.

Several studies suggest that people are more likely to slack off as team size increases because loafing is easier to get away with in a larger group. Yet, another set of studies indicate big groups are more efficient because they allow people to specialize and once people specialize they become more productive.

Siddharth Suri, a senior researcher in Microsoft’s New York lab, is an expert on implementing virtual labs.

Siddharth Suri, a senior researcher in Microsoft’s New York lab, is an expert on implementing virtual labs.

“Depending on which mechanism you are looking at, you can reach very different conclusions about whether big groups are better than small groups,” Watts said.

To figure out whether big groups really are better than small groups, the researchers designed an experiment where teams varying in size from 1 to 32 people collaborated online to map real-time information shared by affected populations on social media during Typhoon Pablo, a category 5 storm that hit the Philippines on December 4, 2012.

Crisis mapping is a real-world task typically performed by volunteers in the emerging field of digital humanitarianism. The social media posts used for the experiment are authentic – they were generated during the typhoon and used to create a real crisis map. The results from the experiment, Suri explained, could have a real-world impact on a real-world problem, which he said motivated the team of researchers.

“Disaster relief is a difficult problem. How do you marshal limited resources to save the most human lives, to reduce the greatest amount of human suffering?” he said. “Crisis mapping was our first crack at it. And the first question you might ask is, ’How big of a team do I need to make this map?’”

The researchers found pronounced social loafing as team size increased, but the loafing was outweighed by the benefit gained by the coordinated efforts of the larger group. Bigger groups, at least for the task of crisis mapping, are better.

“You couldn’t have gotten that result if you were just studying social loafing,” Watts said. “And I would say the same thing about the prisoner’s dilemma paper – it is just a little slice of reality. If you want to really understand cooperation in the wild, you have to pick a real problem.”


John Roach writes about Microsoft research and innovation. Follow him on Twitter.