When people in the United States ask Microsoft’s search engine Bing how big Syria is, they learn the country is 71,498 square miles and about equal to the size of Florida. When they ask Bing how many calories are in a serving of ice cream, they learn that a scoop contains 137 calories, which is equal to about 11 minutes of running.
These two-part answers supplied by Bing are early, real-world examples of a technology being developed inside Microsoft’s research labs to help us make sense of the jumble of numbers we increasingly encounter in the digital world.
“We want to reduce the number of times that people read a number and can’t make sense of it. And we want to do that by providing some context, or an analogy, or perspective, that puts it in more familiar terms usually related to their everyday experience,” said Jake Hofman, a senior researcher in Microsoft’s New York research lab.
The need for a new way to understand numbers stems from the overwhelming abundance of data now available to help us make decisions about everything from federal budgets to personal health and environmental conservation, noted Dan Goldstein, a principal researcher in Microsoft’s New York research lab.
“The solution is a relatively low-tech one. Using perspective sentences is very simple and they help a lot,” he said. “What we’re finding is creating them is a difficult challenge because it requires not only understanding the proper numbers to compare the numbers to, but also understanding what people are familiar with, what kinds of comparisons people like, what kinds of things people can easily imagine.”
On the road to AI
The examples on Bing today are only available for a few specific subjects and required human input to develop. Ultimately, the Microsoft researchers aim to build a service that automatically generates perspectives for any number and communicates them with the ease of a skilled storyteller or teacher. This service would be able to pass a test for general artificial intelligence posed in 1950 by the British computer theorist Alan Turing.
“You would be very sure you were talking to a machine if it says 248,572 square miles as opposed to roughly the size of Texas when you asked it how big France was,” said Goldstein. “To pass the Turing test, you have to talk like a human; someone who can explain something in a way that is personalized to the audience.”
The road to this generalized, automated technology that takes raw numbers from sources such as email, social media feeds and search results and puts them in a personalized context is filled with hurdles. To clear them requires a deep understanding of the nuance and complexity of what makes humans human.
Microsoft’s New York research lab, where Hofman and Goldstein are based, is well suited to clear this hurdle, noted David Pennock, a principal researcher and the lab’s assistant managing director. The lab brings together social scientists and computer scientists to study not just computers, but people and how people behave with computers.
“There’s an extra piece that is important for AI, which is taking the result of the complex algorithm that does all its magic and then actually putting it in a presentable form for people,” said Pennock. “If you want to run a data-driven company, yes you want all the great data; yes, you want to run all the right experiments; and yes, you want to make decisions based on your data. But ultimately, you need it in a form that is presentable to a person who in the end makes the decision.”
Numbers in the news
Hofman and Goldstein started down this road on October 30, 2012. The researchers remember the day because it fell the day after Superstorm Sandy slammed the East Coast. They fought snarled traffic to reach an off-site meeting where they had a brainstorming session on new research directions.
“We proposed the idea of trying to make numbers in the news make sense to the average person,” said Hofman. “Everyone nodded and said, ‘Yeah, that sounds like a good idea.’ We had no idea how good of an idea it was, or wasn’t, or how hard of a problem it was to solve.”
To begin, the researchers recruited people to participate in an online experiment designed to quantify the value of perspectives for the comprehension of unfamiliar numbers. Some participants generated perspective sentences for numbers taken from news articles and others took a series of randomized tests to determine if the perspectives improved recall, estimation and error detection.
For example, a news article noted that “Americans own almost 300 million firearms.” That fact alone might be difficult to estimate or believe if never seen before, and recall even if seen in the past. The researchers found comprehension of U.S. gun ownership improved with the perspective that “300 million firearms is about one firearm for every person in the United States.”
The finding that perspective sentences help people understand numbers in the news prompted the researchers to begin teasing apart why perspectives work. Does merely the repetition of numbers increase memory? Do perspectives add fodder for our brains to noodle over and associate with, leading to more stuff to pull on when it comes time for recall? Do perspectives stake mental flags?
What’s more, are some perspectives better than others? Take the area of Pakistan, for example, which is 307,373 square miles. What comparative rank or measure best helps people understand how big – how much land – 307,373 square miles is? Perhaps, how long it would take to drive across? Or how big it is compared to U.S. states? If comparing to states, which state? Is twice the size of California more helpful than five times larger than Georgia?
“How do you figure out which of those is better? How do you do that in a principled way?” said Chris Riederer, who interned with Hofman and Goldstein while pursuing his Ph.D. at Columbia University and co-authored a paper that describes this phase of the research. “Essentially, what we did is we ran a big survey.”
Study participants compared country sizes and populations to the sizes and populations of various U.S. states. The results show that familiar states combined with simple multipliers, even if less precise, are best. For example, people in the U.S. grasp the area of Pakistan more easily when expressed as roughly twice the size of California than the technically more accurate five times larger than Georgia.
These findings were used to generate the country-area perspectives live on Bing today. Ask the search engine, “How big is Pakistan?” and you’ll learn the square-mile fact along with the pre-computed comparison to California.
Bing and beyond
Bing’s question and answer team is working on additional perspectives to increase comprehension of everything from gas mileage to planet sizes. Bing’s food and drink team deployed perspectives that express calories in terms of minutes of running, protein and sodium in percent of the daily recommendation, grams of sugar in teaspoons of sugar and milligrams of caffeine in cups of coffee.
The decision on how to express each perspective – calories in minutes running versus walking, for example – involves brainstorming over email between the Bing and research teams as well as analysis of data from search logs and surveys, explained Christina Ntouniaoglou, a program manager for Bing’s food and drink team.
“I was thinking it is walking. Why would it be running? There are people who cannot really run. But the survey proved that people actually like the running part, so we went with that,” she said.
The next challenge, said Hofman, is to build a system that automatically creates perspectives so that people can more easily use all the data we have access to today to make informed decisions.
“Computers have lots of facts in lots of databases, but they don’t really know how to rank those facts as more or less useful, or comprehensible, to humans,” he said. “That is the last remaining hurdle – big hurdle – that we need to clear in this project.”
Hofman and Goldstein are applying the latest advances in machine learning, a branch of artificial intelligence, and data analysis to clear this hurdle. Their eyes are fixed on the goal of a generalized service that operates as a plug-in to browsers, email programs and text editors that automatically generates relevant, personalized perspectives for any numbers the users encounter or write.
“If we were infinitely wise and infinitely good at calculating, it wouldn’t really matter how numbers are expressed, it would all be the same to us. But the fact is, some things really cause people to go ‘Aha, now I get it,’” said Goldstein. “This is new territory; looking at how to communicate numbers in a way that gives people insight and memory and comprehension.”
The half decade Hofman has spent on the research project, he said, has already planted perspectives in his brain.
“I am always in the background thinking, ‘Am I presenting this in the most comprehensible way?’”
Related:
- Read the paper: To put that in perspective: Generating analogies that make numbers easier to understand
- Read the paper: Improving Comprehension of Numbers in the News
- Read: Microsoft researchers bring computer power to social science
- Follow Jake Hofman and Dan Goldstein on Twitter
John Roach writes about Microsoft research and innovation. Follow him on Twitter.
