How machine learning forecasts the future for careers at Bing

At first, it sounded like a disaster. Walter Sun, whose job was to make sure Bing gave users the most up-to-the-minute information about current events, was seeing a big spike in the number of people searching for facts about some type of plane incident on the Hudson River.

People who saw the jetliner come down began tweeting details almost immediately. Others who’d been aboard a passing ferry started sharing online photos of a rescue effort. It quickly became clear that instead of tragedy, a heroic pilot’s emergency landing had saved the lives of everyone on board.

Sun says the incredible speed with which the story emerged showed “the power of what’s on the Web in real time” and, along with similar Internet buzz he noticed about school closures during snowstorms, helped him come up with the idea for Bing Predicts. The innovative feature now regularly makes headlines for its ability to analyze massive amounts of Web activity to forecast the outcomes of elections, voting-based reality TV shows, sports matchups and more.

“Our leadership allows us to come up with new ideas and run with them,” says Sun, now a partner data scientist and development manager for Bing. “It’s one of the benefits of working for a big company like Microsoft.”

Sun started Bing Predicts as an “incubation project,” a new idea that leaders see strong potential in and agree to support as they move forward. Such projects essentially give employees the chance to forge their own path, doing something creative in an area they find the most fascinating.

“It’s about innovation and new ideas. The way we make progress is by developing new concepts, new features for a project, new algorithms, etcetera,” says Jan Pedersen, Bing’s chief scientist and distinguished engineer. “For the employees, getting an opportunity to follow where their passions go is also very important.”

Pedersen says such projects are among the many opportunities at Bing, where employees do interesting and challenging work with machine learning, big data and other fields where expertise is highly valued and career possibilities abound.

“Bing is kind of at the fulcrum or focus of this massive industry trend,” he says. “The job satisfaction is enormous because you’re working on technologies that are both interesting and important, and that can actually make a difference in people’s lives.”

Around the world, machine learning is transforming technology, using big data to bring new insights for businesses in a wide variety of industries and helping people achieve more in their daily lives.

“Machine learning is a hot area because there’s a lot of information out there. How do we make sense out of it? How do we find the needle in the haystack?” Sun says. “Using machine learning, data science and other ways to apply your computer science background to solving big data problems is a big area for Bing and Microsoft.”

Bing Predicts uses anonymized search queries, social media sentiment and other signals in Web activity in aggregate to predict outcomes. Machine learning models figure out which things are important — how long users stay on certain websites, for example — and how they correlate with how people will vote or whatever else the team is trying to predict.

“As soon as I heard about it, I knew it was something I wanted to work on,” says Rani Qumsiyeh, now a senior program manager for Bing Predicts. “The potential to grow something from the ground up, rather than work on iterations and different feature additions, is something I wanted to do.”

Qumsiyeh says that in less than five years at Bing, he’s gotten the chance to start two other products as if he were a part of a startup business, as well as “work with some of the smartest people in the world.”

“The reason I joined Microsoft was because of the fact that they’re so diverse in the things that they offer,” he says. “So I knew that whatever my passion was – whether I wanted to work in data, statistics, metrics, even gaming – Microsoft would offer me the opportunity to find the right thing to work on.”

Since its launch in 2014, Bing Predicts has racked up some impressive successes. It picked winners with well over 90 percent accuracy for that year’s midterm election Senate and House races, and its accuracy has also hit 90 percent or higher for the TV reality shows “American Idol” and “Dancing with the Stars.”

In the two months leading up to the 2014 referendum vote in Scotland, it predicted a vote range of 51 to 59 percent of voters rejecting the measure to grant the country independence, and they did, by just over 55 percent. This range was much tighter and more accurate than traditional polls were showing.  For NFL games, Sun says, it chalks up about 71 percent accuracy.

It was right about 17 of this year’s 24 Academy Award winners and went 20-for-22 at the Grammys.

“It turned out to be quite powerful,” Pedersen says. “Walter has developed a very strong winning streak in terms of his ability to predict.”

Kushal Lakhotia joined Bing in August 2013, after finishing his master’s in computer science at Purdue University, and later jumped at the chance to join the Bing Predicts team, which has now grown to include 15 people.

“I decided I would like to try out something that’s still in its nascent stage,” he says. “Neither the process or the technology had solidified, so it was an opportunity for me to contribute to shaping how we did things … There was more room for creativity. That was a major motivator.”

On Bing Predicts, Lakhotia writes code to analyze, process and build statistical models with data. He likes the way it’s not just abstract coding, but work that brings results he can actually explain to people who don’t have to be in a technical field to understand it.

He says it’s exciting to work in a fast-paced environment where he can “see an immediate impact” from his work.

“Bing is a super agile and innovative organization,” he says. “I’ve learned a lot here.”

Sun started thinking about the idea for Bing Predicts about seven years ago. In addition to watching the Hudson River crash landing unfold, he noticed something else that same winter: a sudden uptick in Bing users searching for specific school districts in Washington state.

People were scrambling to find out if their kids’ schools would be closed — which told Sun, without even looking at the forecast, that many people were expecting snow.

He realized that “when you’re looking at search queries, you can actually make inferences to other things,” he says. “And that was kind of the first step.”

Now, he says, Bing Predicts has all sorts of potential, and not just for projecting big pop-culture moments and offering insight into sports events like March Madness. Analyzing search-query data could bring great value to businesses because consumers often do online research before making buying decisions.

In the health care arena, Sun and his team have already put together a research paper on how search queries could provide early warning of adverse drug interactions because people often turn to their search engine to investigate certain symptoms even before asking their doctors.

“If we see a correlation of these types of queries, we could potentially inform the FDA or even the drug makers that there’s a possibility of some interactions they may want to investigate,” Sun says. “That could save lives and avoid a lot of medical bills.”

Sun says Microsoft’s “bottom-up approach” that fosters employees’ ability to explore their own ideas, as well as its resources and enormous amounts of data, were instrumental in bringing his idea to life.

Now, one of the best parts of his job is having even the most seemingly unlikely predictions come true — like when Bing Predicts forecast that Caleb Johnson would win season 13 of “American Idol” or Rumer Willis would win season 20 of “Dancing with the Stars.”

“People say ‘You’re crazy. You have no idea. There’s no way,’” he says. “And then a few weeks later, they come back and say, ‘Wow. Bing’s prediction actually worked out.’”