Next at Microsoft http://blogs.microsoft.com/next Tue, 24 May 2016 21:52:45 +0000 en-US hourly 1 https://wordpress.org/?v=4.5.2 Eric Horvitz receives ACM-AAAI Allen Newell Award for groundbreaking artificial intelligence work http://blogs.microsoft.com/next/2016/04/27/eric-horvitz-receives-acm-aaai-allen-newell-award-groundbreaking-artificial-intelligence-work/ http://blogs.microsoft.com/next/2016/04/27/eric-horvitz-receives-acm-aaai-allen-newell-award-groundbreaking-artificial-intelligence-work/#respond Wed, 27 Apr 2016 13:00:12 +0000 http://blogs.microsoft.com/next/?p=56836 In his many years as an artificial intelligence researcher, Eric Horvitz has worked on everything from systems that help determine what’s funny or surprising to those that know when to … Read more »

The post Eric Horvitz receives ACM-AAAI Allen Newell Award for groundbreaking artificial intelligence work appeared first on Next at Microsoft.

]]>
In his many years as an artificial intelligence researcher, Eric Horvitz has worked on everything from systems that help determine what’s funny or surprising to those that know when to help us remember what we need to do at work.

On Wednesday, Horvitz, a technical fellow and managing director of Microsoft’s Redmond, Washington, research lab, received the ACM – AAAI Allen Newell Award for groundbreaking contributions in artificial intelligence and human-computer interaction. The award honors Horvitz’s substantial theoretical efforts and as well as his persistent focus on using those discoveries as the basis for practical applications that make our lives easier and more productive.

Harry Shum, the executive vice president of Microsoft’s technology and research group, said Horvitz epitomizes a style of research that is unique to places like Microsoft because it is focused on having an impact in both the research and industry domains.

“People talk about basic research and applied research. What we are doing here is Microsoft research,” Shum said. “It’s not just about doing theoretical research and writing more papers. It’s also about applying those technologies in Microsoft products.”

Jeannette M. Wing, the corporate vice president overseeing Microsoft’s core research labs, said that Horvitz’s research has had an impact on countless research projects and commercial products, ranging from systems that help make our commutes easier to ones that seek to prevent hospital readmissions.

“His impact is immeasurable,” she said.

But Wing noted that Horvitz also has been able to step back and see the big picture, becoming a visionary and a thought leader in a field that is growing increasingly complex.

“He asks big questions: How do our minds work? What computational principles and architectures underlie thinking and intelligent behavior? How can computational models perform amidst real-world complexities such as sustainability and development? How can we deploy computation systems that deliver value to people and society?” Wing said.

The Newell award is given to a researcher whose work has breadth within computer science or spans multiple disciplines. Horvitz’s work has combined multiple computer science disciplines and he has been a leader in exploring the interrelationships between artificial intelligence and fields like decision science, cognitive science and neuroscience.

The award comes at a time when the artificial intelligence field is exploding.

Until a few years ago, artificial intelligence wasn’t often part of the public consciousness, except when it came up in a science fiction novel or blockbuster movie.

Now,  thanks to breakthroughs in the availability of data and our ability to process it, artificial intelligence applications are  suddenly everywhere, including systems that can understand and translate language, recognize and caption photos and do increasingly smart and useful things for us.

During a time often referred to as the “AI winter,”  Horvitz was among the nation’s hard-charging researchers plugging away at the difficult work of laying the groundwork for these systems and thinking about how they would work in the real world. Although artificial intelligence was out of the spotlight during that time, researchers were making major breakthroughs in bringing together the logical methods of traditional artificial intelligence work with research in fields such as decision science. This led to new applications that used both logic and probability.

Horvitz said that many of his research projects over the last fifteen years – which have looked at things like what we are most likely to remember or forget and when it’s worth it to interrupt someone while working – foreshadow practical applications that he expects to see in the future.

“To me, Eric is such an epic example of those brilliant researchers who have this huge confidence — not over-confidence, but just confidence — to keep pushing forward,” Shum said.

Horvitz’s attention to both research advances and practical applications of artificial intelligence research began while he was pursuing his Ph.D. on principles of bounded rationality. That’s the idea that when people or computers make decisions, they are limited by time, available information and their reasoning abilities.

Horvitz said he was interested in how computing systems immersed in the real world could make the best decisions in time-critical situations. His research looked at the value of continuing to think about a problem versus stopping early with a good enough answer.

His research considered emergency room scenarios, in which artificial intelligence systems could help doctors with timely recommendations. The work foreshadowed his later research on using similar ideas to guide solutions to some of the hardest challenges known in artificial intelligence, in the realm of theorem proving.

Horvitz also showed how artificial intelligence systems could be used to better understand people’s goals and intentions and provide the best information to decision makers. He collaborated with NASA’s Mission Control Center on how to provide flight engineers with the most valuable information about space shuttle systems when the engineers are under intense time pressure.

To solve these problems — and many more after — Horvitz brought together artificial intelligence methods with ideas drawn from disciplines like probability theory, decision theory and studies of bounded rationality.

In the future, Horvitz said he sees vast possibilities for how artificial intelligence can help to augment human intelligence.

“There’s a huge opportunity ahead in building systems that work closely with people to help them to achieve their goals,” Horvitz said.

Related:

The post Eric Horvitz receives ACM-AAAI Allen Newell Award for groundbreaking artificial intelligence work appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/04/27/eric-horvitz-receives-acm-aaai-allen-newell-award-groundbreaking-artificial-intelligence-work/feed/ 0
You might not see the next wave of breakthrough tech, but it’s all around you http://blogs.microsoft.com/next/2016/04/18/you-might-not-see-the-next-wave-of-breakthrough-tech-but-its-all-around-you/ http://blogs.microsoft.com/next/2016/04/18/you-might-not-see-the-next-wave-of-breakthrough-tech-but-its-all-around-you/#respond Mon, 18 Apr 2016 15:05:51 +0000 http://blogs.microsoft.com/next/?p=56797 Think of your favorite pieces of technology. These are the things that you use every day for work and play, and pretty much can’t live without. Chances are, at least one … Read more »

The post You might not see the next wave of breakthrough tech, but it’s all around you appeared first on Next at Microsoft.

]]>

Think of your favorite pieces of technology. These are the things that you use every day for work and play, and pretty much can’t live without.

Chances are, at least one of them is a gadget – your phone, maybe, or your gaming console.

But if you really think about it, chances also are good that many of your most beloved technologies are no longer made of plastic, metal and glass.

Maybe it’s a streaming video service you use to binge watch “Game of Thrones” on or an app that lets you track your steps and calories so you can fit into those jeans you wore back in high school. Maybe it’s a virtual assistant that helps you remember where your meetings are and when you need to take your medicine, or an e-reader that lets you get lost in your favorite book via your phone, tablet or even car speakers.

Perhaps, quietly and without even realizing it, your most beloved technologies have gone from being things you hold to services you rely on, and that exist everywhere and nowhere. Instead of the gadgets themselves, they are tools that you expect to be able to use on any type of gadget: Your phone, your PC, maybe even your TV.

They are part of what Harry Shum, executive vice president in charge of Microsoft’s Technology and Research division, refers to as an “invisible revolution.”

“We are on the cusp of creating a world in which technology is increasingly pervasive but is also increasingly invisible,” Shum said.

Read the full story.

The post You might not see the next wave of breakthrough tech, but it’s all around you appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/04/18/you-might-not-see-the-next-wave-of-breakthrough-tech-but-its-all-around-you/feed/ 0
Teaching computers to describe images as people would http://blogs.microsoft.com/next/2016/04/14/teaching-computers-to-describe-images-as-people-would/ http://blogs.microsoft.com/next/2016/04/14/teaching-computers-to-describe-images-as-people-would/#respond Thu, 14 Apr 2016 13:00:49 +0000 http://blogs.microsoft.com/next/?p=56611 Let’s say you’re scrolling through your favorite social media app and you come across a series of pictures of a man in a tuxedo and a woman in a long … Read more »

The post Teaching computers to describe images as people would appeared first on Next at Microsoft.

]]>
Let’s say you’re scrolling through your favorite social media app and you come across a series of pictures of a man in a tuxedo and a woman in a long white dress.

An automated image captioning system might describe that scene as “a picture of a man and a woman,” or maybe even “a bride and a groom.” But a person might look at the pictures and think, “Wow, my friends got married! They look so happy. What a beautiful wedding.”

As image captioning tools get increasingly good at correctly recognizing the objects in an image, a group of researchers is taking the technology one step further. They are working on a system that can automatically describe a series of images in the same kind of way that a human would, by focusing not just on the items in the picture but also what’s happening and how it might make a person feel.

“Captioning is about taking concrete objects and putting them together in a literal description,” said Margaret Mitchell, a Microsoft researcher who is leading the research project. “What I’ve been calling visual storytelling is about inferring conceptual and abstract ideas from those concrete objects.”

For example, while another image captioning system might describe an image as “a group of people dancing,” the visual storytelling system would instead say “We had a ton of fun dancing.” And while another captioning system might say, “This is a picture of a float in a parade,” this system would instead say “Some of the floats were very colorful.”

The research project, which relies on a new Microsoft Sequential Image Narrative Dataset, doesn’t just stop at one picture. Instead, it takes a series of pictures about the same event and strings together several sentences describing what’s going on. The work will be presented in June at the annual meeting of the North American Chapter of the Association for Computational Linguistics.

‘Ready for the next step’
The researchers say visual storytelling could eventually be helpful for people who are sharing a number of pictures on social media and want a tool that will help them build a narrative about those pictures. It also could potentially be used to provide richer descriptive tools for people who are blind or visually impaired.

“In image captioning, there are a lot of things we can do reasonably well, and that means we are ready for the next step,” said Ting-Hao (Kenneth) Huang, a Ph.D. candidate at Carnegie Mellon University who worked on the project as part of a summer internship at Microsoft Research. “I think the computer can generate a reasonably simple story, like what we see in a children’s book.”

Huang was the first author on a paper about the work, along with another summer intern from Johns Hopkins University, Francis Ferraro.

 

‘Translating’ from images to sentences
The fields of computer vision and natural language processing have made significant advances in the past few years. That’s thanks in part to the more widespread use of a machine learning methodology called deep neural networks. These methods have helped researchers get much more accurate results for pattern recognition tasks like speech recognition and identifying objects in photos.

To build the visual storytelling system, the researchers used the deep neural networks to create a “sequence to sequence” machine learning system that is similar to the kind other computer scientists have used for automated language translation. In this case, however, instead of translating from, say, French to English, the researchers were training the system to translate from images to sentences.

For a machine learning system to work, it needs a training set of data that it can learn from. To build the visual storytelling system’s training set, the researchers hired crowdsourced workers to write sentences describing various scenes. To account for variations in how people described the scenes, the tool was trained to prefer language in which there was consensus, and to create sentences based on that common ground.

The team also created a separate test set, so they could compare the machine’s descriptions with how a human described the scene.

Then, they fed the system new images and asked it to create sentences based on the knowledge it had from the training set.

 

The research is still in the early stages, and the researchers admit there’s significant progress to be made. Still, the researchers say these most recent advances represent another milestone in the fast-moving effort to use machine learning and other methods from the broader field of artificial intelligence for valuable applications. The new work on visual storytelling brings artificial intelligence a step closer to interpreting the world in the complex, nuanced ways that humans do.

“A picture is worth 1,000 words. It’s not just worth three tags,” Mitchell said.

Still, the researchers caution that this system – and other cutting-edge research projects like it – are still far from reaching a human level of cognition.

“We’re really all scratching the surface,” said Nasrin Mostafazadeh, a Ph.D. candidate at the University of Rochester who worked on the project as an intern at Microsoft Research. “It’s not that we’re doing it, really, in the way that humans do it. It’s just that we’re trying to.”

Related:

Learn more about the Microsoft Sequential Image Narrative Dataset

Read the paper: Visual Storytelling, by Ting-Hao (Kenneth) Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley and Margaret Mitchell.

Decades of computer vision research, one ‘Swiss Army knife’

Project Malmo: Using Minecraft to build more intelligent technology

The quest to create technology that understands speech as well as a human

Allison Linn is a senior writer at Microsoft. Follow her on Twitter

 

 

 

 

 

 

The post Teaching computers to describe images as people would appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/04/14/teaching-computers-to-describe-images-as-people-would/feed/ 0
Decades of computer vision research, one ‘Swiss Army knife’ http://blogs.microsoft.com/next/2016/03/30/decades-of-computer-vision-research-one-swiss-army-knife/ http://blogs.microsoft.com/next/2016/03/30/decades-of-computer-vision-research-one-swiss-army-knife/#respond Wed, 30 Mar 2016 21:00:21 +0000 http://blogs.microsoft.com/next/?p=56506 When Anne Taylor walks into a room, she wants to know the same things that any person would. Where is there an empty seat? Who is walking up to me, … Read more »

The post Decades of computer vision research, one ‘Swiss Army knife’ appeared first on Next at Microsoft.

]]>
When Anne Taylor walks into a room, she wants to know the same things that any person would.

Where is there an empty seat? Who is walking up to me, and is that person smiling or frowning? What does that sign say?

For Taylor, who is blind, there aren’t always easy ways to get this information. Perhaps another person can direct her to her seat, describe her surroundings or make an introduction.

There are apps and tools available to help visually impaired people, she said, but they often only serve one limited function and they aren’t always easy to use. It’s also possible to ask other people for help, but most people prefer to navigate the world as independently as possible.

That’s why, when Taylor arrived at Microsoft about a year ago, she immediately got interested in working with a group of researchers and engineers on a project that she affectionately calls a potential “Swiss Army knife” of tools for visually impaired people.

“I said, ‘Let’s do something that really matters to the blind community,’” said Taylor, a senior project manager who works on ways to make Microsoft products more accessible. “Let’s find a solution for a scenario that really matters.”

That project is Seeing AI, a research project that uses computer vision and natural language processing to describe a person’s surroundings, read text, answer questions and even identify emotions on people’s faces. Seeing AI, which can be used as a cell phone app or via smart glasses from Pivothead, made its public debut at the company’s Build conference this week. It does not currently have a release date.

Taylor said Seeing AI provides another layer of information for people who also are using mobility aids such as white canes and guide dogs.

“This app will help level the playing field,” Taylor said.

At the same conference, Microsoft also unveiled CaptionBot, a demonstration site that can take any image and provide a detailed description of it.

Very deep neural networks, natural language processing and more
Seeing AI and CaptionBot represent the latest advances in this type of technology, but they are built on decades of cutting-edge research in fields including computer vision, image recognition, natural language processing and machine learning.

In recent years, a spate of breakthroughs has allowed computer vision researchers to do things they might not have thought possible even a few years before.

“Some people would describe it as a miracle,” said Xiaodong He, a senior Microsoft researcher who is leading the image captioning effort that is part of Microsoft Cognitive Services. “The intelligence we can say we have developed today is so much better than six years ago.”

The field is moving so fast that it’s substantially better than even six months ago, he said. For example, Kenneth Tran, a senior research engineer on his team who is leading the development effort, recently figured out a way to make the image captioning system more than 20 times faster, allowing people who use tools like Seeing AI to get the information they need much more quickly.

A major a-ha moment came a few years ago, when researchers hit on the idea of using deep neural networks, which roughly mimic the biological processes of the human brain, for machine learning.

Machine learning is the general term for a process in which systems get better at doing something as they are given more training data about that task. For example, if a computer scientist wants to build an app that helps bicyclists recognize when cars are coming up behind them, it would feed the computer tons of pictures of cars, so the app learned to recognize the difference between a car and, say, a sign or a tree.

Computer scientists had used neural networks before, but not in this way, and the new approach resulted in big leaps in computer vision accuracy.

Several months ago, Microsoft researchers Jian Sun and Kaiming He made another big leap when they unveiled a new system that uses very deep neural networks – called residual neural networks – to correctly identify photos. The new approach to recognizing images resulted in huge improvements in accuracy. The researchers shocked the academic community and won two major contests, the ImageNet and Microsoft Common Objects in Context challenges.

Tools to recognize and accurately describe images
That approach is now being used by Microsoft researchers who are working on ways to not just recognize images but also write captions about them. This research, which combines image recognition with natural language processing, can help people who are visually impaired get an accurate description of an image. It also has applications for people who need information about an image but can’t look at it, such as when they are driving.

The image captioning work also has received accolades for its accuracy as compared to other research projects, and it is the basis for the capabilities in Seeing AI and Caption Bot. Now, the researchers are working on expanding the training set so it can give users a deeper sense of the world around them.

Margaret Mitchell

Margaret Mitchell

Margaret Mitchell, a Microsoft researcher who specializes in natural language processing and has been one of the industry’s leading researchers on image captioning, said she and her colleagues also are looking at ways a computer can describe an image in a more human way.

For example, while a computer might accurately describe a scene as “a group of people that are sitting next to each other,” a person may say that it’s “a group of people having a good time.” The challenge is to help the technology understand what a person would think was most important, and worth saying, about the picture.

“There’s a separation between what’s in an image and what we say about the image,” said Mitchell, who also is one of the leads on the Seeing AI project.

Other Microsoft researchers are developing ways that the latest image recognition tools can provide more thorough explanations of pictures. For example, instead of just describing an image as “a man and a woman sitting next to each other,” it would be more helpful for the technology to say, “Barack Obama and Hillary Clinton are posing for a picture.”

That’s where Lei Zhang comes in.

When you search the Internet for an image today, chances are high that the search engine is relying on text associated with that image to return a picture of Kim Kardashian or Taylor Swift.

Zhang, a senior researcher at Microsoft, is working with researchers including Yandong Guo on a system that uses machine learning to identify celebrities, politicians and public figures based on the elements of the image rather than the text associated with it.

Zhang’s research will be included in the latest vision tools that are part of Microsoft Cognitive Services. That’s a set of tools that is  based on Microsoft’s cutting-edge machine learning research, and which developers can use to build apps and services that do things like recognize faces, identify emotions and distinguish various voices. Those tools also have provided the technical basis for Microsoft showcase apps and demonstration websites such as how-old.net, which guesses a person’s age, and Fetch, which can  identify a dog’s breed.

Microsoft Cognitive Services is an example of what is becoming a more common phenomenon – the lightning-fast transfer of the latest research advances into products that people can actually use. The engineers who work on Microsoft Cognitive Services say their job is a bit like solving a puzzle, and the pieces are the latest research.

“All these pieces come together and we need to figure out, how do we present those to an end user?” said Chris Buehler, a software engineering manager who works on Microsoft Cognitive Services.

From research project to helpful product
Seeing AI, the research project that could eventually help visually impaired people, is another example of how fast research can become a really helpful tool. It was conceived at last year’s //oneweek Hackathon, an event in which Microsoft employees from across the company work together to try to make a crazy idea become a reality.

The group that built Seeing AI included researchers and engineers from all over the world who were attracted to the project because of the technological challenges and, in many cases, also because they had a personal reason for wanting to help visually impaired people operate more independently.

“We basically had this super team of different people from different backgrounds, working to come up with what was needed,” said Anirudh Koul, who has been a lead on the Seeing AI project since its inception and became interested in it because his grandfather is losing his ability to see.

For Taylor, who joined Microsoft to represent the needs of blind people, it was a great experience that also resulted in a potential product that could make a real difference in people’s lives.

“We were able to come up with this one Swiss Army knife that is so valuable,” she said.

Related:

Learn more about Build 2016

Read more about the latest news from Build

Watch the audio description version of the Seeing AI video

Read about Cortana Intelligence Suite

Allison Linn is a senior writer at Microsoft. Follow her on Twitter.

The post Decades of computer vision research, one ‘Swiss Army knife’ appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/03/30/decades-of-computer-vision-research-one-swiss-army-knife/feed/ 0
Project Malmo: Using Minecraft to build more intelligent technology http://blogs.microsoft.com/next/2016/03/13/project-malmo-using-minecraft-build-intelligent-technology/ http://blogs.microsoft.com/next/2016/03/13/project-malmo-using-minecraft-build-intelligent-technology/#respond Mon, 14 Mar 2016 05:00:36 +0000 http://blogs.microsoft.com/next/?p=56433 Editor’s note, April 1, 2016: This project was formerly known as Project AIX and has now been renamed Project Malmo. In the airy, loft-like Microsoft Research lab in New York … Read more »

The post Project Malmo: Using Minecraft to build more intelligent technology appeared first on Next at Microsoft.

]]>
Editor’s note, April 1, 2016: This project was formerly known as Project AIX and has now been renamed Project Malmo.

In the airy, loft-like Microsoft Research lab in New York City, five computer scientists are spending their days trying to get a Minecraft character to climb a hill.

That may seem like a pretty simple job for some of the brightest minds in the field, until you consider this: The team is trying to train an artificial intelligence agent to learn how to do things like climb to the highest point in the virtual world, using the same types of resources a human has when she learns a new task.

That means that the agent starts out knowing nothing at all about its environment or even what it is supposed to accomplish. It needs to understand its surroundings and figure out what’s important – going uphill – and what isn’t, such as whether it’s light or dark. It needs to endure a lot of trial and error, including regularly falling into rivers and lava pits. And it needs to understand – via incremental rewards – when it has achieved all or part of its goal.

Project Malmo research

Fernando Diaz, Akshay Krishnamurthy and Alekh Agarwal are using Project Malmo for AI research. Photo by Scott Eklund/Red Box Pictures.

“We’re trying to program it to learn, as opposed to programming it to accomplish specific tasks,” said Fernando Diaz, a senior researcher in the New York lab and one of the people working on the project.

The research project is possible thanks to Project Malmo, a platform developed by Katja Hofmann and her colleagues in Microsoft’s Cambridge, UK, lab and unveiled publicly on Monday. Project Malmo allows computer scientists to use the world of Minecraft as a testing ground for conducting research designed to improve artificial intelligence.

Microsoft researchers are using Project Malmo for their own research, and they have made it available to a small group of academic researchers under a private beta. This summer, Project Malmo will be available via an open-source license.

Hofmann came up with the idea for Project Malmo about a year ago, in part because she was frustrated by the limitations of other platforms that use simpler, less sophisticated games for artificial intelligence research.

Minecraft is ideal for artificial intelligence research for the same reason it is addictively appealing to the millions of fans who enter its virtual world every day. Unlike other computer games, Minecraft offers its users endless possibilities, ranging from simple tasks like walking around looking for treasure to complex ones like building a structure with a group of teammates.

“Minecraft is the perfect platform for this kind of research because it’s this very open world,” Hofmann said. “You can do survival mode, you can do ‘build battles’ with your friends, you can do courses, you can implement our own games. This is really exciting for artificial intelligence because it allows us to create games that stretch beyond current abilities.”

From doing to learning
Over the past few years, artificial intelligence researchers have gotten very good at teaching computers to do specific, often complicated tasks. Computers can now understand speech and translate it. They can recognize images and write captions about them.

But despite all these advances, computers still aren’t very good at what researchers call general intelligence, which is more similar to the nuanced and complex way humans learn and make decisions. A computer algorithm may be able to take one task and do it as well or even better than an average adult, but it can’t compete with how an infant is taking in all sorts of inputs – light, smell, touch, sound, discomfort – and learning that if you cry chances are good that Mom will feed you.

“The things that seem really easy for us are actually the things that are really difficult for an artificial intelligence,” said Robert Schapire, a principal researcher at Microsoft Research who is part of the team using Project Malmo in Microsoft’s New York lab.

Hofmann said artificial intelligence researchers are able to take tiny slices of that total awareness and build tools that do one thing, like recognize words, but they haven’t been able to combine them in the way that humans do effortlessly. She said that’s partly because we don’t really know how humans are combining those senses.

“We don’t understand ourselves well enough,” she said.

Project Malmo demonstration

David Bignell, Tim Hutton, Katja Hofmann and Matthew Johnson are working on Project Malmo. Photography by Scott Eklund/Red Box Pictures.

From theory to practice
There has been plenty of theoretical research into general artificial intelligence, but researchers have always been limited by practical ways to test their systems.

Building a robot and trying to teach it to climb a real hill is costly and impractical; unlike in Minecraft, you’d have to repair or replace the robot with another costly machine each time it fell into a river.

It’s also not that easy to test general artificial intelligence research on systems people are using in the real world. Hofmann’s background is in how to make search more of an intelligent assistant than a simple information retrieval system, but she said one problem with testing her theories in real-world scenarios is that millions of people are depending on search engines to work in a predictable way.

“It’s hard to test some of this in practice, and that’s one of the main motivations for building the platform,” Hofmann said.

The Minecraft platform was especially appealing because it allows players to make really complex decisions that have consequences, and to add more and more difficult elements as they get better. It also lets users work together, which could help researchers experiment with how humans and artificially intelligent agents could work together.

“It’s a digital playpen for artificial intelligence,” Diaz said. “It’s an environment in which we can develop an algorithm for teaching a young artificial intelligence to learn different concepts in the world.”

Advancing all artificial intelligence research
From the beginning, Hofmann said the goal with Project Malmo was to build a system that would be useful both for Microsoft’s own research and for the broader artificial intelligence research community.

“We’re looking for opportunities where we can really help accelerate the pace of artificial intelligence innovation in a way that is going to be very close to the real world, with real experiences and real data,” said Evelyne Viegas, the director of artificial intelligence outreach at Microsoft Research.

The Project Malmo platform consists of a mod for the Java version and code that helps artificial intelligence agents sense and act within the Minecraft environment. The two components can run on Windows, Linux or Mac OS, and researchers can program their agents in any programming language they are comfortable with.

Matthew Johnson, the development lead on the project who also works in Microsoft’s UK lab, said the team developed the system with the hope that it would attract a broad range of academic researchers and motivated amateurs, with all levels of programming skills, background and goals. That said, the platform is intended for research into various forms of artificial intelligence and is not a consumer product.

“Our focus, from the beginning, has been on making sure that there’s the lowest possible barrier to innovation,” Johnson said.

Related:

Read more about the research group behind Project Malmo

Research paper: Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains

The future of artificial intelligence: Myths, realities and aspirations

Follow Katja Hofmann on Twitter

Allison Linn is a senior writer at Microsoft. Follow her on Twitter.

The post Project Malmo: Using Minecraft to build more intelligent technology appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/03/13/project-malmo-using-minecraft-build-intelligent-technology/feed/ 0
How Microsoft and Novartis created Assess MS http://blogs.microsoft.com/next/2016/02/08/how-microsoft-and-novartis-created-assess-ms/ http://blogs.microsoft.com/next/2016/02/08/how-microsoft-and-novartis-created-assess-ms/#respond Tue, 09 Feb 2016 05:07:58 +0000 http://blogs.microsoft.com/next/?p=56361 When Microsoft released the Kinect system for playing Xbox video games about five years ago, it attracted the interest of an unlikely source: the healthcare company Novartis. For years, Novartis … Read more »

The post How Microsoft and Novartis created Assess MS appeared first on Next at Microsoft.

]]>
When Microsoft released the Kinect system for playing Xbox video games about five years ago, it attracted the interest of an unlikely source: the healthcare company Novartis.

For years, Novartis has been trying to find more consistent ways to quantify whether the treatments it is developing for multiple sclerosis are working, but assessing whether a patient’s symptoms are stabilizing or getting worse is complicated.

The possibility of using computer vision, which is the type of technology found in the Kinect system, was intriguing. Using a tool like the Kinect, the researchers at Novartis figured they could get a more consistent reading of how a patient performed on a set of standardized tests for MS patients, bringing a new level of uniformity that would help doctors better assess the progress of the disease. That, in turn, could speed up the process of getting the right treatments to patients.

To find out more about the Assess MS research project that resulted from that idea, read the full story

The post How Microsoft and Novartis created Assess MS appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/02/08/how-microsoft-and-novartis-created-assess-ms/feed/ 0
Microsoft releases CNTK, its open source deep learning toolkit, on GitHub http://blogs.microsoft.com/next/2016/01/25/microsoft-releases-cntk-its-open-source-deep-learning-toolkit-on-github/ http://blogs.microsoft.com/next/2016/01/25/microsoft-releases-cntk-its-open-source-deep-learning-toolkit-on-github/#respond Mon, 25 Jan 2016 14:00:04 +0000 http://blogs.microsoft.com/next/?p=56181 Microsoft is making the tools that its own researchers use to speed up advances in artificial intelligence available to a broader group of developers by releasing its Computational Network Toolkit … Read more »

The post Microsoft releases CNTK, its open source deep learning toolkit, on GitHub appeared first on Next at Microsoft.

]]>
Microsoft is making the tools that its own researchers use to speed up advances in artificial intelligence available to a broader group of developers by releasing its Computational Network Toolkit on GitHub.

The researchers developed the open-source toolkit, dubbed CNTK, out of necessity. Xuedong Huang, Microsoft’s chief speech scientist, said he and his team were anxious to make faster improvements to how well computers can understand speech, and the tools they had to work with were slowing them down.

So, a group of volunteers set out to solve this problem on their own, using a homegrown solution that stressed performance over all else.

The effort paid off.

In internal tests, Huang said CNTK has proved more efficient  than four other popular computational toolkits that developers use to create deep learning models for things like speech and image recognition, because it has better communication capabilities

“The CNTK toolkit is just insanely more efficient than anything we have ever seen,” Huang said.

Those types of performance gains are incredibly important in the fast-moving field of deep learning, because some of the biggest deep learning tasks can take weeks to finish.

A comparison of toolkit speed ratesOver the past few years, the field of deep learning has exploded as more researchers have started running machine learning algorithms using deep neural networks, which are systems that are inspired by the biological processes of the human brain. Many researchers see deep learning as a very promising approach for making artificial intelligence better.

Those gains have allowed researchers to create systems that can accurately recognize and even translate conversations, as well as ones that can recognize images and even answer questions about them.

Internally, Microsoft is using CNTK on a set of powerful computers that use graphics processing units, or GPUs.

Although GPUs were designed for computer graphics, researchers have found that they also are ideal for processing the kind of algorithms that are leading to these major advances in technology that can speak, hear and understand speech, and recognize images and movements.

Chris Basoglu, a principal development manager at Microsoft who also worked on the toolkit, said one of the advantages of CNTK is that it can be used by anyone from a researcher on a limited budget, with a single computer, to someone who has the ability to create their own large cluster of GPU-based computers. The researchers say it can scale across more GPU-based machines than other publicly available toolkits, providing a key advantage for users who want to do large-scale experiments or calculations.

Xuedong Hong pictured in office

Xuedong Huang (Photography by Scott Eklund/Red Box Pictures)

Huang said it was important for his team to be able to address Microsoft’s internal needs with a tool like CNTK, but they also want to provide the same resources to other researchers who are making similar advances in deep learning.

That’s why they decided to make the tools available via open source licenses to other researchers and developers.

Last April, the researchers made the toolkit available to academic researchers, via Codeplex and under a more restricted open-source license.

But starting Monday it also will be available, via an open-source license, to anyone else who wants to use it. The researchers say it could be useful to anyone from deep learning startups to more established companies that are processing a lot of data in real time.

“With CNTK, they can actually join us to drive artificial intelligence breakthroughs,” Huang said.

Related:

Allison Linn is a senior writer at Microsoft Research. Follow her on Twitter.

The post Microsoft releases CNTK, its open source deep learning toolkit, on GitHub appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/01/25/microsoft-releases-cntk-its-open-source-deep-learning-toolkit-on-github/feed/ 0
Molecular biology meets computer science tools in new system for CRISPR http://blogs.microsoft.com/next/2016/01/18/molecular-biology-meets-computer-science-tools-in-new-system-for-crispr/ http://blogs.microsoft.com/next/2016/01/18/molecular-biology-meets-computer-science-tools-in-new-system-for-crispr/#respond Mon, 18 Jan 2016 16:47:46 +0000 http://blogs.microsoft.com/next/?p=56091 A team of researchers from Microsoft and the Broad Institute of MIT and Harvard has developed a new system that allows researchers to more quickly and effectively use the powerful … Read more »

The post Molecular biology meets computer science tools in new system for CRISPR appeared first on Next at Microsoft.

]]>
A team of researchers from Microsoft and the Broad Institute of MIT and Harvard has developed a new system that allows researchers to more quickly and effectively use the powerful gene editing tool CRISPR.

The system, unveiled Monday and dubbed Azimuth, uses machine learning, in which a computer takes a limited set of training data and uses that to learn how to make predictions about data it hasn’t yet seen.

In this case, the machine learning system is being used to predict which part of a gene to target when a scientist wants to knockout – or shut off – a gene. Machine learning enables the model to make predictions for any gene of interest, including those not seen in the experimental training data.

The two Microsoft researchers who led the computational modelling aspect of the project, Jennifer Listgarten and Nicolo Fusi, got excited about working on CRISPR after they happened to attend a lecture given by their future collaborator, John Doench, an associate director at the Broad Institute who led the biological portion of the project.

The partnership allowed the two sets of researchers, who are working on the bleeding edge of machine learning and gene editing, respectively, to collaborate on ways to advance the revolutionary new CRISPR technology.

Jennifer Listgarten photo

Jennifer Listgarten

“We couldn’t have done it without them and they couldn’t have done it without us,” Listgarten said.

Other computer scientists have tried to apply machine learning to CRISPR. Fusi said this project uses a more sophisticated machine learning model than previous efforts, and it also takes into account what worked and what didn’t with the previous models.

“Our goal was to not only understand why some features were important, but also to comprehensively evaluate all the other work that had been done before,” Fusi said.

The research team, which also includes collaborators from the Dana-Farber Cancer Institute and Washington University School of Medicine, published their findings this week in the journal Nature Biotechnology. In addition to the computational modelling, the team also released screening libraries that will help scientists more easily identify which of the hundreds, if not thousands, of places within a gene they should target with CRISPR to get the result they want.

CRISPR – it stands for clustered regularly interspaced short palindromic repeats – makes it much easier to precisely edit the DNA of living cells. Experts say CRISPR holds the promise of eventually allowing scientists to make major breakthroughs such as eradicating malaria.

“This is ultimately going to be a powerful approach, but there are many, many technical hurdles that stand in the way of having a direct impact on human health,” Doench said.

One of those big hurdles: Figuring out where exactly in a gene you want to use CRISPR to achieve the desired result. To do that manually requires hours in the lab, a generous research budget – and lots of trial and error.

“Very few people have the expertise or the resources or the time to do this kind of work,” Fusi said.

With the Azimuth machine learning tools, Doench said researchers will be able to streamline that process.

John Doench photo

John Doench

Listgarten and Fusi, who work out of Microsoft Research’s Cambridge, Massachusetts, lab, said scientists can use their models to figure out the best approach to take to shut down a gene.

The researchers are continuing their collaboration with a predictive analysis project that also will make it easier for researchers to figure out when and where the use of CRISPR to edit one gene will have unintended consequences elsewhere in the genome. Researchers call this an “off target” effect and it’s one of the biggest hurdles to using CRISPR for things like curing diseases in humans.

The ability to edit genes has long been one of the core goals of molecular biology, and other tools have been developed to perform that task. But CRISPR, which uses the mechanisms found in bacteria as the basis for its editing ability, is considered much more scalable and precise than previous efforts.

Scientists say CRISPR has the potential to help researchers understand when someone might become resistant to a cancer drug or an antibiotic, fix the mutation that causes sickle cell anemia and help with the quest to cure a rare form of blindness. It also could be used for non-medical purposes, such as to create crops that are more resilient in the face of climate change and other challenges.

“CRISPR is really revolutionizing many fields at once,” Listgarten said.

Related:

Read the full paper in the journal Nature Biotechnology

Read the Broad Institute’s blog post about the research

Learn more about Microsoft Research’s work in computational biology

Learn more about the Broad Institute of MIT and Harvard.

Follow Nicolo Fusi on Twitter

Allison Linn is a senior writer at Microsoft Research. Follow her on Twitter.

The post Molecular biology meets computer science tools in new system for CRISPR appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/01/18/molecular-biology-meets-computer-science-tools-in-new-system-for-crispr/feed/ 0
Microsoft researchers win ImageNet computer vision challenge http://blogs.microsoft.com/next/2015/12/10/microsoft-researchers-win-imagenet-computer-vision-challenge/ http://blogs.microsoft.com/next/2015/12/10/microsoft-researchers-win-imagenet-computer-vision-challenge/#respond Thu, 10 Dec 2015 15:09:31 +0000 http://blogs.microsoft.com/next/?p=55911 Microsoft researchers on Thursday announced a major advance in technology designed to identify the objects in a photograph or video, showcasing a system whose accuracy meets and sometimes exceeds human-level … Read more »

The post Microsoft researchers win ImageNet computer vision challenge appeared first on Next at Microsoft.

]]>
Microsoft researchers on Thursday announced a major advance in technology designed to identify the objects in a photograph or video, showcasing a system whose accuracy meets and sometimes exceeds human-level performance.

Microsoft’s new approach to recognizing images also took first place in several major categories of image recognition challenges Thursday, beating out many other competitors from academic, corporate and research institutions in the ImageNet and Microsoft Common Objects in Context challenges.

Like many other researchers in this field, Microsoft relied on a method called deep neural networks to train computers to recognize the images. Their system was more effective because it allowed them to use extremely deep neural nets, which are as much as five times deeper than any previously used.

The researchers say even they weren’t sure this new approach was going to be successful – until it was.

Kaiming He

Kaiming He

“We even didn’t believe this single idea could be so significant,” said Jian Sun, a principal research manager at Microsoft Research who led the image understanding project along with teammates Kaiming He, Xiangyu Zhang and Shaoqing Ren in Microsoft’s Beijing research lab.

The major leap in accuracy surprised others as well. Peter Lee, a corporate vice president in charge of Microsoft Research’s NExT labs, said he was shocked to see such a major breakthrough.

“It sort of destroys some of the assumptions I had been making about how the deep neural networks work,” he said.

The contests, organized by researchers from top universities and corporations, have in the past few years become a leading barometer of success in this exploding field.

In the ImageNet challenge, the Microsoft team won first place in all three categories it entered: classification, localization and detection. Its system was better than the other entrants by a large margin.

In the Microsoft Common Objects in Context challenge, also known as MS COCO, the Microsoft team won first place for image detection and segmentation . The MS COCO project was originally funded by Microsoft and started as a collaboration between Microsoft and a few universities, but it is now run by academics outside of Microsoft.

A long research path and a recent breakthrough

Computer scientists have for decades been trying to train computer systems to do things like recognize images and comprehend speech, but until recently those systems were plagued with inaccuracies.

Then, about five years ago, researchers hit upon the idea of using a technology called neural networks, which are inspired by the biological processes of the brain. The neural networks themselves weren’t new, but the method of using them was – and it resulted in big leaps in accuracy in image recognition.

The system also proved very successful for recognizing speech, and it’s been the basis for the real-time translation capability in Skype Translator.

Neural networks are built in a series of layers. Theoretically, more layers should lead to better results, but in practice one big challenge has been that the signals vanish as they pass through each layer, eventually leading to difficulties in training the whole system.

Sun said researchers were excited when they could successfully train a “deep neural network” system with eight layers three years ago, and thrilled when a “very deep neural network” with 20 to 30 layers delivered results last year.

But he and his team thought they could go even deeper. For months, they toyed with various ways to add more layers and still get accurate results.

After a lot of trial and error, the researchers hit on a system they dubbed “deep residual networks.”

The deep residual net system they used for the ImageNet contest has 152 layers – fives time more than any past system – and it uses a new “residual learning” principle to guide the network architecture designs.

Residual learning reformulates the learning procedure and redirects the information flow in deep neural networks. That helped the researchers solve the accuracy problem that has traditionally dogged attempts to build extremely deep neural networks.

Transfer of knowledge

One key advantage of neural networks is that they get better at one task when they are given another. For example, with Skype Translator, a neural network that is designed to translate from English to German gets better at translating German once it has been trained for the additional task of translating Chinese.

Sun said his team saw similar results when they tested their residual neural networks in advance of the two competitions. After researchers used the system for the classification tasks in the ImageNet challenge, they found that it was significantly better at the three other metrics: detection, localization and segmentation.

“What we learned from our extremely deep networks is so powerful and generic that it can substantially improve many other vision tasks,” Sun said.

The researchers believe they would see a similar effect if they used the same principle for other problems, such as speech recognition.

They are already using these new advances to help improve the tools in Microsoft Project Oxford, which help developers build more intelligent apps for things like speech and image recognition. They also are working tightly with Microsoft’s product groups to include the best image understanding in existing or future Microsoft products and services.

None of this means that computers are getting smarter than humans, in a general way. The researchers say what it shows is that computers are getting very good at very narrow tasks, like identifying images in a database.

Still, that has big implications for how computers could eventually help people in any number of ways, like recognizing the difference between a tree and a car in your side view mirror or the frustrating task of sorting through photos for specific things, like a great picture of your dog.

“We don’t believe we’re anywhere close to the limit of the ultimate improvement in data classification accuracy for any of these tasks,” Lee said.

Related:

Research paper : Deep residual learning for image recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun

Microsoft Computational Network Toolkit offers most efficient distributed deep learning computational performance

The quest to create technology that understands speech as well as a human

System trains machines to look at images the way people do — and answer questions about them

Picture this: Microsoft research project can interpret, caption photos

Microsoft researchers’ algorithm sets ImageNet challenge milestone

Follow Peter Lee on Twitter

Allison Linn is a senior writer at Microsoft Research. Follow her on Twitter.

The post Microsoft researchers win ImageNet computer vision challenge appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2015/12/10/microsoft-researchers-win-imagenet-computer-vision-challenge/feed/ 0
The quest to create technology that understands speech as well as a human http://blogs.microsoft.com/next/2015/12/03/the-quest-to-create-technology-that-understands-speech-as-well-as-a-human/ http://blogs.microsoft.com/next/2015/12/03/the-quest-to-create-technology-that-understands-speech-as-well-as-a-human/#respond Thu, 03 Dec 2015 14:06:21 +0000 http://blogs.microsoft.com/next/?p=55881 Not too long ago, the idea of creating tools that could understand speech as well as a human seemed more like the stuff of science fiction than computer science. These … Read more »

The post The quest to create technology that understands speech as well as a human appeared first on Next at Microsoft.

]]>
Not too long ago, the idea of creating tools that could understand speech as well as a human seemed more like the stuff of science fiction than computer science.

These days, a game console that understands voice commands, apps that can translate your conversation in real time and a virtual assistant that provides you with the numbers of nearby pizza places are all fact, not fiction.

These systems not only exist but are getting better every day, thanks to improvements in data availability, computing power and a subfield of artificial intelligence called machine learning, in which systems improve as they take in more data.

Within just a few years, some researchers now believe, these technologies could reach a point where computers can understand the words people are saying about as well as a person would.

“We are reaching that inflection point,” said Xuedong Huang, the company’s chief speech scientist.

To find out more, read the full story .

The post The quest to create technology that understands speech as well as a human appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2015/12/03/the-quest-to-create-technology-that-understands-speech-as-well-as-a-human/feed/ 0