Next at Microsoft http://blogs.microsoft.com/next Mon, 18 Jul 2016 16:11:16 +0000 en-US hourly 1 https://wordpress.org/?v=4.5.3 Project Malmo, which lets researchers use Minecraft for AI research, makes public debut http://blogs.microsoft.com/next/2016/07/07/project-malmo-lets-researchers-use-minecraft-ai-research-makes-public-debut/ http://blogs.microsoft.com/next/2016/07/07/project-malmo-lets-researchers-use-minecraft-ai-research-makes-public-debut/#respond Fri, 08 Jul 2016 04:00:05 +0000 http://blogs.microsoft.com/next/?p=57190 Microsoft has made Project Malmo, a platform that uses the world of Minecraft as a testing ground for advanced artificial intelligence research, available for novice to experienced programmers on GitHub … Read more »

The post Project Malmo, which lets researchers use Minecraft for AI research, makes public debut appeared first on Next at Microsoft.

]]>
Microsoft has made Project Malmo, a platform that uses the world of Minecraft as a testing ground for advanced artificial intelligence research, available for novice to experienced programmers on GitHub via an open-source license.

The system, which had until now only been open to a small group of computer scientists in private preview, is primarily designed to help researchers develop sophisticated, more general artificial intelligence, or AI, that can do things like learn, hold conversations, make decisions and complete complex tasks.

That’s key to creating systems that can augment human intelligence — and eventually help us with everything from cooking and doing laundry to driving and performing lifesaving tasks in an operating room.

Katja Hofmann, a researcher in Microsoft’s Cambridge, UK, research lab, who leads the development of Project Malmo, said the system will help researchers develop new techniques and approaches to reinforcement learning. That’s an area of AI in which agents learn how to complete a task by being given a lot of room for trial and error and then being rewarded when they make the right decision.

“We’re trying to put out the tools that will allow people to make progress on those really, really hard research questions,” Hofmann said.

For example, computer scientists have gotten exceptionally good at creating tools that can understand the words we say, whether we’re asking a gadget for directions or navigating an automated customer service line.

But when it comes to actually comprehending the meaning of those audio waves – well, in most cases a baby could do better.

“We’ve trained the artificial intelligence to identify patterns in the dictation, but the underlying technology doesn’t have any understanding of what those words mean,” Hofmann said. “They’re just statistical patterns, and there’s no connection to any experience.”

Microsoft researchers working on Project Malmo include, from top left, Fernando Diaz, Evelyne Viegas, David Bignell, Alekh Agarwal, Matthew Johnson, Akshay Krishnamurthy, Katja Hofmann and Tim Hutton. Photography by Scott Eklund/Red Box Pictures

Microsoft researchers working on Project Malmo include, from top left, Fernando Diaz, Evelyne Viegas, David Bignell, Alekh Agarwal, Matthew Johnson, Akshay Krishnamurthy, Katja Hofmann and Tim Hutton. (Photography by Scott Eklund/Red Box Pictures)

Beyond understanding to comprehension
Teaching AI agents to comprehend humans in the same way we comprehend each other is one of the core goals of advanced artificial intelligence research. With Project Malmo’s public launch, the team has added functionality that will let computer scientists create bots that can learn to talk to each other, and to people.

Project Malmo also can be used to teach AI to do crafting – using tools and resources to build things like a table or a sword – and to learn how to get around on their own without falling down a hill or into a lava pit. They also can learn to build with blocks, navigate mazes and do any number of other tasks that mimic the types of things we might want AI to one day do in real life.

The researchers who have been part of Project Malmo’s private preview say Minecraft, with its rich, immersive world and endless possibilities for collaboration and exploration, is ideally suited for general AI research.

“Minecraft is very close to the real world in many ways,” said Jose Hernandez-Orallo, a professor at the Technical University of Valencia, Spain, who has been part of the private preview. “There are so many possibilities.”

Doing this kind of research requires a lot of trial and error, with small and incremental victories along the way. That’s why, when Project Malmo launches publicly, it also will have another new feature: Overclocking, or the ability to run experiments faster than the usual pace of Minecraft’s world.

Evelyne Viegas, director of AI outreach at Microsoft Research, said that will allow researchers to get results, and make adjustments, more quickly.

“It’s accelerating the pace of those experiments,” she said.

A standard for measuring progress
The AI researchers who have gotten a sneak peek at Project Malmo say another key advantage to the system is that it will let researchers compare their progress against the work of others, by seeing how well their theories perform in the same environment.

Hernandez-Orallo said AI researchers are often developing their own systems for testing their theories and algorithms. That allows them to solve isolated problems, but it can be tough to know how those results compare to, or would complement, the work of others.

With a system like Project Malmo, he said researchers can test their systems in the same Minecraft setting. The ability to use the same testing ground “is music to my ears,” said Hernandez-Orallo, who has a particular interest in AI evaluation and is spending the summer at Microsoft’s UK lab so he can work directly with the Project Malmo researchers.

The open-source environment also allows researchers to much more easily collaborate, sharing research insights and bringing their findings together.

“There’s no question that it vastly speeds up the research process,” said Matthew Johnson, the development lead on Project Malmo, who also works in Microsoft’s Cambridge, UK, lab.

All coders welcome
Hofmann and her team created Project Malmo to help seasoned AI researchers conduct their research. But they’ve been pleasantly surprised to find that everyone from tweens with an early passion for programming to professors trying to train the next generation of AI researchers want to work with it as well.

Viegas said more novice coders can experience the system.

“You need to know how to program, but you don’t need to be an advanced programmer,” she said.

The Project Malmo platform consists of a mod for the Java version and code that helps AI agents sense and act within the Minecraft environment. The two components can run on Windows, Linux or Mac OS, and programmers can use most popular programming languages.

The team also has heard from several professors who want to incorporate Project Malmo into their lesson plans.

That makes sense. Hernandez-Orallo said his students – who may well spend their free time playing Minecraft – are going to be a lot more excited by an assignment using Project Malmo than by one that asks them to work with a more generic algorithm pulled from a research paper.

“This is going to have an impact in education, at least at the university level,” he said.

Johnson said they are already seeing people produce academic research based on Project Malmo, and that’s the core reason for doing a project like this. But he concedes that it’s also fun to imagine that a more mainstream audience might want to check it out.

“If I come across some YouTube video showing off some exciting new functionality enabled by our mod, that would make my day,” he said.

Related:

Allison Linn is a senior writer at Microsoft. Follow her on Twitter.

The post Project Malmo, which lets researchers use Minecraft for AI research, makes public debut appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/07/07/project-malmo-lets-researchers-use-minecraft-ai-research-makes-public-debut/feed/ 0
Microsoft and University of Washington researchers set record for DNA storage http://blogs.microsoft.com/next/2016/07/07/microsoft-university-washington-researchers-set-record-dna-storage/ http://blogs.microsoft.com/next/2016/07/07/microsoft-university-washington-researchers-set-record-dna-storage/#respond Thu, 07 Jul 2016 12:30:08 +0000 http://blogs.microsoft.com/next/?p=57202 Researchers at Microsoft and the University of Washington have reached an early but important milestone in DNA storage by storing a record 200 megabytes of data on the molecular strands. … Read more »

The post Microsoft and University of Washington researchers set record for DNA storage appeared first on Next at Microsoft.

]]>
Researchers at Microsoft and the University of Washington have reached an early but important milestone in DNA storage by storing a record 200 megabytes of data on the molecular strands.

The impressive part is not just how much data they were able to encode onto synthetic DNA and then decode. It’s also the space they were able to store it in.

Once encoded, the data occupied a spot in a test tube “much smaller than the tip of a pencil,” said Douglas Carmean, the partner architect at Microsoft overseeing the project.

Think of the amount of data in a big data center compressed into a few sugar cubes. Or all the publicly accessible data on the Internet slipped into a shoebox. That is the promise of DNA storage – once scientists are able to scale the technology and overcome a series of technical hurdles.

Test tube holding data next to pencil

Digital data from more than 600 basic smartphones can be stored in the faint pink smear of DNA at the end of this test tube. Photo by Tara Brown Photography/University of Washington.

The Microsoft-UW team stored digital versions of works of art (including a high-definition video by the band OK Go!), the Universal Declaration of Human Rights in more than 100 languages, the top 100 books of Project Guttenberg and the nonprofit Crop Trust’s seed database on DNA strands.

Demand for data storage is growing exponentially, and the capacity of existing storage media is not keeping pace.  That’s making it hard for organizations that need to store a lot of data – such as hospitals with vast databases of patient data or companies with lots of video footage – to keep up. And it means information is being lost, and the problem will only worsen without a new solution.

DNA could be the answer.

It has several advantages as a storage medium. It’s compact, durable – capable of lasting for a very long time if kept in good conditions (DNA from woolly mammoths was recovered several thousand years after they went extinct, for instance) – and will always be current, the researchers believe.

“As long as there is DNA-based life on the planet, we’ll be interested in reading it,” said Karin Strauss, the principal Microsoft researcher on the project. “So it’s eternally relevant.”

This explains why the Microsoft-UW team is just one of a number of research groups around the globe pursuing the potential of DNA as a vast digital attic.

The researchers acknowledge they have a long way to go.

Luis Henrique Ceze, a UW associate professor of computer science and engineering and the university’s principal researcher on the project, said the biotechnology industry made big advances in both “synthesizing” (encoding) and “sequencing” (decoding) data in recent years. Even so, he said, the team still has a long way to go to make it viable as an archival technology.

But the researchers are upbeat.

They note that their diverse team of computer scientists, computer architects and molecular biologists already has increased storage capacity a thousand times in the last year. And they believe they can make big advances in speed by applying computer science principles like error correction to the process.

Carmean, who was involved in development of Intel’s microprocessor architecture beginning in 1989, puts it this way:

“It’s one of those serendipitous partnerships where a strong understanding of processors and computation married with molecular biology experts has the potential of producing major breakthroughs.”

To get an idea of how the Microsoft-UW team does its work, flash back to high school biology and recall that DNA – or deoxyribonucleic acid – is a molecule that contains the biological instructions used in the growth, development, functioning and reproduction of all known living organisms.

“DNA is an amazing information storage molecule that encodes data about how a living system works. We’re repurposing that capacity to store digital data — pictures, videos, documents,” said Ceze, who is conducting research in the team’s Molecular Information Systems Lab (MISL), which is housed in a basement on the University of Washington campus. “This is one important example of the potential of borrowing from nature to build better computer systems.”

Storing digital data on DNA works like this:

First the data is translated from 1s and 0s into the “letters” of the four nucleotide bases of a DNA strand — (A)denine, (C)ytosine, (G)uanine and (T)hymine.

Karin Strauss

Karin Strauss. Photo by Scott Eklund/Red Box Pictures

Then they have vendor Twist Bioscience “translate those letters, which are still in electronic form, into the molecules themselves, and send them back,” Strauss said. “It’s essentially a test tube and you can barely see what’s in it. It looks like a little bit of salt was dried in the bottom.”

Reading the data uses a biotech tweak to random access memory (RAM), another concept borrowed from computer science. The team uses polymerase chain reaction (PCR), a technique that molecular biologists use routinely to manipulate DNA, to multiply or “amplify” the strands it wants to recover. Once they’ve sharply increased the concentration of the desired snippets, they take a sample, sequence or decode the DNA and then run error correction computations.

The lab tour complete, one question needed asking: Why an OK Go video?

“We like that a lot because there are many parallels with the work,” Strauss said with a laugh. “They’re very innovative and are bringing different things from different areas into their field and we feel we are doing something very similar.”

Related:

Learn more about Microsoft’s DNA storage project

Read the University of Washington story and Q&A on the project

Read the Twist Bioscience press release

Follow Karin Strauss on Twitter

New York Times: Data storage on DNA can keep it safe for centuries

Mike Brunker is a freelance writer and editor. Follow him on Twitter.

 

The post Microsoft and University of Washington researchers set record for DNA storage appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/07/07/microsoft-university-washington-researchers-set-record-dna-storage/feed/ 0
Talking with your hands: How Microsoft researchers are moving beyond keyboard and mouse http://blogs.microsoft.com/next/2016/06/26/talking-hands-microsoft-researchers-moving-beyond-keyboard-mouse/ http://blogs.microsoft.com/next/2016/06/26/talking-hands-microsoft-researchers-moving-beyond-keyboard-mouse/#respond Mon, 27 Jun 2016 04:00:09 +0000 http://blogs.microsoft.com/next/?p=57052 Kfir Karmon imagines a world in which a person putting together a presentation can add a quote or move an image with a flick of the wrist instead of a … Read more »

The post Talking with your hands: How Microsoft researchers are moving beyond keyboard and mouse appeared first on Next at Microsoft.

]]>
Kfir Karmon imagines a world in which a person putting together a presentation can add a quote or move an image with a flick of the wrist instead of a click of a mouse.

Jamie Shotton envisions a future in which we can easily interact in virtual reality much like we do in actual reality, using our hands for small, sophisticated movements like picking up a tool, pushing a button or squeezing a soft object in front of us.

And Hrvoje Benko sees a way in which those types of advances could be combined with simple physical objects, such as a few buttons on a piece of wood, to recreate complex, immersive simulators – replacing expensive hardware that people use today for those purposes.

Microsoft researchers are looking at a number of ways in which technology can start to recognize detailed hand motion — and engineers can put those breakthroughs to use in a wide variety of fields.

The ultimate goal: Allowing us to interact with technology in more natural ways than ever before.

“How do we interact with things in the real world? Well, we pick them up, we touch them with our fingers, we manipulate them,” said Shotton, a principal researcher in computer vision at Microsoft’s Cambridge, UK, research lab. “We should be able to do exactly the same thing with virtual objects. We should be able to reach out and touch them.”

This kind of technology is still evolving. But the computer scientists and engineers who are working on these projects say they believe they are on the cusp of making hand and gesture recognition tools practical enough for mainstream use, much like many people now use speech recognition to dictate texts or computer vision to recognize faces in photos.

That’s a key step in Microsoft’s broader goal to provide more personal computing experiences by creating technology that can adapt to how people move, speak and see, rather than asking people to adapt to how computers work.

“If we can make vision work reliably, speech work reliably and gesture work reliably, then people designing things like TVs, coffee machines or any of the Internet of Things gadgets will have a range of interaction possibilities,” said Andrew Fitzgibbon, a principal researcher with the computer vision group at the UK lab.

That will be especially important as computing becomes more ubiquitous and increasingly anticipates our needs, as opposed to responding to our commands. To make these kinds of ambient computing systems truly work well, experts say, they must be able to combine all our senses, allowing us to easily communicate with gadgets using speech, vision and body language together – just like we do when communicating with each other.

The team working on hand track in Microsoft's UK lab includes Tom Cashman (top left, standing), Andrew Fitzgibbon, Lucas Bordeaux, John Bronskill, (bottom row) David Sweeney, Jamie Shotton, Federica Bogo. Photo by Jonathan Banks.

The team working on hand tracking in Microsoft’s UK lab includes Tom Cashman (top left, standing), Andrew Fitzgibbon, Lucas Bordeaux, John Bronskill, (bottom row) David Sweeney, Jamie Shotton, Federica Bogo. Photo by Jonathan Banks.

Smooth, accurate and easy

In order to accomplish a component of that vision, Fitzgibbon and other researchers believe the technology must track hand motion precisely and accurately, using as little computing power as possible. That will allow people to use their hands naturally and with ease, and for consumer gadgets to respond accordingly.

It’s easier said than done, in large part because the hand itself is so complex. Hands can rotate completely around, and they can do things like ball up into a fist, which means the fingers disappear and the tool needs to make its best guess as to where they’ve gone and what they are doing. Also, a hand is obviously smaller than an entire body, so there’s more detailed motion to track.

The computer vision team’s latest advances in detailed hand tracking, which are being unveiled at two prestigious academic research conferences this summer, combine new breakthroughs in methods for tracking hand movement with an algorithm dating back to the 1940s – when computing power was less available and a lot more expensive.  Together, they create a system that can track hands smoothly, quickly and accurately – in real time – but can run on a regular consumer gadget.

“We’re getting to the point that the accuracy is such that the user can start to feel like the avatar hand is their real hand,” Shotton said.

The system, still a research project for now, can track detailed hand motion with a virtual reality headset or without it, allowing the user to poke a soft, stuffed bunny, turn a knob or move a dial.

What’s more, the system lets you see what your hands are doing, fixing a common and befuddling disconnect that happens when people are interacting with virtual reality but can’t see their own hands.

From dolphins to detailed hand motion

The project, called Handpose, relies on a wealth of basic computer vision research. For example, a research project that Fitzgibbon and his colleague Tom Cashman worked on years earlier, looking at how to make 2D images of dolphins into 3D virtual objects, proved useful in developing the Handpose technology.

The researchers say that’s an example of how a long-term commitment to this kind of research can pay off in unexpected ways.

Although hand movement recognition isn’t being used broadly by consumers yet, Shotton said that he thinks the technology is now getting good enough that people will start to integrate it into mainstream experiences.

“This has been a research topic for many, many years, but I think now is the time where we’re going to see real, usable, deployable solutions for this,” Shotton said.

A virtual sense of touch

The researchers behind Handpose say they have been surprised to find that a lack of haptics – or the sense of actually touching something – isn’t as big of a barrier as they thought when people test systems like theirs, which let people manipulate virtual objects with their hands.

That’s partly because of how they are designing the virtual world. For example, the researchers created virtual controls that are thin enough that you can touch your fingers together to get an experience of touching something hard. They also developed sensory experiences that allow people to push against something soft and pliant rather than hard and unforgiving, which appears to feel more authentic.

The researchers say they also notice that other senses, such as sight and sound, can convince people they are touching something real when they are not – especially once the systems are good enough to work in real time.

Andy Wilson, left, and Hrvoje Benko are among the researchers working on haptic retargeting.

Andy Wilson, left, and Hrvoje Benko are among the researchers working on haptic retargeting. Photo by Jeremy Mashburn.

Still, Benko, a senior researcher in the natural interaction group at Microsoft’s Redmond, Washington, lab, noted that as virtual reality gets more sophisticated, it may become harder to trick the body into immersing itself in the experience without having anything at all to touch.

Benko said he and his lab colleagues have been working on ways to use limited real-world objects to make immersive virtual reality experiences seem more like what humans expect from the real world.

“There’s some value in haptics and so we’re trying to understand what that is,” said Andy Wilson, a principal researcher who directs Microsoft Research’s natural interaction group.

But that doesn’t mean the entire virtual world needs to be recreated. Eyal Ofek, a senior researcher in the natural interaction group, said people can be fooled into believing things about a virtual world if that world is presented with enough cues to mimic reality.

For example, let’s say you want to build a structure using toy blocks in a virtual environment. Using the haptic retargeting research project the Microsoft team created, one building block could be used over and over again, with the virtual environment shifting to give the impression you are stacking those blocks higher and higher even as, in reality, you are placing the same one on the same plane.

The same logic could be applied to a more complex simulator, using just a couple of simple knobs and buttons to recreate a complex system for practicing landing an airplane or other complex maneuvers.

“A single physical object can now simulate multiple instances in the virtual world,” Ofek said.

The language of gesture

Let’s say you’re talking to a colleague over Skype and you’re ready to end the call. What if, instead of using your mouse or keyboard to click a button, you could simply make the movement of hanging up the phone?

Need to lock your computer screen quickly? What if, instead of scrambling to close windows and hit keyboard shortcuts, you simply reach out and mimic the gesture of turning a key in a lock?

Researchers and engineers in Microsoft’s Advanced Technologies Lab in Israel are investigating ways in which developers could create tools that would allow people to communicate with their computer utilizing the same kind of hand gestures they use in everyday life.

The goal of the research project, called Project Prague, would be to provide developers with basic hand gestures, such as the one that switches a computer off. And it also makes it easy for developers to create customized gestures for their own apps or other products, with very little additional programming or expertise.

The system, which utilizes machine learning to train systems to recognize motions, runs using a retail 3D camera.

“It’s a super easy experience for the developers and for the end user,” said Karmon, a principal engineering manager who is the project’s lead.

To build the system, the researchers recorded millions of hand images and then used that data set to train the technology to recognize every possible hand pose and motion.

Eyal Krupka, a principal applied researcher and head of the lab’s computer vision and machine learning research, said the technology then uses hundreds of micro artificial intelligence units, each analyzing a single aspect of the user’s hand, to accurately interpret each gesture.

The end result is a system that doesn’t just recognize a person’s hand, but also understands that person’s intent.

Adi Diamant, who directs the Advanced Technologies Lab, said that when people think about hand and gesture recognition, they often think about ways it can be used for gaming or entertainment. But he also sees great potential for using gesture for everyday work tasks, like designing and giving presentations, flipping through spreadsheets, editing e-mails and browsing the web.

People also could use them for more creative tasks, like creating art or making music.

Diamant said these types of experiences are only possible because of advances in fields including machine learning and computer vision, which have allowed his team to create a system that gives people a more natural way of interacting with technology.

“We chose a project that we knew was a tough challenge because we knew there was a huge demand for hand gesture,” he said.

Related:

Allison Linn is a senior writer at Microsoft. Follow her on Twitter.

 

The post Talking with your hands: How Microsoft researchers are moving beyond keyboard and mouse appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/06/26/talking-hands-microsoft-researchers-moving-beyond-keyboard-mouse/feed/ 0
How web search data might help diagnose serious illness earlier http://blogs.microsoft.com/next/2016/06/07/how-web-search-data-might-help-diagnose-serious-illness-earlier/ http://blogs.microsoft.com/next/2016/06/07/how-web-search-data-might-help-diagnose-serious-illness-earlier/#respond Tue, 07 Jun 2016 20:27:57 +0000 http://blogs.microsoft.com/next/?p=56890 Early diagnosis is key to gaining the upper hand against a wide range of diseases. Now Microsoft researchers are suggesting that records of the topics that people search for on … Read more »

The post How web search data might help diagnose serious illness earlier appeared first on Next at Microsoft.

]]>
Early diagnosis is key to gaining the upper hand against a wide range of diseases. Now Microsoft researchers are suggesting that records of the topics that people search for on the Internet could one day prove as useful as an X-ray or MRI in detecting some illnesses before it’s too late.

The potential of using engagement with search engines to predict an eventual diagnosis – and possibly buy critical time for a medical response — is demonstrated in a new study by Microsoft researchers Eric Horvitz and Ryen White, along with former Microsoft intern and Columbia University doctoral candidate John Paparrizos.

In a paper published Tuesday in the Journal of Oncology Practice, the trio detailed how they used anonymized Bing search logs to identify people whose queries provided strong evidence that they had recently been diagnosed with pancreatic cancer – a particularly deadly and fast-spreading cancer that is frequently caught too late to cure. Then they retroactively analyzed searches for symptoms of the disease over many months prior to identify patterns of queries most likely to signal an eventual diagnosis.

“We find that signals about patterns of queries in search logs can predict the future appearance of queries that are highly suggestive of a diagnosis of pancreatic adenocarcinoma,” – the medical term for pancreatic cancer, the authors wrote. “We show specifically that we can identify 5 to 15 percent of cases while preserving extremely low false positive rates” of as low as 1 in 100,000.

The researchers used large-scale anonymized data and complied with best practices in ethics and privacy for the study.

Eric Horvitz

Eric Horvitz, a technical fellow and managing director of Microsoft’s Redmond, Washington, research lab (Photography by Scott Eklund/Red Box Pictures)

Horvitz, a technical fellow and managing director of Microsoft’s research lab in Redmond, Washington, said the method shows the feasibility of a new form of screening that could ultimately allow patients and their physicans to diagnose pancreatic cancer and begin treatment weeks or months earlier than they otherwise would have. That’s an important advantage in fighting a disease with a very low survival rate if it isn’t caught early.

Pancreatic cancer — the fourth leading cause of cancer death in the United States – was in many ways the ideal subject for the study because it typically produces a series of subtle symptoms, like itchy skin, weight loss, light-colored stools, patterns of back pain and a slight yellowing of the eyes and skin that often don’t prompt a patient to seek medical attention.

Horvitz, an artificial intelligence expert who holds both a Ph.D. and an MD from Stanford University, said the researchers found that queries entered to seek answers about that set of symptoms can serve as an early warning for the onset of illness.

But Horvitz said that he and White, chief technology officer for Microsoft Health and an information retrieval expert, believe that analysis of search queries could have broad applications.

“We are excited about applying this analytical pipeline to other devastating and hard-to-detect diseases,” Horvitz said.

Horvitz and White emphasize that the research was done as a proof of concept that such a “different kind of sensor network or monitoring system” is possible. The researchers said Microsoft has no plans to develop any products linked to the discovery.

Instead, the authors said, they hope the positive results from the feasibility study will excite the broader medical community and generate discussion about how such a screening methodology might be used.  They suggest that it would likely involve analyzing anonymized data and having a method for people who opt in to receive some sort of notification about health risks, either directly or through their doctors, in the event algorithms detected a pattern of search queries that could signal a health concern.

But White said the search analysis would not be a medical opinion.

“The goal is not to perform the diagnosis,” he said. “The goal is to help those at highest risk to engage with medical professionals who can actually make the true diagnosis.”

White and Horvitz said they wanted to take the results of the pancreatic cancer study directly to those in a position to do something with the results, which is why they chose to first publish in a medical journal.

“I guess I’m at a point now in my career where I’m not interested in the potential for impact,” White said of the decision. “I actually want to have impact. I would like to see the medical community pick this up and take it as a technology, and work with us to enable this type of screening.”

And Horvitz, who said he lost his best childhood friend and, soon after, a close colleague in computer science to pancreatic cancer, said the stakes are too high to delay getting the word out.

“People are being diagnosed too late,” he said. “We believe that these results frame a new approach to pre-screening or screening, but there’s work to do to go from the feasibility study to real-world fielding.”

Horvitz and White have previously teamed up on other search-related medical studies – notably a 2008 analysis of “cyberchondria” – or “medical anxiety that is stimulated by symptom searches on the web,” as Horvitz puts it – and analyses of search logs that identify adverse effects of medications.

Related:

Decades of computer vision research, one ‘Swiss Army knife’

From gaming system to medical breakthrough

Eric Horvitz receives AAAI-Allen Newell Award

Follow Eric Horvitz on Twitter

Article on data, privacy, and the greater good

Mike Brunker is a freelance writer and editor. Follow him on Twitter.

The post How web search data might help diagnose serious illness earlier appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/06/07/how-web-search-data-might-help-diagnose-serious-illness-earlier/feed/ 0
Eric Horvitz receives ACM-AAAI Allen Newell Award for groundbreaking artificial intelligence work http://blogs.microsoft.com/next/2016/04/27/eric-horvitz-receives-acm-aaai-allen-newell-award-groundbreaking-artificial-intelligence-work/ http://blogs.microsoft.com/next/2016/04/27/eric-horvitz-receives-acm-aaai-allen-newell-award-groundbreaking-artificial-intelligence-work/#respond Wed, 27 Apr 2016 13:00:12 +0000 http://blogs.microsoft.com/next/?p=56836 In his many years as an artificial intelligence researcher, Eric Horvitz has worked on everything from systems that help determine what’s funny or surprising to those that know when to … Read more »

The post Eric Horvitz receives ACM-AAAI Allen Newell Award for groundbreaking artificial intelligence work appeared first on Next at Microsoft.

]]>
In his many years as an artificial intelligence researcher, Eric Horvitz has worked on everything from systems that help determine what’s funny or surprising to those that know when to help us remember what we need to do at work.

On Wednesday, Horvitz, a technical fellow and managing director of Microsoft’s Redmond, Washington, research lab, received the ACM – AAAI Allen Newell Award for groundbreaking contributions in artificial intelligence and human-computer interaction. The award honors Horvitz’s substantial theoretical efforts and as well as his persistent focus on using those discoveries as the basis for practical applications that make our lives easier and more productive.

Harry Shum, the executive vice president of Microsoft’s technology and research group, said Horvitz epitomizes a style of research that is unique to places like Microsoft because it is focused on having an impact in both the research and industry domains.

“People talk about basic research and applied research. What we are doing here is Microsoft research,” Shum said. “It’s not just about doing theoretical research and writing more papers. It’s also about applying those technologies in Microsoft products.”

Jeannette M. Wing, the corporate vice president overseeing Microsoft’s core research labs, said that Horvitz’s research has had an impact on countless research projects and commercial products, ranging from systems that help make our commutes easier to ones that seek to prevent hospital readmissions.

“His impact is immeasurable,” she said.

But Wing noted that Horvitz also has been able to step back and see the big picture, becoming a visionary and a thought leader in a field that is growing increasingly complex.

“He asks big questions: How do our minds work? What computational principles and architectures underlie thinking and intelligent behavior? How can computational models perform amidst real-world complexities such as sustainability and development? How can we deploy computation systems that deliver value to people and society?” Wing said.

The Newell award is given to a researcher whose work has breadth within computer science or spans multiple disciplines. Horvitz’s work has combined multiple computer science disciplines and he has been a leader in exploring the interrelationships between artificial intelligence and fields like decision science, cognitive science and neuroscience.

The award comes at a time when the artificial intelligence field is exploding.

Until a few years ago, artificial intelligence wasn’t often part of the public consciousness, except when it came up in a science fiction novel or blockbuster movie.

Now,  thanks to breakthroughs in the availability of data and our ability to process it, artificial intelligence applications are  suddenly everywhere, including systems that can understand and translate language, recognize and caption photos and do increasingly smart and useful things for us.

During a time often referred to as the “AI winter,”  Horvitz was among the nation’s hard-charging researchers plugging away at the difficult work of laying the groundwork for these systems and thinking about how they would work in the real world. Although artificial intelligence was out of the spotlight during that time, researchers were making major breakthroughs in bringing together the logical methods of traditional artificial intelligence work with research in fields such as decision science. This led to new applications that used both logic and probability.

Horvitz said that many of his research projects over the last fifteen years – which have looked at things like what we are most likely to remember or forget and when it’s worth it to interrupt someone while working – foreshadow practical applications that he expects to see in the future.

“To me, Eric is such an epic example of those brilliant researchers who have this huge confidence — not over-confidence, but just confidence — to keep pushing forward,” Shum said.

Horvitz’s attention to both research advances and practical applications of artificial intelligence research began while he was pursuing his Ph.D. on principles of bounded rationality. That’s the idea that when people or computers make decisions, they are limited by time, available information and their reasoning abilities.

Horvitz said he was interested in how computing systems immersed in the real world could make the best decisions in time-critical situations. His research looked at the value of continuing to think about a problem versus stopping early with a good enough answer.

His research considered emergency room scenarios, in which artificial intelligence systems could help doctors with timely recommendations. The work foreshadowed his later research on using similar ideas to guide solutions to some of the hardest challenges known in artificial intelligence, in the realm of theorem proving.

Horvitz also showed how artificial intelligence systems could be used to better understand people’s goals and intentions and provide the best information to decision makers. He collaborated with NASA’s Mission Control Center on how to provide flight engineers with the most valuable information about space shuttle systems when the engineers are under intense time pressure.

To solve these problems — and many more after — Horvitz brought together artificial intelligence methods with ideas drawn from disciplines like probability theory, decision theory and studies of bounded rationality.

In the future, Horvitz said he sees vast possibilities for how artificial intelligence can help to augment human intelligence.

“There’s a huge opportunity ahead in building systems that work closely with people to help them to achieve their goals,” Horvitz said.

Related:

The post Eric Horvitz receives ACM-AAAI Allen Newell Award for groundbreaking artificial intelligence work appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/04/27/eric-horvitz-receives-acm-aaai-allen-newell-award-groundbreaking-artificial-intelligence-work/feed/ 0
You might not see the next wave of breakthrough tech, but it’s all around you http://blogs.microsoft.com/next/2016/04/18/you-might-not-see-the-next-wave-of-breakthrough-tech-but-its-all-around-you/ http://blogs.microsoft.com/next/2016/04/18/you-might-not-see-the-next-wave-of-breakthrough-tech-but-its-all-around-you/#respond Mon, 18 Apr 2016 15:05:51 +0000 http://blogs.microsoft.com/next/?p=56797 Think of your favorite pieces of technology. These are the things that you use every day for work and play, and pretty much can’t live without. Chances are, at least one … Read more »

The post You might not see the next wave of breakthrough tech, but it’s all around you appeared first on Next at Microsoft.

]]>

Think of your favorite pieces of technology. These are the things that you use every day for work and play, and pretty much can’t live without.

Chances are, at least one of them is a gadget – your phone, maybe, or your gaming console.

But if you really think about it, chances also are good that many of your most beloved technologies are no longer made of plastic, metal and glass.

Maybe it’s a streaming video service you use to binge watch “Game of Thrones” on or an app that lets you track your steps and calories so you can fit into those jeans you wore back in high school. Maybe it’s a virtual assistant that helps you remember where your meetings are and when you need to take your medicine, or an e-reader that lets you get lost in your favorite book via your phone, tablet or even car speakers.

Perhaps, quietly and without even realizing it, your most beloved technologies have gone from being things you hold to services you rely on, and that exist everywhere and nowhere. Instead of the gadgets themselves, they are tools that you expect to be able to use on any type of gadget: Your phone, your PC, maybe even your TV.

They are part of what Harry Shum, executive vice president in charge of Microsoft’s Technology and Research division, refers to as an “invisible revolution.”

“We are on the cusp of creating a world in which technology is increasingly pervasive but is also increasingly invisible,” Shum said.

Read the full story.

The post You might not see the next wave of breakthrough tech, but it’s all around you appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/04/18/you-might-not-see-the-next-wave-of-breakthrough-tech-but-its-all-around-you/feed/ 0
Teaching computers to describe images as people would http://blogs.microsoft.com/next/2016/04/14/teaching-computers-to-describe-images-as-people-would/ http://blogs.microsoft.com/next/2016/04/14/teaching-computers-to-describe-images-as-people-would/#respond Thu, 14 Apr 2016 13:00:49 +0000 http://blogs.microsoft.com/next/?p=56611 Let’s say you’re scrolling through your favorite social media app and you come across a series of pictures of a man in a tuxedo and a woman in a long … Read more »

The post Teaching computers to describe images as people would appeared first on Next at Microsoft.

]]>
Let’s say you’re scrolling through your favorite social media app and you come across a series of pictures of a man in a tuxedo and a woman in a long white dress.

An automated image captioning system might describe that scene as “a picture of a man and a woman,” or maybe even “a bride and a groom.” But a person might look at the pictures and think, “Wow, my friends got married! They look so happy. What a beautiful wedding.”

As image captioning tools get increasingly good at correctly recognizing the objects in an image, a group of researchers is taking the technology one step further. They are working on a system that can automatically describe a series of images in the same kind of way that a human would, by focusing not just on the items in the picture but also what’s happening and how it might make a person feel.

“Captioning is about taking concrete objects and putting them together in a literal description,” said Margaret Mitchell, a Microsoft researcher who is leading the research project. “What I’ve been calling visual storytelling is about inferring conceptual and abstract ideas from those concrete objects.”

For example, while another image captioning system might describe an image as “a group of people dancing,” the visual storytelling system would instead say “We had a ton of fun dancing.” And while another captioning system might say, “This is a picture of a float in a parade,” this system would instead say “Some of the floats were very colorful.”

The research project, which relies on a new Microsoft Sequential Image Narrative Dataset, doesn’t just stop at one picture. Instead, it takes a series of pictures about the same event and strings together several sentences describing what’s going on. The work will be presented in June at the annual meeting of the North American Chapter of the Association for Computational Linguistics.

‘Ready for the next step’
The researchers say visual storytelling could eventually be helpful for people who are sharing a number of pictures on social media and want a tool that will help them build a narrative about those pictures. It also could potentially be used to provide richer descriptive tools for people who are blind or visually impaired.

“In image captioning, there are a lot of things we can do reasonably well, and that means we are ready for the next step,” said Ting-Hao (Kenneth) Huang, a Ph.D. candidate at Carnegie Mellon University who worked on the project as part of a summer internship at Microsoft Research. “I think the computer can generate a reasonably simple story, like what we see in a children’s book.”

Huang was the first author on a paper about the work, along with another summer intern from Johns Hopkins University, Francis Ferraro.

 

‘Translating’ from images to sentences
The fields of computer vision and natural language processing have made significant advances in the past few years. That’s thanks in part to the more widespread use of a machine learning methodology called deep neural networks. These methods have helped researchers get much more accurate results for pattern recognition tasks like speech recognition and identifying objects in photos.

To build the visual storytelling system, the researchers used the deep neural networks to create a “sequence to sequence” machine learning system that is similar to the kind other computer scientists have used for automated language translation. In this case, however, instead of translating from, say, French to English, the researchers were training the system to translate from images to sentences.

For a machine learning system to work, it needs a training set of data that it can learn from. To build the visual storytelling system’s training set, the researchers hired crowdsourced workers to write sentences describing various scenes. To account for variations in how people described the scenes, the tool was trained to prefer language in which there was consensus, and to create sentences based on that common ground.

The team also created a separate test set, so they could compare the machine’s descriptions with how a human described the scene.

Then, they fed the system new images and asked it to create sentences based on the knowledge it had from the training set.

 

The research is still in the early stages, and the researchers admit there’s significant progress to be made. Still, the researchers say these most recent advances represent another milestone in the fast-moving effort to use machine learning and other methods from the broader field of artificial intelligence for valuable applications. The new work on visual storytelling brings artificial intelligence a step closer to interpreting the world in the complex, nuanced ways that humans do.

“A picture is worth 1,000 words. It’s not just worth three tags,” Mitchell said.

Still, the researchers caution that this system – and other cutting-edge research projects like it – are still far from reaching a human level of cognition.

“We’re really all scratching the surface,” said Nasrin Mostafazadeh, a Ph.D. candidate at the University of Rochester who worked on the project as an intern at Microsoft Research. “It’s not that we’re doing it, really, in the way that humans do it. It’s just that we’re trying to.”

Related:

Learn more about the Microsoft Sequential Image Narrative Dataset

Read the paper: Visual Storytelling, by Ting-Hao (Kenneth) Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley and Margaret Mitchell.

Decades of computer vision research, one ‘Swiss Army knife’

Project Malmo: Using Minecraft to build more intelligent technology

The quest to create technology that understands speech as well as a human

Allison Linn is a senior writer at Microsoft. Follow her on Twitter

 

 

 

 

 

 

The post Teaching computers to describe images as people would appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/04/14/teaching-computers-to-describe-images-as-people-would/feed/ 0
Decades of computer vision research, one ‘Swiss Army knife’ http://blogs.microsoft.com/next/2016/03/30/decades-of-computer-vision-research-one-swiss-army-knife/ http://blogs.microsoft.com/next/2016/03/30/decades-of-computer-vision-research-one-swiss-army-knife/#respond Wed, 30 Mar 2016 21:00:21 +0000 http://blogs.microsoft.com/next/?p=56506 When Anne Taylor walks into a room, she wants to know the same things that any person would. Where is there an empty seat? Who is walking up to me, … Read more »

The post Decades of computer vision research, one ‘Swiss Army knife’ appeared first on Next at Microsoft.

]]>
When Anne Taylor walks into a room, she wants to know the same things that any person would.

Where is there an empty seat? Who is walking up to me, and is that person smiling or frowning? What does that sign say?

For Taylor, who is blind, there aren’t always easy ways to get this information. Perhaps another person can direct her to her seat, describe her surroundings or make an introduction.

There are apps and tools available to help visually impaired people, she said, but they often only serve one limited function and they aren’t always easy to use. It’s also possible to ask other people for help, but most people prefer to navigate the world as independently as possible.

That’s why, when Taylor arrived at Microsoft about a year ago, she immediately got interested in working with a group of researchers and engineers on a project that she affectionately calls a potential “Swiss Army knife” of tools for visually impaired people.

“I said, ‘Let’s do something that really matters to the blind community,’” said Taylor, a senior project manager who works on ways to make Microsoft products more accessible. “Let’s find a solution for a scenario that really matters.”

That project is Seeing AI, a research project that uses computer vision and natural language processing to describe a person’s surroundings, read text, answer questions and even identify emotions on people’s faces. Seeing AI, which can be used as a cell phone app or via smart glasses from Pivothead, made its public debut at the company’s Build conference this week. It does not currently have a release date.

Taylor said Seeing AI provides another layer of information for people who also are using mobility aids such as white canes and guide dogs.

“This app will help level the playing field,” Taylor said.

At the same conference, Microsoft also unveiled CaptionBot, a demonstration site that can take any image and provide a detailed description of it.

Very deep neural networks, natural language processing and more
Seeing AI and CaptionBot represent the latest advances in this type of technology, but they are built on decades of cutting-edge research in fields including computer vision, image recognition, natural language processing and machine learning.

In recent years, a spate of breakthroughs has allowed computer vision researchers to do things they might not have thought possible even a few years before.

“Some people would describe it as a miracle,” said Xiaodong He, a senior Microsoft researcher who is leading the image captioning effort that is part of Microsoft Cognitive Services. “The intelligence we can say we have developed today is so much better than six years ago.”

The field is moving so fast that it’s substantially better than even six months ago, he said. For example, Kenneth Tran, a senior research engineer on his team who is leading the development effort, recently figured out a way to make the image captioning system more than 20 times faster, allowing people who use tools like Seeing AI to get the information they need much more quickly.

A major a-ha moment came a few years ago, when researchers hit on the idea of using deep neural networks, which roughly mimic the biological processes of the human brain, for machine learning.

Machine learning is the general term for a process in which systems get better at doing something as they are given more training data about that task. For example, if a computer scientist wants to build an app that helps bicyclists recognize when cars are coming up behind them, it would feed the computer tons of pictures of cars, so the app learned to recognize the difference between a car and, say, a sign or a tree.

Computer scientists had used neural networks before, but not in this way, and the new approach resulted in big leaps in computer vision accuracy.

Several months ago, Microsoft researchers Jian Sun and Kaiming He made another big leap when they unveiled a new system that uses very deep neural networks – called residual neural networks – to correctly identify photos. The new approach to recognizing images resulted in huge improvements in accuracy. The researchers shocked the academic community and won two major contests, the ImageNet and Microsoft Common Objects in Context challenges.

Tools to recognize and accurately describe images
That approach is now being used by Microsoft researchers who are working on ways to not just recognize images but also write captions about them. This research, which combines image recognition with natural language processing, can help people who are visually impaired get an accurate description of an image. It also has applications for people who need information about an image but can’t look at it, such as when they are driving.

The image captioning work also has received accolades for its accuracy as compared to other research projects, and it is the basis for the capabilities in Seeing AI and Caption Bot. Now, the researchers are working on expanding the training set so it can give users a deeper sense of the world around them.

Margaret Mitchell

Margaret Mitchell

Margaret Mitchell, a Microsoft researcher who specializes in natural language processing and has been one of the industry’s leading researchers on image captioning, said she and her colleagues also are looking at ways a computer can describe an image in a more human way.

For example, while a computer might accurately describe a scene as “a group of people that are sitting next to each other,” a person may say that it’s “a group of people having a good time.” The challenge is to help the technology understand what a person would think was most important, and worth saying, about the picture.

“There’s a separation between what’s in an image and what we say about the image,” said Mitchell, who also is one of the leads on the Seeing AI project.

Other Microsoft researchers are developing ways that the latest image recognition tools can provide more thorough explanations of pictures. For example, instead of just describing an image as “a man and a woman sitting next to each other,” it would be more helpful for the technology to say, “Barack Obama and Hillary Clinton are posing for a picture.”

That’s where Lei Zhang comes in.

When you search the Internet for an image today, chances are high that the search engine is relying on text associated with that image to return a picture of Kim Kardashian or Taylor Swift.

Zhang, a senior researcher at Microsoft, is working with researchers including Yandong Guo on a system that uses machine learning to identify celebrities, politicians and public figures based on the elements of the image rather than the text associated with it.

Zhang’s research will be included in the latest vision tools that are part of Microsoft Cognitive Services. That’s a set of tools that is  based on Microsoft’s cutting-edge machine learning research, and which developers can use to build apps and services that do things like recognize faces, identify emotions and distinguish various voices. Those tools also have provided the technical basis for Microsoft showcase apps and demonstration websites such as how-old.net, which guesses a person’s age, and Fetch, which can  identify a dog’s breed.

Microsoft Cognitive Services is an example of what is becoming a more common phenomenon – the lightning-fast transfer of the latest research advances into products that people can actually use. The engineers who work on Microsoft Cognitive Services say their job is a bit like solving a puzzle, and the pieces are the latest research.

“All these pieces come together and we need to figure out, how do we present those to an end user?” said Chris Buehler, a software engineering manager who works on Microsoft Cognitive Services.

From research project to helpful product
Seeing AI, the research project that could eventually help visually impaired people, is another example of how fast research can become a really helpful tool. It was conceived at last year’s //oneweek Hackathon, an event in which Microsoft employees from across the company work together to try to make a crazy idea become a reality.

The group that built Seeing AI included researchers and engineers from all over the world who were attracted to the project because of the technological challenges and, in many cases, also because they had a personal reason for wanting to help visually impaired people operate more independently.

“We basically had this super team of different people from different backgrounds, working to come up with what was needed,” said Anirudh Koul, who has been a lead on the Seeing AI project since its inception and became interested in it because his grandfather is losing his ability to see.

For Taylor, who joined Microsoft to represent the needs of blind people, it was a great experience that also resulted in a potential product that could make a real difference in people’s lives.

“We were able to come up with this one Swiss Army knife that is so valuable,” she said.

Related:

Learn more about Build 2016

Read more about the latest news from Build

Watch the audio description version of the Seeing AI video

Read about Cortana Intelligence Suite

Allison Linn is a senior writer at Microsoft. Follow her on Twitter.

The post Decades of computer vision research, one ‘Swiss Army knife’ appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/03/30/decades-of-computer-vision-research-one-swiss-army-knife/feed/ 0
Project Malmo: Using Minecraft to build more intelligent technology http://blogs.microsoft.com/next/2016/03/13/project-malmo-using-minecraft-build-intelligent-technology/ http://blogs.microsoft.com/next/2016/03/13/project-malmo-using-minecraft-build-intelligent-technology/#respond Mon, 14 Mar 2016 05:00:36 +0000 http://blogs.microsoft.com/next/?p=56433 Editor’s note, April 1, 2016: This project was formerly known as Project AIX and has now been renamed Project Malmo. In the airy, loft-like Microsoft Research lab in New York … Read more »

The post Project Malmo: Using Minecraft to build more intelligent technology appeared first on Next at Microsoft.

]]>
Editor’s note, April 1, 2016: This project was formerly known as Project AIX and has now been renamed Project Malmo.

In the airy, loft-like Microsoft Research lab in New York City, five computer scientists are spending their days trying to get a Minecraft character to climb a hill.

That may seem like a pretty simple job for some of the brightest minds in the field, until you consider this: The team is trying to train an artificial intelligence agent to learn how to do things like climb to the highest point in the virtual world, using the same types of resources a human has when she learns a new task.

That means that the agent starts out knowing nothing at all about its environment or even what it is supposed to accomplish. It needs to understand its surroundings and figure out what’s important – going uphill – and what isn’t, such as whether it’s light or dark. It needs to endure a lot of trial and error, including regularly falling into rivers and lava pits. And it needs to understand – via incremental rewards – when it has achieved all or part of its goal.

Project Malmo research

Fernando Diaz, Akshay Krishnamurthy and Alekh Agarwal are using Project Malmo for AI research. Photo by Scott Eklund/Red Box Pictures.

“We’re trying to program it to learn, as opposed to programming it to accomplish specific tasks,” said Fernando Diaz, a senior researcher in the New York lab and one of the people working on the project.

The research project is possible thanks to Project Malmo, a platform developed by Katja Hofmann and her colleagues in Microsoft’s Cambridge, UK, lab and unveiled publicly on Monday. Project Malmo allows computer scientists to use the world of Minecraft as a testing ground for conducting research designed to improve artificial intelligence.

Microsoft researchers are using Project Malmo for their own research, and they have made it available to a small group of academic researchers under a private beta. This summer, Project Malmo will be available via an open-source license.

Hofmann came up with the idea for Project Malmo about a year ago, in part because she was frustrated by the limitations of other platforms that use simpler, less sophisticated games for artificial intelligence research.

Minecraft is ideal for artificial intelligence research for the same reason it is addictively appealing to the millions of fans who enter its virtual world every day. Unlike other computer games, Minecraft offers its users endless possibilities, ranging from simple tasks like walking around looking for treasure to complex ones like building a structure with a group of teammates.

“Minecraft is the perfect platform for this kind of research because it’s this very open world,” Hofmann said. “You can do survival mode, you can do ‘build battles’ with your friends, you can do courses, you can implement our own games. This is really exciting for artificial intelligence because it allows us to create games that stretch beyond current abilities.”

From doing to learning
Over the past few years, artificial intelligence researchers have gotten very good at teaching computers to do specific, often complicated tasks. Computers can now understand speech and translate it. They can recognize images and write captions about them.

But despite all these advances, computers still aren’t very good at what researchers call general intelligence, which is more similar to the nuanced and complex way humans learn and make decisions. A computer algorithm may be able to take one task and do it as well or even better than an average adult, but it can’t compete with how an infant is taking in all sorts of inputs – light, smell, touch, sound, discomfort – and learning that if you cry chances are good that Mom will feed you.

“The things that seem really easy for us are actually the things that are really difficult for an artificial intelligence,” said Robert Schapire, a principal researcher at Microsoft Research who is part of the team using Project Malmo in Microsoft’s New York lab.

Hofmann said artificial intelligence researchers are able to take tiny slices of that total awareness and build tools that do one thing, like recognize words, but they haven’t been able to combine them in the way that humans do effortlessly. She said that’s partly because we don’t really know how humans are combining those senses.

“We don’t understand ourselves well enough,” she said.

Project Malmo demonstration

David Bignell, Tim Hutton, Katja Hofmann and Matthew Johnson are working on Project Malmo. Photography by Scott Eklund/Red Box Pictures.

From theory to practice
There has been plenty of theoretical research into general artificial intelligence, but researchers have always been limited by practical ways to test their systems.

Building a robot and trying to teach it to climb a real hill is costly and impractical; unlike in Minecraft, you’d have to repair or replace the robot with another costly machine each time it fell into a river.

It’s also not that easy to test general artificial intelligence research on systems people are using in the real world. Hofmann’s background is in how to make search more of an intelligent assistant than a simple information retrieval system, but she said one problem with testing her theories in real-world scenarios is that millions of people are depending on search engines to work in a predictable way.

“It’s hard to test some of this in practice, and that’s one of the main motivations for building the platform,” Hofmann said.

The Minecraft platform was especially appealing because it allows players to make really complex decisions that have consequences, and to add more and more difficult elements as they get better. It also lets users work together, which could help researchers experiment with how humans and artificially intelligent agents could work together.

“It’s a digital playpen for artificial intelligence,” Diaz said. “It’s an environment in which we can develop an algorithm for teaching a young artificial intelligence to learn different concepts in the world.”

Advancing all artificial intelligence research
From the beginning, Hofmann said the goal with Project Malmo was to build a system that would be useful both for Microsoft’s own research and for the broader artificial intelligence research community.

“We’re looking for opportunities where we can really help accelerate the pace of artificial intelligence innovation in a way that is going to be very close to the real world, with real experiences and real data,” said Evelyne Viegas, the director of artificial intelligence outreach at Microsoft Research.

The Project Malmo platform consists of a mod for the Java version and code that helps artificial intelligence agents sense and act within the Minecraft environment. The two components can run on Windows, Linux or Mac OS, and researchers can program their agents in any programming language they are comfortable with.

Matthew Johnson, the development lead on the project who also works in Microsoft’s UK lab, said the team developed the system with the hope that it would attract a broad range of academic researchers and motivated amateurs, with all levels of programming skills, background and goals. That said, the platform is intended for research into various forms of artificial intelligence and is not a consumer product.

“Our focus, from the beginning, has been on making sure that there’s the lowest possible barrier to innovation,” Johnson said.

Related:

Read more about the research group behind Project Malmo

Research paper: Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains

The future of artificial intelligence: Myths, realities and aspirations

Follow Katja Hofmann on Twitter

Allison Linn is a senior writer at Microsoft. Follow her on Twitter.

The post Project Malmo: Using Minecraft to build more intelligent technology appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/03/13/project-malmo-using-minecraft-build-intelligent-technology/feed/ 0
How Microsoft and Novartis created Assess MS http://blogs.microsoft.com/next/2016/02/08/how-microsoft-and-novartis-created-assess-ms/ http://blogs.microsoft.com/next/2016/02/08/how-microsoft-and-novartis-created-assess-ms/#respond Tue, 09 Feb 2016 05:07:58 +0000 http://blogs.microsoft.com/next/?p=56361 When Microsoft released the Kinect system for playing Xbox video games about five years ago, it attracted the interest of an unlikely source: the healthcare company Novartis. For years, Novartis … Read more »

The post How Microsoft and Novartis created Assess MS appeared first on Next at Microsoft.

]]>
When Microsoft released the Kinect system for playing Xbox video games about five years ago, it attracted the interest of an unlikely source: the healthcare company Novartis.

For years, Novartis has been trying to find more consistent ways to quantify whether the treatments it is developing for multiple sclerosis are working, but assessing whether a patient’s symptoms are stabilizing or getting worse is complicated.

The possibility of using computer vision, which is the type of technology found in the Kinect system, was intriguing. Using a tool like the Kinect, the researchers at Novartis figured they could get a more consistent reading of how a patient performed on a set of standardized tests for MS patients, bringing a new level of uniformity that would help doctors better assess the progress of the disease. That, in turn, could speed up the process of getting the right treatments to patients.

To find out more about the Assess MS research project that resulted from that idea, read the full story

The post How Microsoft and Novartis created Assess MS appeared first on Next at Microsoft.

]]>
http://blogs.microsoft.com/next/2016/02/08/how-microsoft-and-novartis-created-assess-ms/feed/ 0