Jason Holmberg locked eyes with a shark in Africa more than a decade ago. It changed his life—a common experience when in this kind of situation. But more importantly, this experience sparked interest that led to research developing an AI-enabled approach to conservation.
“Back in 2002, off the coast of Djibouti, I went scuba diving and saw my first whale shark. I thought it was amazing and loved the species,” says Holmberg, Director of Engineering for Wild Me and the Chief Information Architect of Wildbook. Joining a research expedition later that year, Holmberg learned that scientists would spear plastic tags onto whale sharks to track movement and get a better population count to further understand their place in the ocean ecosystem. However, the tags were rarely re-sighted—less than one percent of the time, meaning this method of animal identification was largely ineffective.
Holmberg believed that identification rate could be improved through a different path, by developing a computer vision algorithm to recognize the unique pattern of spots on different sharks. But then he ran into an unexpected block for sharing this tool with researchers around the world: “We obviously needed a web-based platform so that everyone can access it through their browser. And in fact, no such platform existed, there was nothing even close to it, so I would have to invent it from scratch, and I ended up doing that.”
This led to the formation of Wild Me, a non-profit organization that develops the data management platform and framework for wildlife research and conservation called Wildbook.
Today, Microsoft named Wild Me as one of their newest featured projects within AI for Earth, which will provide new resources and support to scale up and accelerate this conservation science work. Through this collaborative work, the organizations see a future where conservation work is more informed by data, yielding more information about species and better outcomes for species populations around the world.
Meeting the challenges of species extinction
Multiple independent analyses demonstrate that species are going extinct at unprecedented rates around the globe. In March 2018, the U.N.’s Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES) released a new set of reports finding that animal and plant species are under severe threat in nearly every region of the world.
That’s just a progress report on what we do know—which turns out to be very little. Scientists have observed and described only 1.5 million species of the estimated 10 million on Earth. And we are not on track to close this gap very quickly. To make progress on these challenges, researchers need to collect, process, and analyze more data on species more efficiently than ever, so that they can devise counter-strategies and engage the public for support. But many research teams lack the tools and resources to gather that data and work together on a large scale.
“Wildlife conservation desperately needs a digital transformation, it’s hungry for it,” explains Holmberg. “It needs to move from the world of small, proprietary data sets to large collaborative data sets. And to make sense of that, an individual research team in the lab just doesn’t scale appropriately. We need to add machine learning, computer vision, to look at larger data sets and to automate data processing.”
Tracking animals through image-based identification
That’s where Wild Me and the Wildbook project comes in. Wildbook blends a diversity of functions and disciplines into a cloud-based common platform with tools for data collection, storage, analysis, and collaboration that help researchers operate on a larger stage. Wildbook is developed by a corps of non-profit software engineers at Wild Me and incorporates the skills and research of a unique collaboration of experts, including:
- Tanya Berger-Wolf, Professor of Computer Science and head of the Computational Population Biology Lab at the University of Illinois at Chicago, providing computer science expertise, data science insight, and overall project leadership.
- Charles Stewart, Professor and Head of the Department of Computer Science at Rensselaer Polytechnic Institute (RPI) in Troy, NY, leading the computer vision algorithm development.
- Daniel Rubenstein, Professor of Ecology and Evolutionary Biology at Princeton University, providing ecological expertise and driving development for real world use cases through on site research in Kenya.
That collaboration of experts backed by engineers is crucial to solving these conservation challenges at the speed, resolution, and scale needed, Holmberg says. “The wildlife research community is still very much dealing in the world of small data. That limits the analyses, the research techniques, the statistical techniques that they can use. And their animals frankly don’t care about the scope of a research project or geographic borders, they wander across borders, oceans, and data sets.”
One way to address this challenge? Use crowd-sourced data in the form of images.
Thanks to the widespread availability and use of digital cameras, together with rapid innovations in storage technology and automatic image analysis software, there is an abundance of images as a cheap source of data. These, in turn, can be used to produce a database that records the animals’ identity, location, and time and date when they were photographed.
With the computer vision algorithms implemented by Wild Me—and drawing from the original research of Dr. Stewart and Dr. Berger-Wolf—individual members from a growing number of species can be identified by their unique patterns of spots, stripes, or other markings and physical features. Combining this with geographic, environmental, behavioral, and climate data helps to answer ecological and conservation queries, such as population sizes, species interactions, and movement patterns.
That, of course, depends on the ability of a person to sort through and identify each animal in every single photo. However, the power of the cloud and AI can rapidly increase the accuracy and speed of this classification. Images can be analyzed automatically through machine learning algorithms, which reduces the identification task per photo from several hours of human time to a few seconds of computer time. That frees up researchers to focus on devising conservation strategies, informed by ever expanding, accurate databases.
Scaling up with cloud computing and AI
Wild Me has a team of professional software engineers solely focused on developing Wildbook, so that field researchers don’t have to devote precious budget or time resources to develop their own tool for this work. But it’s still a very small team and the work needs a bigger platform to scale. To make that happen, Wild Me turned to Microsoft and its AI for Earth program.
“Microsoft is the perfect partner for our work,” says Holmberg. “We need scalability, and Azure has that, and the fact that Microsoft is investing so heavily into AI—it’s impressive and much needed.”
As Holmberg notes, Microsoft Azure offers the scalability and computing power to run the services of Wildbook, and also offers the Azure Marketplace, which makes Wildbook more widely available. Furthermore, the Marketplace offers the opportunity for Wild Me to make its computer vision algorithms available for other organizations to use in developing new services.
“Wildbook democratizes science and conservation. The partnership with Microsoft will allow us to enable science and conservation at planetary scale and high resolution over time, space, and individual animals,” says Dr. Berger-Wolf.
The maturity of the computational and scientific work, and the shared commitment to solving challenges at scale, made Wild Me a natural partner for Microsoft. “The world is facing a major biodiversity crisis, and Wild Me’s work in harnessing computer vision and machine learning to monitor and track individual animals is truly groundbreaking,” says Bonnie Lei, AI for Earth Project Manager at Microsoft. As Lei notes, “By working together, we can enable wider usage of their open source algorithms by making them available on Microsoft Azure as APIs.”
Revolutionizing biodiversity research
Looking ahead, Wild Me and Microsoft have ambitious aspirations together. One such initiative that takes advantage of the scalability of Azure is an intelligent agent, already in production on Wildbook for Whale Sharks, that can check YouTube nightly for videos titled or tagged “whale shark”. This agent uses machine learning to determine whether the video has relevant footage of a whale shark, and Microsoft Translator with natural language processing to read the video description and determine when and where the shark was sighted.
The agent can also use the same translation and natural language tools to automatically ask the video poster, in their own language, for information on when and where the video was taken, if it cannot be determined from the video posting itself. This marks an important step forward, as Wildbook can now proactively gather scientifically useful data on its own from the web, rather than wait for data to be submitted.
Wild Me is also building a tweet bot it calls Tweet-A-Whale that could interact with people on whale watching tours. People will be able to tweet whale photographs with a time and place to the bot for automatic identification—again, taking advantage of the power of AI and machine learning to recognize specific individual whales from the photos.
One thing is clear: wildlife researchers need more time, more resources, and a great deal of innovation to quickly address the growing biodiversity crisis. The partnership between Wild Me and Microsoft provides an opportunity to do just that.