Plastic bags. Plastic bottles. Plastic toys. Nearly 9 million tons of plastic debris wind up in the ocean every year. Environmental experts say the problem is serious enough that by the year 2050, the amount of plastics in the ocean could outweigh the fish in it if action isn’t taken sooner.
It was among the environmental concerns that gnawed at Drew Wilkinson. Two years ago, as a paralegal for Microsoft in Redmond, Washington, he had an idea for the company’s annual Global Hackathon, and contacted a nonprofit organization he greatly admired, The Ocean Cleanup, based in Rotterdam in the Netherlands. Maybe Microsoft could help?
Wilkinson called it a “shot-in-the-dark” email, the kind you send without knowing if it will land in the bulging inbox of an overloaded employee, or in a giant spam folder. And one that would travel 4,800 miles to another continent.
“I didn’t know anybody at The Ocean Cleanup,” Wilkinson says. “I just went on their website, and found their generic email address, and sent that email, not really expecting much. To my surprise, they responded.”
This was his email:
The communication was forwarded within The Ocean Cleanup offices to its geospatial analyst Robin de Vries. He received it like a Christmas gift in June.
“It felt that we could be helped by a network of experts,” De Vries remembers thinking. “This could be a great opportunity that only comes seldomly.”
The Ocean Cleanup is known worldwide for its innovative efforts to rid the ocean of plastics. It has also started focusing on eliminating plastics at major sources – rivers – before they reach the sea.
The organization built technology, unveiled last fall, that has been deployed in rivers in Indonesia and Malaysia, to remove plastics. But a key aspect was figuring out how to identify the waste that was collected – was it plastic or other material, such as sticks and leaves?
For two Microsoft global hackathons, in 2018 and 2019, Hackathon team members in Redmond and from around the world worked with The Ocean Cleanup to build a machine learning model to help quantify the amount of plastic pollution flowing down rivers en route to the ocean.
Subsequent models were then developed to replicate the process on cameras mounted to drones and ships crossing the ocean, and a blueprint for cloud computing infrastructure was created to help the project in the future.
Volunteers from Microsoft’s AI for Earth initiative also participated in the hackathon. AI for Earth is a Microsoft initiative that supports, and partners with, environmental groups and researchers to use artificial intelligence technology and advanced cloud software to solve environmental challenges.
AI for Earth sponsored the Hack for Sustainability Challenge at the hackathons in 2018 and 2019, with dozens of machine learning specialists, data scientists, software engineers, cloud architects, generalists and interns from around the company volunteering their time for The Ocean Cleanup.
De Vries and another team member from The Ocean Cleanup, project engineer Kees van Oeveren, traveled from the Netherlands to participate.
“That decision turned out to be the fuse to a phase of explosive development,” De Vries says. “When we were in Seattle, we got introduced to so many aspects of the Microsoft organization that it became clear they could be a powerful ally in the quest for clean oceans.”
Before Wilkinson’s email arrived, Van Oeveren recalled the tedious work of labeling some images and doing the work by himself. “I hadn’t embarked yet on doing machine learning and image recognition, but I really wanted to,” he says, although he was not sure of the next steps.
The project was rudimentary, painstaking and time-consuming. “I remember mounting security cameras with power banks attached to them at rivers that would just simply store some images locally” on a computer, Van Oeveren says.
It was a Tuesday when he shared a dataset of images with Microsoft Hackathon volunteers from the “Plastic Free Oceans” team to review. On Friday, in a video phone call meeting with the volunteers, he learned the dataset had already been put through a machine learning model, and bounding boxes had been created for the images, distinguishing what was plastic and what wasn’t.
“That was such a magical moment,” van Oeveren says. “I expected that even if you knew what you were doing, that it would be a project that would take weeks to set up right.”
“We labeled over 30,000 ocean photos with the help of Hackathon volunteers and their network” in the summer of 2019, De Vries says. “Some datasets were even finished before the Hackathon even started. Work is now in progress on the development of machine learning models for ocean photos.”
Dan Morris, AI for Earth program director, says the most important result from the hackathon was that AI for Earth taught The Ocean Cleanup a lot about machine learning. “The real value was teaching them through interaction with data scientists and engineers at Microsoft,” he says.
This year, The Ocean Cleanup was named an AI for Earth grantee for its work.
“Using the AI for Earth grant, we’ve been able to set up and run the machine learning models,” De Vries says. “Having the resources at our fingertips has greatly accelerated the technical progress, by taking away practical concerns and letting us focus on the development.
“It allowed us to develop the vision that this is something we can do, not just for one river, but eventually for rivers across the globe.”