Water Music: Navigating toward increased independence through Soundscape

By: Amos Miller, Microsoft AI & Research

Imagine you’re paddling on the lake, you feel the warm sun on your face, listen quietly to the calm ripples of the water. You can hear a sound beacon, in 3D, on the water, just in front of you and over to the right; you know exactly where it is, you steer the kayak towards it, paddle like crazy and catch it, collecting 25 points.

When your ability to perceive the space around you is enhanced in a natural, intuitive way, the urge to get out and enjoy that heightened sense of independence is irresistible. We experienced that feeling a few weeks ago on a beautiful sunny day on Lake Sammamish at the Microsoft Soundscape kayaking PaddlePalooza. What was that all about? Let me share the story.

In my June post, I noted the continued excitement about the Soundscape beacon from people who are blind or have low vision. Many of you have asked to be able to place the Soundscape audio beacon anywhere. In response, we recently introduced Soundscape markers. With Soundscape you can now quickly and easily mark-up anything in your surroundings that you care about. That could be, for example, your front door or the post box at the top of your street, or you can mark-up key landmarks in your area to help with learning a route. You might even mark the entrance to the park or the picnic spot on the lawn.

With your markers in place you can hear them through Soundscape or set a beacon to them, so you can always know where they are. This is a wonderful way for you to personalize the areas you walk in and build up confidence to go beyond what you are familiar with, always knowing that you have your markers for orientation. Do give it a try and tell us what you think. It’s a lot of fun.

We then thought, well, if you can place a marker anywhere what if we marked up a lake and used the markers for orientation on the water while kayaking? Hmmm. Ok, so this sounds new, and we figured it would be something that takes empowerment to a completely new level!

As part of the Microsoft One Week hackathon, we marked up seven virtual beacons on the lake. In partnership with Outdoors For All, we invited a group of people with blindness or low vision to join us for a kayaking scavenger hunt.

Click here for the audio description of the Soundscape Kayaking Scavenger Hunt video.

Wow! What an extraordinary day. When you wake up in the morning and by the end of the day realize you have done something you never imagined would be possible.

Now for the cool statistics! Of the 13 competing teams, 3 had only sighted paddlers, and 10 had blind or low vision paddlers. 11 of the 13 teams successfully found all seven beacons (covering an area roughly within a circle of 1000 meters in diameter) and scored the maximum points. The winning team completed the course in just 16 minutes!! The best sighted team came in sixth place at 25 minutes!

“We were having a blast, the sun was shining, the water was calm and warm, I had an attentive partner, and I was independently captaining a kayak and successfully found all seven beacons, earning the maximum 140 points in 25 minutes…the exhilaration of being able to compete, to experience the freedom of paddling on the water, and truly being the navigator of our team is something I’ll remember always,” noted Cindy Van Winkle, a participant from the Seattle Lighthouse. You can read more about her experience on her blog.

The kayaking experience motivated us to reflect, once again, on what we mean by the sense of independence and about presence and empowerment. Have people who are blind kayaked before? Sure, they do all the time. However, are they really the driver? On this occasion they indeed were the driver in every sense of the term.

Firstly, in the teams with a paddler who is blind, only the blind participant knew where the beacons were located; the sighted companions had no indication where they were, and therefore they were not able to help. That meant that the paddler who is blind didn’t even need to ask for that independence, it was built into the game. Secondly, the participants didn’t have an agent whispering in their ear, go left a bit, straight, right, nor did they receive any form of instructions they had to listen to and process. Instead, Soundscape, using 3D audio, enabled the participant to effortlessly hear exactly where the beacons were, leaving it for them to decide where to steer. This transfer of control not only creates a true sense of independence, but the participants in fact performed much better. I talk about this phenomenon in my TedX talk where I describe the feeling I had flying a glider.

A few months ago, at an event in Montana I had the pleasure of meeting Sharon. As we made our way out to learn how to use the Soundscape beacon, I noticed that Sharon stayed back. When I checked with her I understood she chose not to join because she is hard of hearing. After we established that she was still able to hear the high pitch of the Soundscape beacon, she gave it a try, and, with her husband walking along with her, off they went and within a few minutes made it to her destination. When I caught up with them I noticed she was overcome with emotion, this had been the first time in many years she truly felt that sense of independence and that she was in the driver’s seat. For her, this is what mattered at that moment, that sense of joy and empowerment; and, knowing that her life would now be different.

There is no greater gift that technology can provide than a true sense of independence.

I must say that the impact the kayak hack has really surprised and motivated us to think about how we can bring Soundscape to more people and more organizations to really benefit from the value it provides and the empowering changes it has started to enable for people. This is something that we are looking into, and if you, or an organization you know, run programs or activities in which you believe Soundscape can help then do get in touch with us at [email protected].

Before wrapping up let me highlight some of the other recent updates we made to Soundscape:

• We’ve been working to light up the experience while you’re travelling, whether in a bus, a car, or a train. To minimize interruptions and avoid getting in the way of whatever else you may be doing, Soundscape now identifies when you are in a moving vehicle and adjusts callouts to include only major landmarks, and update you only when you change roads.
• Battery savings! You no longer need to close Soundscape to save battery; you can now just select the “Sleep” button on the home screen to stop Soundscape from using GPS and your data plan.
• For our low vision users, you no longer need to hold the phone flat when pressing buttons; instead, you can now bring the phone right up close and press buttons, and everything will continue working correctly.

We can’t wait to hear what you make of them, please do continue to write to us on [email protected] and of course, check our FAQ’s on our website for more detailed information and guidance.

In closing we welcome our friends from down under to the Soundscape community! Soundscape was launched in Australia on September 12, 2018, with a great partnership with Vision Australia and the extended community of people who are blind. Based on our learnings from the launch in Australia we continue to look at how we can bring Soundscape to other countries and aim to support the local language where possible. We will be sure to provide further updates as soon as we can.

We continue to be humbled by the response to Soundscape, we recognize that this is just the start and there is more work to be done; but please keep the feedback coming, on anything from the app, to the content we produce, to the way we can engage with you. Let’s continue to empower everyone, everywhere with the benefits of technology.

Amos and the Soundscape team.

Microsoft Soundscape is available for free on iOS and iPhone in the US, UK and Australia.