Kinect recreates reality and brings new dimension to live show at CMJ Music Marathon – and beyond

At CMJ Music Marathon in New York City this week, attendees who went to Neon Indian’s live show saw the five members of the band, but they also saw abstract visual projections of them in motion during their performance, made possible by Kinect sensors.

“Since the beginning, we’ve had a lot of different permutations of the live show that involves visuals.  Much of the ethos behind that is to be feeding information to the synthesizer and having it generate visuals that are corresponding with the actual movements happening in the songs. So when I heard about this opportunity to work with the Kinect, it seemed like a new iteration of that concept and something I wanted to be part of,” says Alan Palomo, lead vocalist for Neon Indian, an electronic music band whose third album, “Vega Intl. Night School,” debuts Oct. 16. CMJ each year draws more than 120,000 music fans to hundreds of artists’ performances at more than 80 venues. “And now the new component is we’re not just feeding information from the audio tracks, but the movement of us as performers, in reaction to the audience.”

Palomo and Neon Indian will take the Kinect set-up on the road with them on a grueling tour that keeps them on the move until just before Christmas. Each of five Kinect sensors capture a different band member, reacting to the music and movements happening on stage. And depending on the space, it’ll be a completely different experience every time. Shortly after the festival, a performance toolkit will be released to the CMJ community with a how-to and code, so other bands and musicians can create this type of stage presence as well. CMJ’s 35-year history includes more than 60,000 performances, some of which launched the careers of musical acts such as Mumford & Sons, Arcade Fire and Lady Gaga.

For bands like Neon Indian, it’s a good place to stand out.

“It’s now topographical type information, reading movements, playing alongside you as self-generative content,” says Palomo, who recently performed on “The Tonight Show Starring Jimmy Fallon.” The new album was chosen by Pitchfork as Best New Music. “It’s definitely a new paradigm, at least for us.”

Throughout the event, technology intertwined with music.

CMJ badge holders attended a panel discussion on the evolving role of interactive technology in the music space. They got a behind-the-scenes perspective on Neon Indian’s show as well as the creation of DELQA, an interactive installation powered by Kinect that transformed the music of Matthew Dear. The Microsoft mini Cube, an experiential art installation, was also on-site where education and networking sessions took place.

“We’re always looking for partners who want to use technology in interesting and innovative ways. We were happy to find a true spirit of collaboration and desire to do something new in the CMJ team and the artists who perform in the festival’s lineup. Technology provides ways for artists to push the boundaries of their show. Using Microsoft Kinect technology, the band members’ movements control all the aspects of the performance,” says Amy Sorokas, director of Brand Studios for Microsoft. “Collaborating with experts from the community already doing incredible things with Kinect and other Microsoft technology was a great way to come up with an innovative stage presence with the band. It was a great fit.”

Kamil Nawratil of the Brooklyn-based VolvoxLabs, the art director of the live performance piece, says the Neon Indian experience is the inaugural debut of the “Recreating Reality” concept.

“I think the most interesting part is looking at the real world through the Kinect sensors, recreating virtual reality,” says Nawratil, who worked with programmers and other designers to create the visual effect and put it all together. While lead singer Palomo gets a whole silhouette as he plays the synthesizer, sings and dances, the other images will focus more on the instruments the rest of the band is playing, and the audience sees those projections on floor-to-ceiling screens. “These dynamic objects represent human body and geometry, but it’s all virtual reality, flying around in virtual space. So you’ll see the humans, and then extensions of their reality behind them.”

Kinect drove the thinking behind the “Recreating Reality” concept.

“We wanted to do something more extreme in generating the visuals, push the envelope. There’s no way you can achieve something like this without the Kinect. There are other sensors, but they can’t compare,” says Nawratil, who chose Kinect for its plethora of data, availability of the SDK and device price. “Overall, it’s a winner.”

Combining creative aesthetics with technical innovation is a big part of Nawratil’s job.

“If you have a vision, you’ll find the tools,” he says.

Athima Chansanchai
Microsoft News Center Staff