Skip to content
Link copied to clipboard

New musical resonance, via your cell phone

It is the 11th commandment for concertgoers, especially in the august world of classical music: Thou shalt turn off thy cell phone before the performance begins.

It is the 11th commandment for concertgoers, especially in the august world of classical music: Thou shalt turn off thy cell phone before the performance begins.

But now, some Drexel University computer scientists would like you to turn the phone on - provided it's an iPhone.

They've programmed a computer to "listen" to a live orchestral performance and then transmit real-time tidbits to the phone's display screen, describing the relevant music theory and context.

There might be a heads-up that, say, a trumpet fanfare is about to begin, or perhaps an explanation as to why the piece is shifting to a minor key.

The Philadelphia Orchestra is not quite ready to let this device into the Kimmel Center's Verizon Hall, but it has been working with Drexel on the project and sent representatives to the campus this week for a demonstration. In an era of shrinking audiences, could this be the way to resonate with the masses who don't know a coloratura from a contralto?

"It's a very exciting technology," violinist Philip Kates said, though he wondered if it might make it harder for some to experience "the spirit of the music."

Kates and two dozen others, including orchestra board members and Drexel students, tried out the system in a Drexel auditorium Wednesday. They watched a high-definition broadcast of a June 6 orchestra performance, each holding an iPod touch or an iPhone (ringer off, of course).

The listening software was loaded onto a Macbook Pro, located in the auditorium control room, and as it followed the music, the relevant information was beamed to the iPhones.

The computer appeared unfazed by differences between the orchestra's renditions and those on the CDs that the software had used to "learn" the pieces.

For example, at one point in Mozart's Violin Concerto No. 5 in A major, violinist Arabella Steinbacher launched into a cadenza - a solo that allowed her to improvise and showcase her virtuosity.

Steinbacher's solo was far different from the CD version that the computer program had used as a reference point. Yet the software correctly identified the beginning and ending of the cadenza, and the iPhone screens offered a link for those who needed the word defined.

Then, during a performance of Richard Strauss' Don Quixote, the screens provided a running narrative of the hero's exploits and explained how they were represented by the various instruments. At one point, the devices pointed out that the tuba and contrabassoon conveyed the snores of sidekick Sancho Panza.

The comments, written by Leonardo Dugan, an education coordinator for the orchestra, appeared on the screens as white letters on a dark background. The faint glow did not seem as if it would distract a nearby audience member who had chosen not to use the device.

It all began two years ago when Youngmoo Kim, a Drexel assistant professor of electrical and computer engineering who has a classical-music background, told his lab members to attend a high-definition broadcast of an orchestra concert.

Afterward, Kim recalled, this was a typical reaction from the group of twenty-somethings:

"That was very nice, but I didn't really understand the piece."

That gave Kim an idea. Why not invent something like those portable audio guides at art museums? Without the audio, of course. A former member of the Boston Symphony Orchestra's Tanglewood Festival Chorus, Kim knew that such a device would have to keep a very low profile, given the exacting standards of classical-music audiences.

During the current school year, three of his students - David Grunberg, Alex Hrybyk, and Matthew Prockup - set about making the idea a reality.

The device works by taking snapshots of the music every third of a second. Each segment of the audio signal is analyzed to determine how much of each of the 12 pitch classes is present - A, B-flat, B, and so on.

The system doesn't distinguish between notes in different octaves; low, middle, and high C notes, for example, are all lumped together.

This is done first from a reference recording, and then someone writes the content to correspond with the various passages.

During any subsequent performance of the same music, the software analyzes it with the same technique, and matches the new performance to the reference version with a process called dynamic time-warping.

"It's the process of trying to stretch or shrink one audio source into another," Kim said, accounting for differences in tempo.

There is a human precedent for this. During operas, many theaters provide "supertitles" - translations of the words on a screen above the stage. But a live person makes the decision on when to advance from one line to the next.

With the Drexel application, it's automatic. And if you miss one of the comments, a flick of the finger allows you to go back and read it again.

Thus far, Drexel has tested the system only on live, high-definition broadcasts of the orchestra's performances - provided by SpectiCast, a Philadelphia company. But it would be no different if the software were operating in the theater; it would be analyzing the same kind of audio feed.

The orchestra's point person on the project is Ayden Adler, who contributed numerous ideas. Among them: The application includes a sort of map to indicate where the orchestra is in the performance - somewhat like one of those maps displayed on the screens during a long airplane flight.

Adler, director of the orchestra's education and community partnerships department, proclaimed the result to be "fantastic" but said the orchestra had yet to decide how it might be used.

Kim said that it was primarily an academic exercise so far, but that he would be interested in exploring commercial opportunities.

All three of the students who led the yearlong project are musicians, and two knew something about classical music. The third, Hrybyk, likes rock music and plays electric guitar.

"It definitely has made me appreciate classical music a lot more," Hrybyk said of the project.

Kim seems to have accomplished his goal already with one person, at least.