Data from functional magnetic resonance imaging (fMRI) have provided eye-popping pictures of the way the brain is wired, and allowed neuroscientists and laypeople alike to view intricate anatomical and functional connections between regions of the brain. But what if a new tool could be applied to MRI and other data, to listen to the way the brain works and how it is forged with connections?
An emerging effort to “sonify” imaging data is taking root at UT Dallas’ Center for Vital Longevity, in the lab of Dr. Gagan Wig. The approach, now funded by the Defense Advanced Research Projects Agency (DARPA), allows data to be represented by sounds from which a trained listener might be able to discern patterns of brain connectivity not readily seen in available visualization strategies.
Wig, an assistant professor in the School of Behavioral and Brain Sciences, is working with his UT Dallas colleagues Dr. Roger Malina, Arts and Technology Distinguished Chair, Scot Gresham-Lancaster, assistant professor in the sound design program in the School of Arts, Technology, and Emerging Communication, and a mix of scientists, computer programmers and artists to translate data to sight and sound.
The yearlong effort is designed to create a dynamic prototype tool that will enable exploration of brain connections in a three-dimensional interactive video game environment.
“We have largely tried to understand how brain networks function by visualizing them. Certain insights, however, might be contained in a neural ‘song’ or signature that allows researchers to discern distinct rhythms or patterns of brain networks that might in themselves reflect sonic signatures, much like a chorus or the way a beehive might hum rhythmically with activity.”
The approach can be likened to the 1960s sci-fi movie Fantastic Voyage, where a small medical crew was shrunk to microscopic size and injected into the bloodstream of an injured scientist, yielding a stunning panorama of visuals and sounds in the inner workings of the human vascular system, traveling in a small submarine to repair a blood clot in the brain.
“We have largely tried to understand how brain networks function by visualizing them,” Wig said. “Certain insights, however, might be contained in a neural ‘song’ or signature that allows researchers to discern distinct rhythms or patterns of brain networks that might in themselves reflect sonic signatures, much like a chorus or the way a beehive might hum rhythmically with activity.”
The team’s initial data set is functional brain connectivity MRI information collected from a large sample of healthy adults ranging in age from 20 to 89 years, taken while they were at rest. Initial results, based on analyzing the data using mathematical analysis and visualization of brain networks, were recently reported by Wig’s group in the Proceedings of the National Academy of Sciences. In addition, the team is particularly emphasizing generating a user interface that allows a broader range of data sets to be incorporated, thus allowing use of data sets beyond ones studied by Wig’s lab
“The project aims both to create data exploration software for the use of the scientists, but also to enable performance,” Malina said. Gresham-Lancaster, with collaborating artists Tim Perkis and Andrew Blanton, will “perform” the data in art settings.
“Our new approach for representing complex patterns from information will have applications in medical research and also extend beyond the neuroscience domain,” Wig said. “The yearlong project will conclude with the development of a prototype that could be called or analogized to a data stethoscope that allows us to compare brain networks of healthy and unhealthy individuals,” just as standard stethoscopes are used by physicians to take vital signs and quickly determine abnormalities in the heart and lungs.
The installation was created by artistic duo Kristin Lee & Frank Dufour of Agence5970 an independent laboratory dedicated to conceptual art, using predominately sound, as well as image, exploring concepts emerging at the conjunction of perception and representation and of Time as a structural support of expression.
View the full artist statement on the Agence5970 website.
During the last year, a group of undergraduates created soundscapes for the museum, giving the students hands-on experience in the field of sound design.
“This is a typical project that comes out of ATEC in that it connects art, science and technology in a creative dialogue and also contributes to positioning the University as a pertinent partner with major cultural institutions in the Metroplex,” said Dr. Frank Dufour, a professor of sound design who led the digital music production course. “Students were given a wonderful opportunity to express themselves in a professional environment in which their creativity was welcome.”
The students worked closely with Dufour and Roxanne Minnish, a UT Dallas sound design instructor, composer and project manager, and with museum officials to refine their designs throughout the semester.
“Dr. Dufour was conceptual in the way he taught, but also practical and logical. He is inspiring as a teacher,” said Josh Casey, a senior who worked on the project. “He guided us and gave us advice and challenged students to think about what the soundscapes should really represent.”
Located north of downtown Dallas near Victory Park, the 180,000-square-foot museum features five floors of public space with 11 permanent exhibit halls, including a children’s museum and a hall designed to host traveling exhibitions. The museum opens to the public on Saturday, Dec. 1.
The digital music production class worked on sound designs for the 11 exhibit halls, which include the T. Boone Pickens Life Then and Now Hall, Being Human Hall, Discovering Life Hall, Rose Hall of Birds, Sports Hall and the Texas Instruments Engineering and Innovation Hall, among others.
The students worked together, collaborating on certain designs and critiquing each other’s projects throughout the semester.
“At first, when people criticize your work you may be offended, but I learned to become grateful for criticism – I learned a lot about the creative process and how to collaborate and bring new ideas together,” Casey said.
Each design required specific sounds that would enhance the educational and artistic quality of the exhibit. For example, there are heartbeats for baselines in the Being Human Hall soundscape and sweeping notes that rise and fall – like a flight pattern – in the Rose Hall of Birds design.
“Collaborating with UT Dallas on this project is just one of the ways that the Perot Museum seeks to enhance the visitor experience through highly immersive exhibits that cater to diverse learning styles,” said Steve Hinkley, vice president of programs at the Museum. “The students’ work demonstrates not only their dedication to innovation, but is a testament to visitors that the intersection of art and science is all around us.”
Frank and Kristin Lee Dufour and two Arts and Technology graduate students will discuss how they used choreography and lighting to interpret movement in their experimental interactive sound installation, Acoustic Shadows, an Exploration of the Sense of Space.
Acoustic Shadows is an audiovisual immersive and interactive installation that depicts Orpheus surrounded by shadows of the underworld consumed by the shadow of his wife, Eurydice.
The Dufours and the cast and production team will discuss the immersive qualities in the work and the use of choreography and lighting to interpret movement in space. They are interested in exposing the question of unity and identity of a work of art, as it integrates multiple individual talents into a single expression.
The discussion will take place Thursday, May 17, 6:30-8:30 p.m. in The Center for Creative Connections Theater at the Dallas Museum of Art.
The School of Arts and Humanities opens its spring season by examining the relationship between sound and art with the mixed-media exhibit Sonic Architectonic.
Curated by visual arts faculty member Lorraine Tady, the exhibit features both local and national artists who work directly with noise or frequency, examining what is heard or felt through sound waves, and some who work with images that suggest sound. Other artists in the exhibit anticipate our relationship to sound by addressing our expectations and cognitive reflexes.
“In contemporary art, sound is a medium used as a separate tool, or is intertwined with other mediums,” said Tady. “Some artists infuse their own open, hybrid visual forms and multimedia explorations with sound. This exhibit considers these approaches and more.”
Artists utilizing real sound with their visual works or as their artwork include Jill Auckenthaler, who, in collaboration with Sarah Phillips, will display What My Schedule Sounds Like. The work is both an instrumental score for an atonal sound piece and a watercolor and graphite work on paper.
Brad Tucker is creating Bagdad Bass Club, an interactive sound and object installation that combines videotaped music performance, customized audio equipment, handmade plastic records, ambient music and thumping intermittent bass sounds.
Dr. Frank Dufour, assistant professor of sound design at UT Dallas, and adjunct art professor Stephen Lapthisophon, will, in separate works of art, direct sound into the gallery to inhabit and transform the architectural space of the gallery. Lapthisophon interprets Karl Marx through a disembodied voice reminiscent of German lieder. Dufour collaborates with David Searcy and Nancy Rebal to create an interactive soundscape alluding to world peace.
Paul Slocum offers his iPhone/iPad/iPod Touch app Magic Carpet. Visitors are invited to install the free app “carpet lite” to experience the hypnotic and meditative graphics in synchronization with the app’s generative music synthesizer.
Artists who will study sound in various visual ways using painting, drawing, and sculpture include John Pomara, professor of visual arts in the School of Arts and Humanities. Included are computer ink jet drawings by Robert Ortega, who is interested in patterns and “how to graphically relate light wavelength to audio frequency,” and Diane Fitch’s realist paintings of casual living room musicians.
The show opens with a reception on Friday, Jan. 27 from 6:30 p.m. – 8:30 p.m. in the Visual Arts Building. Visiting artist Brad Tucker will share a lecture of his work Friday Jan. 27at 10 a.m. in AS 1.116. The group exhibit will on display until Feb. 18, 2012.
Using the ancient myth of Orpheus, video projection and three-dimensional sound, a UT Dallas professor is examining the relationship of sound to perception of movement in Acoustic Shadows, an Exploration of the Sense of Space.
Acoustic Shadows is an audio-visual immersive and interactive installation that depicts Orpheus surrounded by shadows of the underworld consumed by the shadow of his wife, Eurydice.
In the myth, the gifted and musical Orpheus travels to the underworld after his wife dies to beg Hades to allow his wife to return to earth. After violating a condition made by Hades, Orpheus loses his wife forever.
Dr. Frank Dufour worked on the project with his wife, Kristin Lee Dufour, a creative art director and international consultant for visual communications.
“The viewer is enveloped in a multisensory, reactive system that actually ‘listens’ for changes in the environment generated by your presence and movement,” said Dufour, assistant professor of sound design in UT Dallas’ Arts and Technology program.
“This results in noticeable changes to the sound and projected images. Your body reflects and absorbs sound waves to create the auditory manifestation or form of silent movement, which, in this context, is termed ‘Acoustic Shadows.’”
The exhibit is currently on display in The Center for the Creative Connections at the Dallas Museum of Art until April 2012. Acoustic Shadows made its debut at The Vasarely Foundation in Aix-en-Provence, France, earlier this year.
More about Dr. Dufour’s research is available in SoundEffects, a journal on sound and sound experience.
‘Acoustic Shadows’ Installation Explores Movement of Silent Objects
With a new exhibition in France, Dr. Frank Dufour, assistant professor of sound design in UT Dallas’ Arts and Technology (ATEC) program, has officially gone global.
“Acoustic Shadows: An Exploration of Sense of Space” is an experimental interactive sound installation Dufour created alongside wife Kristin Lee through the entity Agency 5970. The piece is currently on display at The Vasarely Foundation in Aix-en-Provence, France. Dufour worked on the sound and music aspects, while his wife tackled the visual side.
Two ATEC graduate students have been working with the Dufours: PhD student Sherri Segovia as choreographer and graduate student Djakhangir Zakhidov as videographer.
Of the installation, Dufour said, “It is an immersive audiovisual environment that senses and responds to the presence and movements of the spectators on the basis of the concept of ‘Acoustic Shadows.’ ”
Dufour describes the latter as “the auditory perception of the movements of silent objects and bodies by means of the changes they cast on a background sound.”
The installation was designed to exemplify the phenomenon and make it known to a wider audience.
“Exhibiting in France is great, but our ultimate goal is to present this work here in the U.S.,” Dufour acknowledged.
Dr. Dennis Kratz, dean of the School of Arts and Humanities at UT Dallas, said, “Professor Dufour’s ingenious and vital contribution to the marriage of technology and art is its focus on the artistic possibilities of sound in not only its most sophisticated, but also its most basic aspects.”
On April 15, 2011, French new media philosopher Bernard Stiegler stepped into the Arts and Technology building to a room jam-packed with professionals, educators and students awaiting his arrival.
Thanks to Stiegler’s long time friendship with sound design pro Professor Frank Dufour, he happened to be passing through Dallas and agreed to give a talk entitled Forming and Deforming Attention.
The talk centered on the discussion of the importance of education in early development of attentional forms.
Dr. Stiegler is currently the Director of the Georges Pompidou Institute of Research and Innovation in Paris, which aims to anticipate changes in human behavior brought about by the evolution of technology.
He also holds an assistant professorship at the Goldsmith College in London and at the University of Compiegne.
ATEC Project Makes Student’s Composition an Integral Part of Math Video Game
Can music soothe your soul, or at least ease the pain of calculus? Eddie Healy thinks so.
The doctoral humanities student has composed the score for a video game being developed by UT Dallas Arts and Technology (ATEC) students.
Spearheaded by Assistant Professor Monica Evans, the game designers sought guitar music for their latest game, “The Digital Calculus Coach.” After hearing Healy play at a school concert, Evans knew she had found her musician.
Healy received his bachelor’s degree in classical guitar performance at the University of North Texas. He then completed a master of music degree at Southern Methodist University. He was awarded a scholarship from the Dallas Federation of Music Clubs, and was the recipient of the Alice Jones-Berding scholarship in 1995. Even with all this experience, Dr. Evans’ request was uncharted territory.
Healy says, “I have been asked to compose wedding music, a piece to honor the victims of Hurricane Katrina, a podcast theme by the UT Dallas Communications department, and even a middle school alma mater. But this was a first for me.”
Evans and her team wanted the music to bolster the fun, whimsical character they intended for the educational game. Despite the fact that most of Healy’s academic and professional experience had been in classical guitar, he also drew on his limited exposure to jazz music to ensure that the game remain both educational and fun. He set the music for two guitars and recorded it last week with an ATEC engineer.
“For ‘Digital Calculus Coach’ to be successful, the game world needs to be as engaging as possible,” Evans said. “One of our goals was to include dynamic, inspiring, and memorable music, rather than making yet another simple, forgettable loop of techno-inspired sounds. What Eddie has done for us is make our game aurally stand out, and bring the game world to life with hummable themes – there isn’t an educational game out there that sounds like ours.”
Healy admits the ATEC team has spoken to him about composing for future projects, and he is excited about stretching his compositional legs in the gaming industry. “I teach music at several different institutions (UT Dallas, Eastfield College, the Collin County Community College District and the Gray School of Music), and I want to keep teaching, playing and composing music.”