New Google tool lets you experience synesthesia
When Russian artist Vasily Kandinsky was born, the first computer hadn’t even been built yet.
Now, more than 75 years after Kandinsky’s death, an online tool powered by AI is allowing people to experience his paintings the same way he did: as music.
Hear Colors, See Sounds
Kandinsky had a neurological condition called synesthesia, in which the stimulation of one sense triggers another.
In Kandinsky’s case, seeing certain colors caused him to hear certain sounds and feel certain emotions. Yellow sounded like trumpets and felt cheeky, for example, while red sounded like a violin and felt restless.
“When Kandinsky painted, two senses worked systematically together: hearing and sight,” Serge Lasvignes, president of the Centre Pompidou, an art museum, library, and research center in Paris, wrote in a Google blog post.
“Colors and shapes translated into sounds, harmonies, and vibrations made up lines and patterns,” he continued.
Connecting colors and sounds is just one of as many as 80 different ways to experience synesthesia, though. Other people associate letters with colors or words with flavors.
No one knows for sure how many people have synesthesia — depending on your source, it could be one in 2,000 people, one in 300 people, or (to some extent) everyone — but famous “synesthetes” include musician Kanye West, author Vladimir Nabokov, and composer Richard Wagner.
For people who don’t experience synesthesia naturally, the concept can be hard to understand — how can a color have a sound? — so Google Arts & Culture and the Centre Pompidou collaborated on “Play a Kandinsky.”
This online tool lets users hear what Kandinsky might have heard when he painted his masterpiece, Yellow-Red-Blue, in 1925 — and all of the music in the experiment was written by AI.
Recreating a Masterpiece
The first step to creating “Play a Kandinsky” was understanding exactly how Kandinsky experienced synesthesia, so the researchers studied his writings about his experiences.
Next, they needed the music.
By inviting the machine to express itself, the music kind of emerged by itself.
Antoine Bertin
Experimental musicians NSDOS and Antoine Bertin trained a Google AI to compose music for parts of Yellow-Red-Blue based on what they’d learned about Kandinsky’s synesthesia and the phonographs in the artist’s personal collection.
“We trained Google Transformer, a neural network that was developed by the Google Magenta team, on music from Kandinsky’s time and invited this machine to generate new scores to form a new perspective on what the painting may sound like from today’s point of view,” Bertin said.
“By inviting the machine to express itself, the music kind of emerged by itself,” he added. “And it felt that, as artists, we were more the intermediary from Kandinsky’s time to today rather than the interpreters of the theory ourselves, and that was very exciting.”
Your Chance to Experience Synesthesia
Now, anyone who wants to experience synesthesia the way Kandinsky might have can visit the “Play a Kandinsky” experiment, which is broken into four chapters.
The first chapter introduces the concept of synesthesia and how Kandinsky experienced it.
In the second, users click on one of three shapes — a yellow rectangle, a red cross, or a blue circle — to hear a snippet of music similar to what Kandinsky might have heard and felt when he looked at that shape.
When Kandinsky painted, two senses worked systematically together: hearing and sight.
Serge Lasvignes
The third chapter is meant to let users experience synesthesia the way Kandinsky might have while looking at Yellow-Red-Blue.
Google broke the painting down into seven regions, and by clicking on one, users trigger a musical movement based on the shapes and colors in that region.
The region with a yellow rectangle at its center, for example, triggers the “Yellow Movement,” described as “simple and bright shapes which echo earthly madness and passionate joy.”
In the final chapter, users make their own musical movement based on Yellow-Red-Blue.
They start by choosing two emotions from a list, and then listen to those emotions conveyed through the painting. By clicking on various parts of the painting, they can add or subtract additional sounds.
Whether “Play a Kandinsky” achieves its goal of allowing users to experience synesthesia the way the artist might have is hard to say.
However, it does make the abstract concept easier to grasp, and bringing AI into the experiment could entice a new audience to explore the works of one of the greatest artists of a time before digital technology.
We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].
