California Institute of the Arts :: EXPO 2021
This project connects RunwayML, Processing, MaxMsp and TouchDesigner to create an audio and visual rich performance.
The audio and visuals are all triggered by the text typed into the program. As the text is randomly typed into the program by the audience, RunwayML's AttnGAN takes those words to create a collaged image. Depending on the number of words typed in, Processing will grab that same amount of words and grab the colour values of random points on collaged image. These colour values is then converted into MIDI notes and notated onto a readable score the Cellist can play in real-time. This is updated every time a new text is submitted into the program.
As the Cellist improvises off of the notes that is given to her, the MaxMsp patch will pitch detect the 36 possible notes she is playing (ranging from C2-B4) and trigger a TouchDesigner particle system to change shapes, colour and positions according to the different pitch detected notes.
hover over each icon to learn more about the project process
A demo video showing how I used the text input of RunwayML's AttnGAN to create an
Ai generated image. I took random coordinates of the generated image (via Processing),
added together the RGB colour values to convert them to MIDI values. In MAXMSP, I
turn the MIDI values into notes on a score for the ease of the performer.
A demo video showing how each string' affects the TouchDesigner particle system.
By using OSC between programs, each string triggers a change in the particle system's
parameters. The G string of the cello triggers a burst of lines, the D string triggers the
shape of the particles, the A string triggers alternated between 4 colours, E4 (on the A string)
triggers an amplitude change in the shape.
Although only briefly demonstrated in this video, the pitch detected notes also
moves the center of the particle system depending on if the pervious note and the
current note being played has a Major Second or Perfect Fifth relation to each other.
The text typed into this performance is an excerpt from the community poem "Today, I Am A Witness To Change" .
Today
I'm frozen. Terrified.
I cannot hide
This skin. This hair. These eyes.
I see the punch, punch, punch of a community at war.
Today I am a witness. "
acknowledgements
PRODUCER
Michaella Moon
CELLIST
Michaella Moon
TOUCHDESIGNER ASSISTANCE
Mike Leisz
AI ASSISTANCE
Christine Meinders
feedback