Fluid Soundscapes: Touching Kandinsky
Exhibition: Harvard XR 2024 showcase
Exhibition: Innovative Horizons: A Showcase of Design Excellence
Exhibition: Dawn of the Century: Avant-Garde Art from the Collection of the State Russian Museum
Overview
“Fluid Soundscapes: Touching Kandinsky” is an interactive audiovisual installation that harnesses digital technology to envelop audiences in the synesthetic artistry of Kandinsky. This interactive piece allows spectators to engage directly with Kandinsky’s works through gesture interaction technology, creating a bridge between classical art and contemporary digital expression.
Interaction Architecture
We utilize Leap Motion to track the hand movements of users and determine if their hands are within any predefined shape areas. When a match is identified, corresponding visual effects are triggered along with sound output, creating a seamless interaction between human gesture and digital response.
Leap Motion Technology
Advanced hand tracking with millimeter precision, enabling natural gesture interaction without physical contact.
- Real-time hand position detection
- Gesture recognition algorithms
- Low-latency response system
Interactive Zones
Predefined shape areas mapped to specific visual and audio responses, creating an intuitive interaction space.
- Dynamic zone mapping
- Multi-touch support
- Responsive feedback system
Sound Design
The audio experience combines AI-generated background music with real-time sound synthesis, creating a dynamic soundscape that responds to user interaction.
AI-Generated Music
MusicGen creates the foundational ambient layer, providing a continuous musical backdrop that adapts to the installation's mood.
Real-Time Synthesis
Max/MSP generates distinct sounds for each interactive element, creating a responsive audio environment.
- Shape-specific sound mapping
- Dynamic volume control
- Spatial audio positioning
Visual Effects
TouchDesigner brings Kandinsky’s paintings to life with dynamic animations and real-time visual feedback, creating an immersive visual experience.
Dynamic Animations
Shapes respond to user interaction with smooth scaling and rotation effects, creating engaging visual feedback.
- 2x scale transformation
- Smooth rotation animations
- Color transition effects
Real-Time Rendering
TouchDesigner provides high-performance real-time graphics processing, ensuring smooth visual responses.
- 60fps rendering
- Low-latency response
- High-resolution output
Technical Implementation
The project integrates multiple cutting-edge technologies to create a seamless interactive experience.
Leap Motion
Hand gesture tracking and interaction detection
TouchDesigner
Real-time visual effects and animation
Max/MSP
Audio synthesis and processing
MusicGen
AI-generated background music
Reflection
This is the first time I have seriously engaged with new media art. Using TouchDesigner and Max/MSP has been intriguing, as these tools don’t require extensive coding but still embody programming concepts. I’m pleased to discover that the skills I acquired in college can be used to create artistic works, not just technical projects.
This project is still in its early stages, and there is much room for improvement. For instance, the sound currently utilizes simple chords and matrix music, which could be developed into something more sophisticated. Nevertheless, this has been a great beginning for me to explore the intersection of art and technology.
The installation explores the interplay of technology with the essence of artistic spirit. It serves as a modern interpretation of cross-media expression, enriching sensory experiences and offering new perspectives on how we can interact with classical art through contemporary digital means.