TreeSense: experiencer and the inside view.
TreeSense
How could we bring people to truly empathize with the natural world, facing devastating but ever distant problems such as glaciers melting in the Arctic, water shortages in California, or the deforestation of the Amazon? What would change if an audience could be transported into the center of the story— if one could step into the shoes of another living being?
TreeSense is sensory VR system that transforms you into another life form, a tree.
With arms turning into branches and the body into a trunk, the participant experiences being a seedling rising through the dirt, sprouting branches and growing to full size, until finally witnessing its fate firsthand. By tapping into various senses, TreeSense delivers a vivid body illusion of being another identity to create a personal connection with it. Our research looks into novel sensations such as electronic muscle stimulation, as well as a comprehensive sensory alteration through visual, sound, scent, vibration, temperature and wind, which unlocks a higher level of realism. As an audience member exclaimed, “you know it’s not real but your body really believes it!”
Key collaborator: Yedan Qian
Be the center of the story
Now with Virtual Reality, we gain the opportunity to create experiences that are much more participatory and immersive. What if we could let audiences directly experience empathy for problems that are outside their daily lives? In other words, what if we could shift the perspective of audiences and let them be the center, to live the story and feel the experience firsthand? Would that embodied experience enable audiences to relate to remote problems emotionally and immediately?
Experience with Your Own Body
The beauty of allowing people to be part of a story is that it makes an experience unique to each person each time. In experimental theater, for example, artists blur the boundary of the reality and the play and morph viewers into actors to create the unsettling illusion of “what if I were ... ”. But the embodiment we want to achieve with TreeSense is much more visceral and immersive.
Unlike conventional storytelling mediums, such as films, that are limited to only sight and hearing, we tap into more senses to set up the mood for a immediate mindset change. The body image and body schema inside our brains are plastic and variable. Through a systematic alteration of sensory stimuli, such as vision, touch, motor control and proprioception, the brain can inhabit a body dramatically different from ours, such as a tree. And this body illusion leads to a more direct, personal and emotional connection with the new identity that we embodied.
Our Research
Homunculus Flexibility and Body Ownership Illusion
To what extent can humans accept body distortion and obtain a non-humanoid avatar body? People usually think the body image inside the brain is static. But many phenomenon such as phantom limb and rubber hand illusion prove the existence of this neural plasticity in nature. Lanier (2006) postulates the concept of Homuncular Flexibility that the homunculus—an approximate mapping of the human body in the cortex—is capable of inhabiting novel bodies and control them. This flexibility of owning a new body image and schema unleashed endless possibilities to experience being another life form and ultimately building strong personal connections with the new identity.
Synchronized Sensory Alteration System
In TreeSense, the strong body ownership illusion is induced through synchronized visual, proprioceptive and tactile inputs. We use the Unity3D game engine as the center hub of all the inputs and outputs. The motions and gestures of the head, forearms, and fingers are tracked through infrared light by Leap Motion. Then, we send signals from Unity to Max/MSP to control all the physical elements, such as haptics, heat, wind and scents. Also, the system embeds AI in the virtual animals that interact with the tree. Thus, they behave differently based on the user’s movements, which further makes the experience unique for each person every time.
Electronic Muscle Stimulation for Native Tactile Feedback
In order to generate novel tactile sensations that does not exist in real life, we designed a series of EMS signals by varying combinations of pulse amplitude, pulse width, frequency of the current and the location of the electrodes. The intensity is calibrated per-user with a potentiometer and remains fixed once calibrated. Throughout the experience, various EMS sensations are triggered by the pre-defined hand gestures, timeline events and the interactions with virtual objects. The audience can feel the branches sprouting, touching the leaves and interacting with birds, caterpillars, etc.
Sensory System
Comprehensive sensory experience system
We extended the experience into a hyper-realistic whole body haptic feedback using Subpac, a pair of customized vibration oversleeves with six local points, and a vibrating floor powered by four based transducers. With multi-track bass audios for each part of the body, the audience could feel the thunder, disturbance of the forest fire, as well as a bird landing on the branch. There are also additional physical elements including an air mover for a breath of wind and heaters as the final fire threat. The whole tactile experience is controlled by Max/MSP and Arduino, while communicating with the Unreal engine through OSC. The physical experience was precisely counted to sync with visual experience inside the oculus. We went through various iterations to match the virtual visual details with the intensity, texture and timing of the physical experience.
Awards and Exhibitions
Further Development
Tree, rendered by Jakob Steensen
We collaborated with film directors Milica Zec and Winslow Porter, integrating our research and sensory system as the ground for the VR film Tree.
