ARTWORK EXPERIENCE BALL
Target Problem
An estimated 500,000 Canadians are blind or partially sighted according to the CNIB. The ability for those who are blind and have low vision to experience art is mainly limited to audio descriptions and raised line tactile displays that are all too uncommon in the art world. I believe there is potential to transform a traditionally visual experience into a multi-sensory one to create a more immersive art experience for everybody.
The inclusion of visual, auditory, and haptic perceptions allows more people to participate in the artist’s creation and in fact brings them deeper into the art than ever before possible. The ability to see, touch, and hear a complex world created specifically to evoke emotion, excitement, an tell a story could be dramatically transformed by involving more modalities.
Preliminary Design
Art is experienced individually based on our lived experiences, our access, our education, our willingness to think openly, and so much more. These are particular elements that are very hard to translate to someone that cannot see the art. How does one connect with these feelings and place it in their mind as they are viewing an art piece? An audio description or raised lines tof flat plaque creates a very static understanding of what is being displayed.
The first exploration into cross-sensory design of artwork was finding a way to showcase a photograph in physical form as a three-dimensional model. A blind user would feel their way through a scene just as a sighted user could see the layers and subjects of the photograph. Creating an almost sculpture like version of a photograph.
Artwork for cross-sensory representation
Man riding bike across an intersection in Osaka with diagonal line as people wait on the opposite side
Notes on how I wanted to explore a three-demensional representation
- Use models of people lined up on the other side of the street. Different heights.
-When the user touches the bike, the wheel can spin. There is sound of a jangling bike, a bell ringing.
- Sounds of the city around. People talking, giving a location for the listener. Clicking of a camera shutter.
- Different textures for the lines on the street, wheels for the bicycle. A piece of fabric for the rider.
- Attempt to showcase Black & White.
- Have the stand going upright so the people in the back are higher to represent the layers.
- Place it in a frame to represent the display of artwork
The design affords the ability to touch; to touch a typically two-dimensional experience that is only possible for those with sight. This design affords the ability to hear; to hear the environment that an artwork sits in.
The goal is to incorporate visual, auditory, and haptic outputs to best serve the experience of the artwork to everybody. Combining all perceptions allows the user multiple ways to interact with the artwork and become fully immersed in the experience. This allows artists the ability to integrate all senses when showcasing their work to an audience.
Visual: The viewing of the photograph/ artwork hanging on the wall. The viewer can see the shapes, colours, textures, subjects, and more within the frame.
Auditory: There will be sound output from the device that corresponds with the audio of the environment in stereo* to surround the user in the experience. The sounds will be chosen and arranged for their specific frequency, pitch, rhythm, tempo, and timbre to create an experience that balances an environment while having some key elements stand out to tell the user where they are. The auditory scene analysis will include several sequential sounds as well as simultaneous sounds to complete an immersive sound experience with key sounds sticking out to catch the users attention.
Haptic: The design is based around the ability to experience the artwork through touch. This will be represented by the user placing their hand on the device and be able to feel three-dimensional output to describe the artwork to the user. This will include brushes across the hand, points, and vibrations.
All of these will be combined to create a bottom- up processing experience for the user, where they are taking in sensory information and then assembling and integrating it.
A successful design is one that brings a different dimension to the artwork and allows more people that would not otherwise be able to experience the work to be a part of the artist’s mindset when creating the work. The user will be able to experience a live visual viewing of the artwork, a recorded audio track, and pre-determined physical haptic feedback. It will be successful if all people are able to take time to appreciate and interact with an artwork This design brings a drastic improvement to the current raised line plaques that we see in art galleries. Therefore, bringing a greater mutual knowledge, higher spatial- topological synchrony, and an increase in temporal synchrony.
Process Work - First Design
Image of my first iteration of a three-demensional photograph
Building the first iteration comprised of several steps in order to best represent a photograph as a three-dimensional model. The first was made quickly in the span of one hour.
Each element in the Osaka photograph was represented by a texture or structure to get the most accurate or best metaphoric response from touch. A frame to house the model, a rough canvas to represent asphalt, smooth paper to imitate the painted lines on the road. While roils of film were used for the bike wheels, while an abstract body shape was made using a soft fabric for the human rider. Almonds of various shapes to personify the pedestrians, and cardboard to outline the cityscape.
This prototype worked well to stage a three-dimensional version of the Osaka photograph, however it did not have a refined detail to truly and accurately represent the artwork.
Model for the bicycle man. Made from a toy set and pipe cleaner
Canvas and cut paper lines glued to inclined box, with pipe cleaner pedestrians
Cutting the cityscape out of a large cardboard
The expansion and evolution of the prototype came after discussing my first iteration to the class and professor which helped to guide me towards adding an auditory output to bring the viewer into the atmosphere of the artwork. Another addition moving forward is the orientation of the housing. This iteration essentially is like a pop-up book, still a flat surface with subjects projecting upward. The next prototype will incorporate a truly three-dimensional shape that showcases the depth I am looking to represent.
First Design
Photographs of the complete first iteration. The structure sits at a 45 degree angle for giving a layered feeling to the design which can benefit the viewing and touching perceptions
A screenshot of the audio file that was put togther with four different tracks to put give an auditory output to best insert the viewer into the enviroment of the photograph.
Using the open discussions and knowledge gained in class, I was able to create a more expanded version of a cross-sensory photograph, using physical representations of the subjects and environment within the photograph. I was able to create a model that used the angle to create layers and different heights for subjects, like the bicycle rider in the front and line it up with the pedestrians in the background. Material use was very important as it helps to represent what the user is feeling through touch.
Bike: Metal bicycle toy with spinning wheels
Rider: Soft linen fabric with a pipe cleaner under body
Ground: Rough canvas for asphalt, and smooth paper for painted lines in road
Pedestrians: Soft pipe cleaner in human shape in various heights and widths
Cityscape: Thick rippling cardboard cut into building shapes towering over pedestrians
These representations each help to form an accurate metaphor of the photograph so that is translated through the sense of touch.
Sounds are played through an external speaker to put the viewer into the environment of the photograph. I did this by downloading the audio track from a video of a person walking around Osaka, Japan to fit the city sounds perfectly. I added sounds from a jangling bike, riding down the road, and bell chimes earcon to indicate a bike riding past. To help place the viewer into the environment of the photograph, I added camera shutter earcon as if they were right there beside when the photograph was being taken. Certain sounds were used to be more alerting as they happen less often, but stand out more, but all work together to form one auditory stream.
Translating the photograph through auditory, haptic and originally visual was important to help immerse the viewer in the idea of what the photograph represents, and more importantly allows for more affordances so that almost everyone can interact with the artwork.
Client Feedback
Having discussed the evolution of my idea of a multi-sensory artwork with Dr. Coppin, Robert Ingino, and the class; we had many ideas of how to improve the current system of accessible artwork through the use of touch, sound, and sight. Could a blind person get to experience the artwork with the help of the Micro:bit computer and accessories to increase the interaction? I do believe so.
Haptic
My idea that I discussed with the class was to have a flat surface similar to the picture frame in my early first iteration that combines a soft material screen that a user could place their palm on and feel shapes and movements of a tactile display underneath. This would be able to move around and outline or mimic shapes on the artwork so the user could follow along to understand the artwork. This would make every movement unique to that artwork and feel as if it were more alive then a stationary raised line display. Objects could be represented with movements, points, blocks, and more.
Feedback
While the idea was received well, Dr. Coppin suggested the idea of making the item small, almost handheld. This led to the idea of making the shape a ball rather than a flat surface. The user would be able to comfortably grip the ball and be able to feel all the movements inside within the palm and fingers and not having to search a wide display.
This then lead to the discussion of how to arrange servo motors connected to a Micro:bit to best showcase the movements and attach arms that represent the shape of the intended object. Dr. Coppin suggested an array of three servos arranged together on the X, Y, and Z axis to control one “stylus” so that it can trace out shapes. However, I saw this one “stylus” a bit too difficult to arrange accurately and have it draw out pre-determined shapes. With better computer systems and more practice with the technology that would be a wonderfully interactive way of providing haptic art. I decided to use each servo motor individually to create its own movement and represent multiple things at one time. This allowed me to have distinct shapes on each servo so it can represent something independently. This creates a system where each new artwork needs specifically laid out motors and arms, however I think for this prototype it showcases a way to personalize each artwork being displayed. One other addition thanks to Robert was the inclusion of vibration to complete the “D-Box” style haptics. With slight vibration, the user could feel things that wouldn’t be possible before, the passing of cars, and trains.
Sound
The sound work done on my first iteration described the environment quite well, taking sounds right from the neighborhood that my Osaka photograph was taken, as well as incorporating sounds that describe the actions happening around the scene. I wanted to find a way to bring it a little more in depth for the viewer. There was a discussion about how to showcase the sound, directional speakers, headphones, and stereo speakers. Regardless the sound needed to come from multiple directions and not just a flat front facing type of sound.
Feedback
Robert discussed simple stereo sound to bring the pre-recorded sounds that I had to life with simple left and right. While ideally a quadraphonic sound system would be ideal to fully immerse the viewer, this would prove too big a system, especially when my project would be alongside multiple, if not all artworks in a gallery. Robert informed me of several application like Divinci Resolve, Garage Band, Ableton Live, and more where I could create stereo sounds for the artwork environment to situate the viewer in the space.
Process Work - Final Design
Housing
I began working on the final design for my multi-sensory artwork project. Building off the first design iteration while including the use of the Micro:bit and servo motors to create haptic output to represent the subjects in the photograph.
Early iteration of a flat frame style with soft fabric to feel to mechanics on the bottom side.
Found a round material that is soft enough to feel haptic outputs through. A thin rubber toy ball.
Sketches of housing design, locations for feedback, and what servos are needed.
Cut section of the rubber ball in order to fit the servos inside.
Outer housing built from 0.06” styrene with ball glued to the top.
Outer housing with rubber ball and inner tower for attaching servos.
Servo Tower
The housing and general design of the servo movements was completed and all that was left was to code the movements into the three servo motors and glue them to the inner tower and attached the specifically designed arms that represent the subjects within the photograph. The inclusion of this system allows for complete customization to best represent the artwork, this can be done through shapes, materials, direction of movement, and type of movement.
Completed control tower with three servo motors. Controlled with a Micro:bit on three axis.
Three servo arms to represent subjects in the photograph. Cut styrene with clay attached to two to represent different textures
Code for the servo motors through Microsoft MakeCode
Sound
Expansion on the original “Osaka” sound file was necessary to create a higher level of immersion for the viewer. The sound file that viewers will be listening to needed to be adjust to create a fully stereo sound. The multiple tracks intertwine to create an auditory experience that places the listener in the middle of an active Osaka, Japan street where the image was taken. This level of immersion was a must in order to bring the multi-sensory experience of the project well past was is currently offered with a solely visual interaction. This brings the viewer right beside the artist as they were taking the photograph.
The tracks and their stereo panning and volumes are as follows:
Bicycle: The panning moves from left speaker to right speaker each of the three times it passes. Volume is high to hear details
Camera Shutter: Exclusively in the left speaker. Volume is high to be alerting and to stand out
Osaka Environment: Sound present in both speakers. Volume is medium- high as it was in the environment
Bicycle Bell Ring: Sound exclusively from the right speaker. Volume is medium- high to be somewhat alerting and not overpowering
Japan Bus stop: Sound rises from the left speaker to both and finishes in the right speaker as if it is passing by. Volume is low to act as atmosphere.
Using Garageband to lay tracks with overlay of stereo panning
Final Design
Demonstration of the ideal display. Artwork experience ball on a pedestal in front of the artwork, with headphones attached for audio output.
Augmented reality display of potential pedestal with artwork
Layout of servo tower and the movements and representations of each.
All parts laid out. Headphones, Housing, Servo tower, Microbit, and Battery
Artwork experience ball package
The Artwork Experience Ball creates a simple to understand design, with internal haptic outputs that showcase the art in a way never previously experienced; art in the palm of your hand. Combining the audio output through a specifically designed audio track that is unique to the place being displayed is meant to completely immerse the viewer into the enviroment right beside the artist.
Creating accessbility for blind art lovers who would normally be restricted to a text-to-speech or raised line display brings up the potential to grow the amount of viewers to art galleries and museums. Being able to see, touch, and hear an artwork brings the participation of the viewer to levels like never before. This gives the potential for artist to re-think how they build their art, creating a multi-sensory experience allows for new expression of the artist’s intent.
Future
The expansion of this project could be quite wide ranging, designed for art, I see potential in making the project more integrated with galleries and museums with beautifully crafted stands, they could be made of woods, recycled materials, or they could be built right into the wall beside the artwork, just as the artwork text would be.
I believe with further development into refreshable tactile displays, new nano materials with built in sensors, and exploration into haptic interfaces so that the mechanical movements can be even more refined and deatiled.
My exploration into multi-sensory displays has informed me of the lack of affordances that our devices currently have. Often being built for reduced cost, or sleek design while coming at the expense of people who are blind or partially sighted.
The artwork experience ball is a small step in the direction of using hand built prototypes and combining the use of easily accessible computer controllers and motors to explore the possibilities of making art accessible for the blind.