Limits...
Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution.

Hertz U, Amedi A - Cereb. Cortex (2014)

Bottom Line: Secondly, associative areas changed their sensory response profile from strongest response for visual to that for auditory.Consistent features were also found in the sensory dominance in sensory areas and audiovisual convergence in associative area Middle Temporal Gyrus.These 2 factors allow for both stability and a fast, dynamic tuning of the system when required.

View Article: PubMed Central - PubMed

Affiliation: Department of Medical Neurobiology, Institute for Medical Research Israel-Canada (IMRIC), Hadassah Medical School, Hebrew University of Jerusalem, Jerusalem 91220, Israel Interdisciplinary Center for Neural Computation, The Edmond & Lily Safra Center for Brain Sciences (ELSC), Hebrew University of Jerusalem, Jerusalem 91905, Israel.

No MeSH data available.


Order and design of experimental sessions. (A) Order of experiments—this study consisted of 3 sessions of data acquisition, interspersed by a short (1 h) learning session outside the scanner. (B) Passive paradigm—visual images and auditory soundscapes were presented to the subjects in a semioverlapped manner, while subjects were asked to maintain fixation on a cross in the middle of the screen. Each visual block contained 3 images, each presented for 4 s, totaling 12 s per block. Each auditory block contained 3 soundscapes, each was 1 s long and repeated 4 times, totaling 12 s for block. Auditory and visual blocks had different representation rates, as 3 visual blocks were presented for every 2 auditory blocks, totally auditory blocks were repeated 14 times throughout the experiment, and visual blocks were repeated 21 times. This experiment was carried out once before the learning period (Passive–Pre), and once after learning (Passive–Post). (C) Visual images and their translation to soundscapes via SSA. The soundscapes used throughout the experiments represented 4 visual images. Their translation to sound is represented by the time course of the sound (blue line) and by spectrograms of the soundscape (y-axis depicts frequency, x-axis depicts time, red colors are high energy and blues are low energy). This translation is not trivial and was not deciphered before learning SSA. Shape information can only be extracted from the soundscapes after learning (spectrograms reveal the shape information encoded in the soundscapes). (D) Active Post–Audiovisual plus detection. The subjects were instructed to press a button when they perceived a combination of a vertical line and a horizontal line, each from a different modality, either auditory or visual, that formed a multisensory plus (+) sign (demonstrated in the green dashed boxes). Auditory soundscape blocks and visual image blocks were presented to the subjects in a semioverlapped manner. Auditory blocks lasted 12 s, containing four 1 s soundscapes, each repeated 3 times. Visual blocks lasted 18 s and included 6 images. The auditory blocks were repeated 20 times throughout the experiment, and the visual blocks repeated 15 times. “Plus” events occurred 10 times throughout the experiment (these events are marked in green).
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4494022&req=5

BHU010F1: Order and design of experimental sessions. (A) Order of experiments—this study consisted of 3 sessions of data acquisition, interspersed by a short (1 h) learning session outside the scanner. (B) Passive paradigm—visual images and auditory soundscapes were presented to the subjects in a semioverlapped manner, while subjects were asked to maintain fixation on a cross in the middle of the screen. Each visual block contained 3 images, each presented for 4 s, totaling 12 s per block. Each auditory block contained 3 soundscapes, each was 1 s long and repeated 4 times, totaling 12 s for block. Auditory and visual blocks had different representation rates, as 3 visual blocks were presented for every 2 auditory blocks, totally auditory blocks were repeated 14 times throughout the experiment, and visual blocks were repeated 21 times. This experiment was carried out once before the learning period (Passive–Pre), and once after learning (Passive–Post). (C) Visual images and their translation to soundscapes via SSA. The soundscapes used throughout the experiments represented 4 visual images. Their translation to sound is represented by the time course of the sound (blue line) and by spectrograms of the soundscape (y-axis depicts frequency, x-axis depicts time, red colors are high energy and blues are low energy). This translation is not trivial and was not deciphered before learning SSA. Shape information can only be extracted from the soundscapes after learning (spectrograms reveal the shape information encoded in the soundscapes). (D) Active Post–Audiovisual plus detection. The subjects were instructed to press a button when they perceived a combination of a vertical line and a horizontal line, each from a different modality, either auditory or visual, that formed a multisensory plus (+) sign (demonstrated in the green dashed boxes). Auditory soundscape blocks and visual image blocks were presented to the subjects in a semioverlapped manner. Auditory blocks lasted 12 s, containing four 1 s soundscapes, each repeated 3 times. Visual blocks lasted 18 s and included 6 images. The auditory blocks were repeated 20 times throughout the experiment, and the visual blocks repeated 15 times. “Plus” events occurred 10 times throughout the experiment (these events are marked in green).

Mentions: In this study, visual images were used along with their SSA translation to auditory soundscapes (see Fig. 1C). An SSA used in this study was the “vOICe,” developed by Meijer (1992). The functional basis of this visual-to-auditory transformation lies in a spectrographic sound synthesis from any input image. Time and stereo panning constitute the horizontal axis of the sound representation of an image, tone frequency makes up the vertical axis, and volume corresponds to pixel brightness. Each auditory soundscape displays an image for 1 s as it sweeps from the left side of the image to the right side and usually requires a few repetitions to reconstruct a frame and identify objects. This imposes a serial acquisition of the visual space that differs from the parallel nature of visual acquisition of information from images in which the entire image is available at once. To make the visual images similar to the auditory soundscapes, we did not present the entire image at once, but rather had a mask sweep across the image from left to right for 1 s, revealing the image a little at a time, similar to a vector spotlight sweeping across an image in the soundscapes.Figure 1.


Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution.

Hertz U, Amedi A - Cereb. Cortex (2014)

Order and design of experimental sessions. (A) Order of experiments—this study consisted of 3 sessions of data acquisition, interspersed by a short (1 h) learning session outside the scanner. (B) Passive paradigm—visual images and auditory soundscapes were presented to the subjects in a semioverlapped manner, while subjects were asked to maintain fixation on a cross in the middle of the screen. Each visual block contained 3 images, each presented for 4 s, totaling 12 s per block. Each auditory block contained 3 soundscapes, each was 1 s long and repeated 4 times, totaling 12 s for block. Auditory and visual blocks had different representation rates, as 3 visual blocks were presented for every 2 auditory blocks, totally auditory blocks were repeated 14 times throughout the experiment, and visual blocks were repeated 21 times. This experiment was carried out once before the learning period (Passive–Pre), and once after learning (Passive–Post). (C) Visual images and their translation to soundscapes via SSA. The soundscapes used throughout the experiments represented 4 visual images. Their translation to sound is represented by the time course of the sound (blue line) and by spectrograms of the soundscape (y-axis depicts frequency, x-axis depicts time, red colors are high energy and blues are low energy). This translation is not trivial and was not deciphered before learning SSA. Shape information can only be extracted from the soundscapes after learning (spectrograms reveal the shape information encoded in the soundscapes). (D) Active Post–Audiovisual plus detection. The subjects were instructed to press a button when they perceived a combination of a vertical line and a horizontal line, each from a different modality, either auditory or visual, that formed a multisensory plus (+) sign (demonstrated in the green dashed boxes). Auditory soundscape blocks and visual image blocks were presented to the subjects in a semioverlapped manner. Auditory blocks lasted 12 s, containing four 1 s soundscapes, each repeated 3 times. Visual blocks lasted 18 s and included 6 images. The auditory blocks were repeated 20 times throughout the experiment, and the visual blocks repeated 15 times. “Plus” events occurred 10 times throughout the experiment (these events are marked in green).
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4494022&req=5

BHU010F1: Order and design of experimental sessions. (A) Order of experiments—this study consisted of 3 sessions of data acquisition, interspersed by a short (1 h) learning session outside the scanner. (B) Passive paradigm—visual images and auditory soundscapes were presented to the subjects in a semioverlapped manner, while subjects were asked to maintain fixation on a cross in the middle of the screen. Each visual block contained 3 images, each presented for 4 s, totaling 12 s per block. Each auditory block contained 3 soundscapes, each was 1 s long and repeated 4 times, totaling 12 s for block. Auditory and visual blocks had different representation rates, as 3 visual blocks were presented for every 2 auditory blocks, totally auditory blocks were repeated 14 times throughout the experiment, and visual blocks were repeated 21 times. This experiment was carried out once before the learning period (Passive–Pre), and once after learning (Passive–Post). (C) Visual images and their translation to soundscapes via SSA. The soundscapes used throughout the experiments represented 4 visual images. Their translation to sound is represented by the time course of the sound (blue line) and by spectrograms of the soundscape (y-axis depicts frequency, x-axis depicts time, red colors are high energy and blues are low energy). This translation is not trivial and was not deciphered before learning SSA. Shape information can only be extracted from the soundscapes after learning (spectrograms reveal the shape information encoded in the soundscapes). (D) Active Post–Audiovisual plus detection. The subjects were instructed to press a button when they perceived a combination of a vertical line and a horizontal line, each from a different modality, either auditory or visual, that formed a multisensory plus (+) sign (demonstrated in the green dashed boxes). Auditory soundscape blocks and visual image blocks were presented to the subjects in a semioverlapped manner. Auditory blocks lasted 12 s, containing four 1 s soundscapes, each repeated 3 times. Visual blocks lasted 18 s and included 6 images. The auditory blocks were repeated 20 times throughout the experiment, and the visual blocks repeated 15 times. “Plus” events occurred 10 times throughout the experiment (these events are marked in green).
Mentions: In this study, visual images were used along with their SSA translation to auditory soundscapes (see Fig. 1C). An SSA used in this study was the “vOICe,” developed by Meijer (1992). The functional basis of this visual-to-auditory transformation lies in a spectrographic sound synthesis from any input image. Time and stereo panning constitute the horizontal axis of the sound representation of an image, tone frequency makes up the vertical axis, and volume corresponds to pixel brightness. Each auditory soundscape displays an image for 1 s as it sweeps from the left side of the image to the right side and usually requires a few repetitions to reconstruct a frame and identify objects. This imposes a serial acquisition of the visual space that differs from the parallel nature of visual acquisition of information from images in which the entire image is available at once. To make the visual images similar to the auditory soundscapes, we did not present the entire image at once, but rather had a mask sweep across the image from left to right for 1 s, revealing the image a little at a time, similar to a vector spotlight sweeping across an image in the soundscapes.Figure 1.

Bottom Line: Secondly, associative areas changed their sensory response profile from strongest response for visual to that for auditory.Consistent features were also found in the sensory dominance in sensory areas and audiovisual convergence in associative area Middle Temporal Gyrus.These 2 factors allow for both stability and a fast, dynamic tuning of the system when required.

View Article: PubMed Central - PubMed

Affiliation: Department of Medical Neurobiology, Institute for Medical Research Israel-Canada (IMRIC), Hadassah Medical School, Hebrew University of Jerusalem, Jerusalem 91220, Israel Interdisciplinary Center for Neural Computation, The Edmond & Lily Safra Center for Brain Sciences (ELSC), Hebrew University of Jerusalem, Jerusalem 91905, Israel.

No MeSH data available.