Information that is normally accessed through a sensory modality (substituted modality, e.g., vision) is provided by sensory substitution devices (SSDs) through an alternative modality such as hearing or touch (i.e., substituting modality). SSDs usually support disabled users by replacing sensory inputs that have been lost, but they also offer a unique opportunity to study adaptation and flexibility in human perception. Current debates in sensory substitution (SS) literature focus mostly on its neural correlates and behavioural consequences. In particular, studies have demonstrated the neural plasticity of the visual brain regions that are activated by the substituting modality. Participants also adapt to using the devices for a broad spectrum of cognitive tasks that usually require sight. However, little is known about the SS experience. Also, there is no agreement on how the phenomenology of SS should be studied. Here, we offer guidelines for the methodology of studies investigating behavioural adaptation to SS and the effects of this adaptation on the subjective SS experience. We also discuss factors that may influence the results of SS studies: (1) the type of SSD, (2) the effects of training, (3) the role of sensory deprivation, (4) the role of the experimental environment, (5) the role of the tasks participants follow, and (6) the characteristics of the participants. In addition, we propose combining qualitative and quantitative methods and discuss how this should be achieved when studying the neural, behavioural, and experiential consequences of SS.
AbboudS.HanassyS.Levy-TzedekS.MaidenbaumS.AmediA. (2014). EyeMusic: Introducing a ‘visual’ colorful experience for the blind using auditory sensory substitution. Restorative Neurology and Neuroscience, 32, 247–257.
2.
AlbertazziL. (2013). Handbook of experimental phenomenology: Visual perception of shape, space and appearance. Hoboken, NJ: John Wiley & Sons.
3.
AlfaroA.BernabeuÁ.AgullóC.ParraJ.FernándezE. (2015). Hearing colors: An example of brain plasticity. Frontiers in Systems Neuroscience, 9, 56.
ArnoP.De VolderA. G.VanlierdeA.Wanet-DefalqueM. C.StreelE.RobertA.VeraartC. (2001). Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage, 13, 632–645.
6.
ArnoP.VanlierdeA.StreelE.Wanet-DefalqueM. -C.Sanabria-BohorquezS.VeraartC. (2001). Auditory substitution of vision: Pattern recognition by the blind. Applied Cognitive Psychology, 15, 509–519.
7.
AuvrayM.FarinaM. (2014). Patrolling the boundaries of synaesthesia: A critical appraisal of transient and artificially-acquired forms of synaesthetic experiences. In DeroyO. (Ed.), Sensory blendings: New essays on synaesthesia (pp. 19–40). Oxford, UK: Oxford University Press.
8.
AuvrayM.HannetonS. O.ReganJ. K. (2007). Learning to perceive with a visuo-auditory substitution system: Localisation and object recognition with ‘the vOICe’. Perception, 36, 416–430.
9.
AuvrayM.MyinE. (2009). Perception with compensatory devices: From sensory substitution to sensorimotor extension. Cognitive Science, 33, 1036–1058.
Bach-y-RitaP.KaczmarekK. A.TylerM. E.Garcia-LaraM. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note. Journal of Rehabilitation Research and Development, 35, 427–430.
12.
Bach-y-RitaP. W.KercelS. (2003). Sensory substitution and the human–machine interface. Trends in Cognitive Sciences, 7, 541–546.
13.
BlockN. (2003). Tactile sensation via spatial perception. Trends in Cognitive Sciences, 7, 285–286.
14.
BolognaG.DevilleB.PunT. (2009, June22–26). Blind navigation along a sinuous path by means of the See ColOr interface. Paper presented at International Work-Conference on the Interplay Between Natural and Artificial Computation, Santiago de Compostela, Spain, pp. 235–243. Berlin, Germany; Heidelberg, Germany: Springer.
BrownD.MacphersonT.WardJ. (2011). Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device. Perception, 40, 1120–1135.
17.
BrownD. J.ProulxM. J. (2013). Increased signal complexity improves the breadth of generalization in auditory perceptual learning. Neural Plasticity, 2013, 879047.
18.
BrownD. J.ProulxM. J. (2016). Audio–vision substitution for blind individuals: Addressing human information processing capacity limitations. IEEE Journal of Selected Topics in Signal Processing, 10, 924–931.
19.
BuchsG.MaidenbaumS.AmediA. (2014). Obstacle identification and avoidance using the ‘EyeCane’: A tactile sensory substitution device for blind individuals. In AuvrayM.DuriezC. (Eds.), Haptics: Neuroscience, devices, modeling, and applications (pp. 96–103). Berlin, Germany; Heidelberg, Germany: Springer.
20.
CapelleC.TrullemansC.ArnoP.VeraartC. (1998). A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Transactions on Biomedical Engineering, 45, 1279–1293.
21.
ChalmersD. J. (1995). Facing up to the problem of consciousness. Journal of Consciousness Studies, 2, 200–219.
22.
CollignonO.LassondeM.LeporeF.BastienD.VeraartC. (2007). Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects. Cerebral Cortex, 17, 457–465.
23.
Da PosO.BuriganaL. (2014). Qualitative inference rules for perceptual transparency. In AlbertazziL. (Ed.), Handbook of experimental phenomenology: Visual perception of shape, space and appearance (pp. 343–367). Hoboken, NJ: John Wiley & Sons.
DehaeneS. (2014). Consciousness and the brain: Deciphering how the brain codes our thoughts. London, England: Penguin.
26.
DennettD. C. (1996). Facing backwards on the problem of consciousness. Journal of Consciousness Studies, 3, 4–6.
27.
DeroyO.AuvrayM. (2012). Reading the world through the skin and ears: A new perspective on sensory substitution. Frontiers in Psychology, 3, 457.
28.
DeroyO.AuvrayM. (2014). A crossmodal perspective on sensory substitution. In StokesD.MatthenM.BiggsS. (Eds.), Perception and Its Modalities (pp. 327–349) Oxford, UK: Oxford University Press.
29.
De VolderA. G.Catalan-AhumadaM.RobertA.BolA.LabarD.CoppensA.. . . VeraartC. (1999). Changes in occipital cortex activity in early blind humans using a sensory substitution device. Brain Research, 826, 128–134.
30.
EnglanderM. (2012). The interview: Data collection in descriptive phenomenological human scientific research. Journal of Phenomenological Psychology, 43, 13–35.
31.
FarinaM. (2013). Neither touch nor vision: Sensory substitution as artificial synaesthesia?Biology & Philosophy, 28, 639–655.
32.
FroeseT. (2013). Interactively guided introspection is getting science closer to an effective consciousness meter. Consciousness and Cognition, 22, 672–676.
33.
FroeseT.GouldC.BarrettA. (2011). Re-viewing from within: A commentary on first-and second-person methods in the science of consciousness. Constructivist Foundations, 6, 254–269.
34.
FroeseT.GouldC.SethA. K. (2011). Validating and calibrating first-and second-person methods in the science of consciousness. Journal of Consciousness Studies, 18, 38–64.
35.
FroeseT.McGannM.BiggeW.SpiersA.SethA. K. (2012). The enactive torch: A new tool for the science of perception. IEEE Transactions on Haptics, 5, 365–375.
36.
FroeseT.SpiersA.BristolU. (2007). Toward a phenomenological pragmatics of enactive perception. Cognitive Science Research Papers. Brighton, UK: University of Sussex.
37.
GallagherS. (2010). Phenomenology and non-reductionist cognitive science. In GallagherS.SchmickingD. (Eds.), Handbook of phenomenology and cognitive science (pp. 21–34). Dordrecht, The Netherlands: Springer.
38.
GallagherS.ZahaviD. (2008). The phenomenological mind: An introduction to philosophy of mind and cognitive science. London, England: Routledge.
39.
GrespanL.FroeseT.PaoloE.Di SethA. (2008, November20–21). Investigating the role of movement in the constitution of spatial perception using the Enactive Torch. Paper presented at ENACTIVE08, Pisa, Italy.
40.
GriceH. P. (2002). Some remarks about the senses. In NoeA.ThompsonE. (Eds.), Vision and Mind. Selected papers in philosophy of perception (pp. 35–54). Cambridge, MA: The MIT Press.
41.
HaighA.BrownD. J.MeijerP.ProulxM. J. (2013). How well do you see what you hear? The acuity of visual-to-auditory sensory substitution. Frontiers in Psychology, 4, 330.
42.
HebbD. O. (1949). The Organization of Behavior. New York, NY: Wiley & Sons.
43.
HurlburtR. T.AkhterS. A. (2006). The Descriptive Experience Sampling method. Phenomenology and the Cognitive Sciences, 5, 271–301.
44.
HurlburtR. T.HeaveyC. L. (2001). Telling what we know: Describing inner experience. Trends in Cognitive Sciences, 5, 400–403.
45.
HurlburtR. T.HeaveyC. L. (2004). To beep or not to beep: Obtaining accurate reports about awareness. Journal of Consciousness Studies, 11, 113–128.
KaramM.RussoF. A.FelsD. I. (2009). Designing the model human cochlea: An ambient crossmodal audio-tactile display. IEEE Transactions on Haptics, 2, 160–169.
48.
KimJ. -K.ZatorreR. J. (2008). Generalized learning of visual-to-auditory substitution in sighted individuals. Brain Research, 1242, 263–275.
49.
KiversteinJ.FarinaM. (2012). Do sensory substitution devices extend the conscious mind?Consciousness in Interaction: The Role of the Natural and Social Context in Shaping Consciousness, 86, 19–40.
50.
KiversteinJ.FarinaM.ClarkA. (2014). Substituting the senses. In MatthenM. (Ed.), The Oxford handbook of philosophy of perception. Oxford, UK: Oxford University Press.
51.
Levy-TzedekS.HanassyS.AbboudS.MaidenbaumS.AmediA. (2012). Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. Restorative Neurology and Neuroscience, 30, 313–323.
52.
Levy-TzedekS.NovickI.ArbelR.AbboudS.MaidenbaumS.VaadiaE.AmediA. (2012). Cross-sensory transfer of sensory-motor information: Visuomotor learning affects performance on an audiomotor task, using sensory-substitution. Scientific Reports, 2, 949.
53.
LockeJ. (1836). An essay concerning human understanding. London, England: T. Tegg and Son.
54.
MacphersonF. (2011). Taxonomising the senses. Philosophical Studies, 153, 123–142.
55.
MaidenbaumS.ArbelR.AbboudS.ChebatD.Levy-TzedekS.AmediA. (2012). Virtual 3D shape and orientation discrimination using point distance information. In SharkeyP.KlingerE. (Eds.), Proceedings of the 9th international conference on disability, virtual reality and associated technologies (pp. 471–474). Reading, UK: University of Reading.
56.
MeijerP. B. (1992). An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering, 39, 112–121.
57.
MerabetL. B.BattelliL.ObretenovaS.MaguireS.MeijerP.Pascual-LeoneA. (2009). Functional recruitment of visual cortex for sound encoded object identification in the blind. Neuroreport, 20, 132.
58.
NagelS. K.CarlC.KringeT.MärtinR.KönigP. (2005). Beyond sensory substitution–learning the sixth sense. Journal of Neural Engineering, 2, R13–R26.
59.
NanayB. (2015). Perceptual content and the content of mental imagery. Philosophical Studies, 172, 1723–1736.
NoëA. (2004) Action in perception. Cambridge: The MIT Press.
62.
O’ReganJ. K. (2011). Why red doesn’t sound like a bell: Understanding the feel of consciousness. Oxford, UK: Oxford University Press.
63.
O’ReganJ. K.NoëA. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24, 939–973.
64.
OrtizT.PochJ.SantosJ. M.RequenaC.MartínezA. M.Ortiz-TeránL.. . . MartínezJ. M. (2011). Recruitment of occipital cortex during sensory substitution training linked to subjective experience of seeing in people with blindness. PLoS ONE, 6, e23264.
65.
Pascual-LeoneA.HamiltonR. (2001). The metamodal organization of the brain. Progress in brain research, 134, 427–45.
66.
PaulewiczB.WierzchońM. (2015). Casual considerations in experimental studies on consciousness. Humana Mente, 29, 227–242.
67.
PearsonJ.CliffordC. W.TongF. (2008). The functional impact of mental imagery on conscious perception. Current Biology, 18, 982–986.
68.
PepperK. (2014). Do sensorimotor dynamics extend the conscious mind?Adaptive Behavior, 22, 99–108.
69.
PerkyC. W. (1910). An experimental study of imagination. American Journal of Psychology, 21, 422–452.
70.
PetitmenginC. (2006). Describing one’s subjective experience in the second person: An interview method for the science of consciousness. Phenomenology and the Cognitive Sciences, 5, 229–269.
71.
PetitmenginC.BitbolM. (2009). Listening from within. Journal of Consciousness Studies, 16, 363–404.
72.
PlazaP.CuevasI.GrandinC.De VolderA. G.RenierL. (2012). Looking into task-specific activation using a prosthesis substituting vision with audition. ISRN Rehabilitation, 2012, 490950.
73.
PoirierC.De VolderA. G.ScheiberC. (2007). What neuroimaging tells us about sensory substitution. Neuroscience and Biobehavioral Reviews, 31, 1064–1070.
74.
PoirierC.De VolderA. G.TranduyD.ScheiberC. (2007). Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects. Neuropsychologia, 45, 1108–1121.
75.
PoirierC. C.De VolderA. G.TranduyD.ScheiberC. (2006). Neural changes in the ventral and dorsal visual streams during pattern recognition learning. Neurobiology of Learning and Memory, 85, 36–43.
76.
PrinzJ. (2006). Putting the brakes on enactive perception. Psyche, 12, 1–19.
77.
ProulxM. J. (2010). Synthetic synaesthesia and sensory substitution. Consciousness and Cognition, 19, 501–503.
ProulxM. J.StoerigP. (2006). Seeing sounds and tingling tongues: Qualia in synaesthesia and sensory substitution. Anthropology and Philosophy, 7, 135–150.
80.
ProulxM. J.StoerigP.LudowigE.KnollI. (2008). Seeing ‘where’through the ears: Effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution. PLoS ONE, 3, e1840.
81.
ProulxT.MarkmanK.LindbergM. J. (2013). Introduction: The New Science of Meaning. In ProulxT.MarkmanK.LindbergM. J. (Eds.), The Psychology of Meaning (pp. 3–14). Washington, D.C., US: American Psychological Association.
82.
PtitoM.MoesgaardS. M.GjeddeA.KupersR. (2005). Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blind. Brain, 128, 606–614.
83.
RenierL.De VolderA. (2010). Vision substitution and depth perception: Early blind subjects experience visual perspective through their ears. Disability and Rehabilitation: Assistive Technology, 5, 175–183.
84.
RenierL.LaloyauxC.CollignonO.TranduyD.VanlierdeA.BruyerR.De VolderA. G. (2005). The Ponzo illusion with auditory substitution of vision in sighted and early-blind subjects. Perception, 34, 857–867.
85.
Roxbee-CoxJ. M. (1970). Distinguishing the senses. Mind (New Series), 79, 316530–316550.
86.
SchmickingD. (2010). A toolbox of phenomenological methods. In GallagherS.SchmickingD. (Eds.), Handbook of phenomenology and cognitive science (pp. 35–55). New York, NY: Springer.
87.
SiegleJ. H.WarrenW. H. (2010). Distal attribution and distance perception in sensory substitution. Perception, 39, 208–223.
88.
SlotnickS. D. (2008). Imagery: Mental pictures disrupt perceptual rivalry. Current Biology, 18, R603–R605.
89.
SparingR.MottaghyF. M.GanisG.ThompsonW. L.TöpperR.KosslynS. M.Pascual-LeoneA. (2002). Visual cortex excitability increases during visual mental imagery – A TMS study in healthy human subjects. Brain Research, 938, 92–97.
90.
SpenceC.DeroyO. (2013). Crossmodal mental imagery. In LaceyS.LawsonR. (Eds.), Multisensory imagery (pp. 157–183). New York, NY: Springer.
91.
StilesN.ShimojoS. (2013). Sensory substitution: A new perceptual experience. In WagemansJ. (Ed.), Oxford handbook of perceptual organization (pp. 1–19). Oxford, UK: Oxford University Press.
92.
Striem-AmitE.CohenL.DehaeneS.AmediA. (2012). Reading with sounds: Sensory substitution selectively activates the visual word form area in the blind. Neuron, 76, 640–652.
93.
ThompsonE. (2005). Sensorimotor subjectivity and the enactive approach experience. Phenomenology and the Cognitive Sciences, 4, 407–427.
94.
WardJ.MeijerP. (2010). Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and Cognition, 19, 492–500.
95.
WardJ.WrightT. (2014). Sensory substitution as an artificially acquired synaesthesia. Neuroscience and Biobehavioral Reviews, 41, 26–35.
96.
ZahaviD. (2010). Naturalized phenomenology. In GallagherS.SchmickingD. (Eds.), Handbook of phenomenology and cognitive science (pp. 3–19). Dordrecht, The Netherlands: Springer.
