Look first,feel faster: Prior visual information accelerates haptic material exploration

Abstract

Humans use distinct exploratory procedures (EPs) in active touch, which are typically specialized for materials with particular properties: for example, pressing for deformable objects such as cushions, or stroking to test a fabric's smoothness. Further, humans can use abstract visual priors for fine-tuning of exploratory movement parameters such as exploration direction. We here test the usage of visual priors in the planning of material-specific EPs, using real-life materials and a naturalistic visual virtual reality environment. We show that humans are better at selecting specialized EPs at initial touch when they have access to valid prior visual information on the material: They used specialized EP earlier, with higher probability, and explored materials for a shorter time. We conclude that visual prior information increases the efficiency of haptic explorations by anticipatory planning of appropriate movement schemes.

How to Cite this Article

Michaela Jeschke & Knut Drewing (2025). Look first, feel faster: Prior visual information accelerates haptic material exploration. i-Perception, 16(5), 1–5.

Humans employ specialized movements, so-called exploratory procedures (EPs), tailored to the object being interacted with and the information they aim to retrieve: for example, rubbing movements for coarse textures, rotating for granular materials, or normal pressing for deformable objects (Lederman & Klatzky, 1987). These movements are further fine-tuned regarding force or velocity based on serially gathered sensory info (Katircilar & Drewing, 2023; Weiss & Flanders, 2011). Additionally, predictive mechanisms enable movement preparation before initial contact: for example, humans use abstract visual priors to tune exploratory movement direction at initial contact during grating exploration (Jeschke et al., 2024). Here, we test the usage of visual priors for the selection of appropriate EPs and assess this in a more naturalistic setting involving real-life stimuli and a virtual reality (VR) environment. Regarding real-life material properties, humans derive high-level expectations on mechanical material properties from basic visual images (Wijntjes et al., 2019), which is why we deemed it plausible to observe improved haptic exploration in reaction to static visual information about upcoming materials.

Specifically, we asked: Does prior visual information on an upcoming material guide haptic exploration by supporting the selection of specialized haptic EPs?

We implemented a two-interval forced choice task: 14 participants were instructed to freely explore two haptic stimuli with their dominant hand (setup: Figure 1A). In each trial, the two stimuli consisted of the same material: either sandpaper, sand, or sponge (Figure 1B). Participants judged which one was rougher/more coarsely grained/more compliant. Explorations were filmed (60 Hz webcam). For each material, there were three possible stimulus pairs varying in discriminability (70%, 80%, and 90% accuracy in piloting): sandpapers differing in roughness via grit size, sand samples differing in granularity via particle sizes, or sponge-stimuli differing in Young's modulus. Participants wore a head-mounted display which visually presented a simplified duplicate of the scene in front of them: the haptic stimuli were represented as white boxes and their hand by an oval-shaped object. At the beginning of each trial, the white boxes changed appearance and displayed the type of the upcoming haptic material, or, in 50% of all trials, nonsense-visual information, consisting of all materials combined (Figure 1C). The onset of these visual stimuli indicated the beginning of the trial: participants could now begin their exploration (Figure 1E). As soon as they reached towards the stimuli, the boxes turned white again (signaled by a tracker at the participants’ wrist). After exploration of both stimuli, participants verbally reported their decision. The order and number of trials were randomized and counterbalanced regarding information type (informative/nonsense), material type (sandpaper/sand/sponge), stimulus pair (three pairs), and stimulus position (left/right) (36 trials in total).

Figure 1.

(A) Experimental setup: HTC VIVE HMD and HTC VIVE tracker. (B) Haptic stimuli. (C) Corresponding visual stimulus representation and nonsense visual information. (D) Specialized EPs for each material (one row for each material, respectively). (E) Experimental procedure.

Movement recordings of the first explored stimulus were analyzed because these provide insights into the merits of prior visual information. Recordings were manually decoded by an experienced rater with regard to EP type at initial contact (pressure, stroking, rubbing, running through, stirring, rotating, tapping, and pulling), onset times of the first EPs that are specialized for the respective material (rubbing and stroking for sandpaper; running through, stirring, and rotating for sand; tapping and pressure for sponge) (Figure 1D) (Cavdan et al., 2021; Lederman & Klatzky, 1987), and the exploration duration for the first stimulus. Crucially, the rater was fully blinded regarding the experimental condition (informative visual priors/nonsense) due to the randomization across trials and the fact that visual stimuli were only presented to the participants via the VR glasses. Onsets of initial material-specific EPs were defined as the time that passed between the beginning of the reaching phase (simultaneous to when prior visual information disappears) until the beginning of the first movement segment belonging to those initial EPs, which are specific for the respective explored material.

Onsets of initial material-specific EPs were earlier with informative visual priors, t₁₃ = 8.34, p ≤ .001, d_z = 2.23, Figure 2A; that is, participants reached quicker and employed the appropriate movement scheme sooner. Generally, participants almost always and thus more often used material-specific EPs at initial contact with informative visual priors, t₁₃ = 2.56, p = .012, d_z = 0.68 (informative: M = 97.15%, SD = 5.43; nonsense: M = 91.10%, SD = 11.23). With nonsense priors, participants sometimes used tapping movements initially. With informative priors, total exploration time spent on the first stimulus was reduced as compared to nonsense priors, t₁₃ = 4.49, p ≤ .001, d_z = 1.20, Figure 2B. This decrease likely demonstrates the advantage of specialized EPs, reflecting optimized gathering of sensory signals. Participants showed constant response accuracies across conditions, which is expected as unrestricted exploration tasks allow compensation in various ways, t₁₃ = 1.06, p = .308 (informative: M = 81.24%, SD = 9.93; nonsense: M = 77.01%, SD = 10.20).

Figure 2.

(A) Average onset of the initial EPs specialized for the current material. Error bars indicate the standard error of the mean. Faint dots connected with lines represent individual participant values across the two conditions. (B) Average exploration duration.

We conclude that visual prior information increases the efficiency of haptic explorations by anticipatory planning of appropriate, wholistic movement schemes beyond the fine-tuning of movement parameters previously shown. Behavioral optimization might further depend on the type of provided information (e.g., explicit or implicit) or on the complexity of visual input (e.g., static or dynamic). The default initial exploratory procedure in the absence of any prior information might be light tapping, which maximizes sensory intake while minimizing risks such as potential injury. This may reflect a general strategy across species and could be paralleled, for example, by whisking behavior in rodents (e.g., Hartmann, 2001). In total, this experiment underlines both the benefit of prediction in active touch and the advantage of specialized exploratory movements. These insights might have valuable applications in neurorobotics, where optimizing exploration processes via the exploitation of visual prior information could support object recognition and manipulation in autonomous agents.