Sensory Substitution and Embodiment Shape Performance and Immersion in Teleoperated Settings

Telemanipulation allows remote operations using specialized tools, often under limited sensory input, such as the absence of haptic feedback. With practice, these tools can become embodied extensions of the user, enhancing Presence. Sensory Substitution, conveying information from one sensory modality through another, can mitigate sensory limitations in such tasks. We developed the Visual-Tactile Sensory Substitution Task to investigate learning of Sensory Substitution while accounting for Embodiment and Presence experience. Forty participants completed a within-subjects repeated-measures design in which each individual was required to manipulate 200 virtual elastic slabs using a haptic stylus to maximize deformation without breakage. Each participant performed two trial types in random order: 50% with force feedback (Control Condition) and 50% without it (Sensory Substitution Condition). Performance metrics included Frequency of Errors, Elapsed Time, Maximal Applied Force, along with Maximal Pupil Dilation and Number of Blinks as workload and learning indicators. Participants also assessed their Perceived Sensory Substitution, Embodiment, and Presence after the task. All metrics were analyzed separately for correct, incorrect, and aggregated trials. Structural Equation Modeling was applied to test the effects of Perceived Sensory Substitution (exogenous variable) on Sensory Substitution Learning via Embodiment and Presence (endogenous variables). Participants in the Sensory Substitution Condition showed reduced Elapsed Time, comparable Maximal Applied Force, and higher Frequency of Errors compared to the Control Condition. Blink rate showed phase-dependent modulation during correct trials, consistent with task learning, while pupil dilation remained stable across phases and feedback conditions. Structural Equation Modeling revealed that Perceived Sensory Substitution positively influenced Embodiment, Presence, and Learning. Overall, Sensory Substitution supports telemanipulation by fostering Embodiment and reducing task time, while modulating sensorimotor learning strategies relevant for efficient remote operation.