MELD Projects
Consortium-Wide Projects
Speeded Reaction Time (SRT) to Auditory, Visual, & Audiovisual Stimulation
This study investigates SRT to unisensory and multisensory inputs to investigate the development of multisensory facilitation effects in children.
Active Passive Multisensory Perception Development
Movement can alter the perception of time in a multisensory environment. However, how the movement has an influence during development was not yet investigated comparing directly active and passive movement. Here, we evaluate how the temporal binding window is altered by active and passive movement. Haptic devices will be used with this goal.
Serial Dependence Orientation Tactile Perception in Blind Individuals
The aim of this research line is to investigate how blind individuals perceive tactile stimuli and to examine whether the phenomenon of serial dependence also influences the perception of touch. Through a series of studies, this research seeks to explore whether prior tactile experiences shape the perception of subsequent stimuli, similar to the effect observed in visual perception. By analyzing these interactions, we aim to better understand how the brain processes and integrates sensory information in the absence of vision, shedding light on the mechanisms underlying sensory compensation and multisensory integration.
Serial Dependence in Multisensory Processes
It is currently unknown how performance on preceding trials influence processing of current stimulus information, particularly under multisensory conditions where stimuli can partially “repeat” for one sensory modality and also be “novel” on a given trial for another sensory modality.
Linking Low-Level Multisensory Processes to Higher-Level Cognition
Much of our prior work has suggested there are links between low-level multisensory processes and higher-level cognitive (dys)functions. This remains to be firmly established, particularly in children. We have collected both SRT data as well as measures of cognitive abilities (working memory, fluid intelligence) to assess putative links. This project aims to fill a knowledge gap between data acquired in younger schoolchildren and data acquired in adults by acquiring data from older adolescents (aged 15-18). Identifying such links would open new pathways for screening and intervention strategies.
EEG Frequency Tagging Analysis Pipelines
EEG frequency tagging is a fast and efficient way to get high SNR in a short period of time. This is highly promising for use in pediatric populations. However, the analysis pipelines for this are heterogeneous and ill-defined. This project aims to generate a highly stable pipeline.
EEG Analysis Pipeline for Assessing Multisensory Processes in Children
A crucial element for the MELD consortium will be to establish a streamlined analysis pipeline for EEG data.
Standardized Multisensory Integration Analysis
A GUI that allows detailed analysis of race model violations, one of the standard tools used to assess MSI.
App Based Multisensory Integration Experiments
A phone app for collecting multisensory psychophysical data using standard paradigms.
Vanderbilt University
Consortium Principal Investigator – Mark T. Wallace, PhD
Vanderbilt is the primary site for the consortium, and oversees all aspects of the consortium’s work. One of the major emphasis at the Vanderbilt site will be assessing various facets of multisensory development in children from ages 5-21. In addition, Vanderbilt will serve as a focal site for examining how multisensory function differs in its development in neurodiverse populations, such as in autistic individuals.
Current Projects
Learned and Semantic Binding Windows Across Development
Semantics help us bind information, but much of this is learned. Here we create synthetic audiovisual stimuli that have experimental learned associations. We would like to see how this changes as a function of development with the idea that associations are easier to learn early in development.
Motion Classification of Stereotypies
Individuals with autism often exhibit stereotyped movements (e.g. hand-flapping). As part of a larger project investigating the potential utility of these stereotypies, we are working on pipelines to automatically classify marker-based movement data sets.
Virtual and Augmented Reality Investigations of Peripersonal Space
We are refining existing paradigms and developing a novel bubble-popping task to measure the extent and flexibility of peripersonal space. These are friendly for children and individuals with a variety of motor deficits, and will initially be used to test hypotheses about peripersonal space in individuals with autism and neurotypical.
Audiovisual Motion Processing
We are developing audio and audiovisual versions of the standard visual psychophysics tool known as a Random Dot Kinetogram (RDK). This allows us to test a number of hypotheses about multisensory integration when processing moving stimuli.
Italian Institute of Technology
Principal Investigator – Monica Gori, PhD
The development of multisensory skills still needs to be better understood. Our activity shows that sensory modalities’ interaction and integration are strictly interconnected. However, how these two mechanisms work at different ages considering their longitudinal development, is a mystery. Within the MELD project, we will focus on visual-tactile/haptic and audio-tactile/haptic processing considering typical and atypical sensory impaired children to determine behavioral and neurophysiological levels when and how sensory modalities scaffold multisensory integration. To do so within the consortium, we will develop simple and complex perceptual scenarios to modulate the task complexity and consider low-level and higher-level sensory processing.
Current Projects
Multisensory processing in VI and ADHD
Cross-modal and multisensory effects have been well illustrated on different percepts, affecting both reaction time (RT) and event related potential (ERP). Similarly, oddball response can be influenced by the context e by the multisensory cues. To address how different senses affect attention, audio, visual, tactile, and multisensory oddball tasks will be assessed in adults. Then we will extend the investigation to ADHD children, given that auditory oddball responses have been shown to be altered but the influence of multisensory stimuli remains unexplored, and VI children, to explore how MSI and attentional mechanisms interact in diverse sensory processing contexts.
ROMAT: Visual Tactile Interaction
It has been demonstrated that both the visual and tactile systems share common mechanisms for processing motion. The goal of this line of research is to demonstrate that not only do they share similar mechanisms, but they also interact to create a unified perception of the world around us. We are also investigating the role of this interaction in vision loss, specifically in individuals with visual impairments, and how the system reorganizes to compensate for the lack of integration between these two senses.
Cataract and Multisensory Processing
We will use multisensory integration skills in children with congenital cataract before and after the surgery in comparison to sighted peers.
Cross-Modal Visual to Haptic Transfer
We investigate the role of vision on haptic abilities of blind children and adults. The project has the goal of using real and virtual objects to explore haptic skills and body movement in blind children and adults.
Multisensory Interaction
During every kind of interaction with the environment we intercept objects and enter in contact with them. This project wants to study interception multisensory developmental skills and their alteration in sighted, blind and VI children and adults.
Mixed Reality Validation for Psychophysics Experiments
This research line focuses on the study, validation, and application of extended reality environments, such as virtual and mixed reality, as tools for investigating human perception. Through the use of these immersive environments, the goal is to explore how different sensory modalities interact and how they can be leveraged to conduct psychophysical experiments. By employing advanced technologies, this line of research aims to gain deeper insights into perceptual processes and improve our understanding of sensory integration and human cognition in both controlled and dynamic virtual settings.
University Hospital Center and University of Lausanne
Principal Investigator – Micah Murray, PhD
The University Hospital Center and University of Lausanne (CHUV-UNIL) team will particularly focus on the role of early life experience on multisensory development, including but not limited to premature birth and schooling environment. The CHUV-UNIL team will also put particular emphasis on how multisensory processes provide the scaffolding for higher-cognitive development and thus provide a strong candidate access point for remediation and potential prevention. Finally, the CHUV-UNIL team will concentrate effort on bridging sensation, perception and resultant movement in immersive environments.
Current Projects
Conspecific Voice Recognition in Prematurely Born Schoolchildren
Studies of prematurely born infants have documented impairments in voice recognition. Currently unknown is to what extent this persists into later childhood. This project addresses this knowledge gap by presenting prematurely born schoolchildren with vocalizations of humans vs. other animals to determine the integrity of neural mechanisms of human voice processing.
Integrity of Mirror Neuron Mechanisms in Schoolchildren
The so-called mirror neuron system can be differentially activated by sounds that have associated actions (De Lucia et al., 2010 Neuroimage). What is currently unknown is whether and when this is observable in children. This may give new access into the integrity of this system and the development of theory of mind.
Plasticity of Object Representations in Schoolchildren
Object representations of environmental sounds are subject to short-term plasticity due to repeated exposure (Murray et al., 2008 Neuroimage). This plasticity manifests behaviourally as priming and electrophysiologically as repetition suppression. We currently do not know whether and how this manifests in schoolchildren as well as in children born prematurely.
Intraindividual Variability in Multisensory Processes in Children With and Without ADHD
Prior work shows that children with ADHD present with greater intra-individual variability. Other research shows that multisensory information can reduce variability in behavior. What is unknown is whether multisensory information can reduce variability in behaviour of ADHD children. What is also largely unknown is how children with ADHD integrate multisensory information. This study uses a Go/NoGo multisensory paradigm in combination with EEG measures, actigraphy and emotional momentary assessments (EMAs).
Multisensory Learning of Foreign Languages
This project examines the interplay of learning strategies and multisensory processes for foreign language acquisition in adults and children.
Influence of Montessori Pedagogy on Cognitive Control
Cognitive control – the self-regulation of thoughts and actions – contributes to academic achievement. Oftentimes, this capacity is assessed via short-duration tasks that may rely on a relatively transient instantiation of cognitive control. However, many daily life situations require children to engage in cognitive control over an extended period, relying instead on relatively sustained mechanisms. Although research has increasingly shown that environmental factors shape cognitive control, little is known about the potential influence of different pedagogies.
Learning Strategies in Sensory Substitution
This project investigates long-held assumptions about how visual sensory information can be translated into an interpretable auditory signal for sensory substitution purposes. This is important for the goals of MELD because even children without sensory impaired may likely benefit from sensory substitution for object recognition and interactions. A first step will be to investigate this conversation in adults before assessing children.
Spatio-Temporal Brain Dynamics of Sensory Substitution
It has been proposed that the brain is a sensory-independent task-relevant machine. Empirical support for this proposal largely comes from fMRI data. However, spatio-temporal brain dynamics are needed to support or refute whether a nominally visual brain region is itself the first to perform a given discrimination function.
Eye-tracking Feedback Loops for Stimulus Presentation and Control
This project focuses on setting up an eye-tracker based heat map of gaze direction to in turn control stimulus presentation. It will achieve this both in adults and subsequently in children.
Yale University
Principal Investigators – David Lewkowicz, PhD, and Nick Turk-Browne, PhD
The Yale site will contribute its unique expertise and facilities by pursuing studies in infants and toddlers, in addition to young children. These include fMRI studies in awake infants and toddlers to assess multisensory integration across the early developing brain, as well as behavioral studies with eye-tracking to investigate temporal binding and perceptual segregation of audiovisual speech.
Current Projects
Multisensory Cocktail Party Problem in Children: Effects of Facial Configuration and Linguistic Cues on Perceptual Segregation
This study investigates the effects of multisensory clutter on perception and cognition in young children. The current study examines children’s responsiveness to multiple talking faces and the effects facial configuration cues, as well as A-V synchrony, visual and acoustic identity, and linguistic cues on segregation of multiple talking faces.
Multisensory Cocktail Party Problem in Children: Effects of Early Face & Linguistic Experience on Perceptual Segregation
Segregation of multisensory clutter depends on familiarity with talking faces. Here, we investigate whether segregation of other-race (Asian) faces speaking in an unfamiliar language (Mandarin) affects White-Caucasian, English-learning children’s ability to perceptually segregate them.
Multisensory Cocktail Party Problem in Infants
Integration of auditory and visual speech cues develops rapidly in infancy. Studies to date have only investigated infant integration of isolated speech syllables, raising questions about the generalizability of the findings from these studies for understanding infant processing of fluent audiovisual speech. Here, using our multisensory cocktail party problem paradigm, we investigate infants’ integration of fluent audiovisual speech and the effects of multisensory clutter on this ability.
Neural Processing of Audiovisual Synchrony in Infants and Children
Here we are leveraging state-of-the-art fMRI techniques and intersubject correlation analyses to study the early development of how audio-visual temporal synchrony affects multisensory integration across the brain during naturalistic perception.
Multisensory Integration in Memory Systems
The goal of this fMRI project in adults is to examine the integration of multisensory signals in the hippocampus, a major convergence zone in the human brain that receives inputs from all sensory systems and that encodes memories of multisensory events.
The Use of EEG Frequency Tagging to Extract Separate Measures of Overt and Covert Attention
Using EEG frequency tagging, we have developed a method for rapidly extracting separate measures of overt and covert attention from the brain’s response to visual information. Findings indicate that covert attention reduces overt attention when the two compete with one another. Our plan is to adapt this technique for the study of multisensory overt and covert attention.
