Summary & Key Goals
Summary
EASI-Health develops believable, adaptable embodied AI agents that act as supportive “early responders” in sensitive health interactions.
- Embodied conversational AI
- Communication neuroscience
- Multi-domain deployment
Key Goals
- Develop socially aware agents that complement human professionals.
- Integrate nonverbal skills with advanced language models to “read the room.”
- Validate via behavioral and neurophysiological measures.
- Create a flexible platform for basic and applied research.
Core Challenge
LLM-based AI excels verbally but lacks crucial nonverbal and relational skills for genuine trust and empathy.
Methodology
1) Build: Engineering Socially Aware Agents
- High-fidelity embodied avatars
- Customized Language Models (CLMs): domain-specific, secure, locally hosted
- Generative Body Language (GBL): nonverbal repertoire derived from real interactions
2) Test: Real-World Health Scenarios
- Post-operative recovery coaching
- Empathetic health motivation & counseling
- Patient intake & information gathering
- Mental health screening & support
3) Validate: Measuring "Under the Skin"
- Psychophysiology: Heart Rate (HR), Electrodermal Activity (EDA)
- Neurophysiology: EEG and fMRI; future: fNIRS
- Behavior analysis: facial/eye-tracking, nonverbal & conversation analytics
Our Research Streams
Embodied Agents for Coaching
Agents developed and tested in health & wellness contexts.
Neuroscience of Message Processing
fMRI, EEG, and psychophysiology to understand how health messages are received and predict outcomes.
Human–AI Communication Dynamics
How humans perceive, trust, and build rapport with AI counterparts.
Collaborative Ecosystem: University & Industry
Center for Avatar Research & Immersive Social Media Applications
Role: Research on human-to-human and human–AI interaction in VR/AR.
Facilities: Professional motion capture, high-fidelity 3D character creation, VR/XR dev in Unreal/Unity.
Neuroscience of Messages Lab
Role: Communication neuroscience research using real-life interpersonal and mediated messages.
Facilities: Multi-channel mobile EEG, comprehensive psychophysiology, neural responses to media.
Great Lakes Reality Labs & Haptix Studio
Role: Technology development, digital asset creation, implementation.
Broader Impact
Basic Science
Investigate fundamental mechanisms of communication, trust, and message understanding; frontier research in social AI.
Clinical Practice
Pathways to scalable tools to reduce staff burnout, improve patient education, and provide empathetic first contact.
University–Hospital Partnership
Synergy across clinical needs, communication science, neuroscience, and AI development for patient care.
Future Directions
Rapid Prototyping Platform
Flexible system to quickly build and test AI-driven health interventions.
Deeper Neuro-Validation
fMRI, EEG, and fNIRS to investigate the social brain’s response to AI agents in health situations.
Adaptive Interactions
Closed-loop systems where agents adapt verbal and nonverbal behavior in real time to user state (e.g., distress).
References
- Bente, G. Facilities for the graphical computer simulation of head and body movements. Behav Res Methods Instrum Comput. 1989;21:455–462.
- Bente, G., Schmälzle, R., Jahn, N.T., Schaaf, A. Measuring the effects of co-location on emotion perception in shared virtual environments: An ecological perspective. Frontiers in Virtual Reality. 2023;4.
- Grall, C., Tamborini, R., Weber, R., Schmälzle, R. Stories collectively engage listeners’ brains: Enhanced intersubject correlations during reception of personal narratives. J Commun. 2021.
- Kalyanaraman, S., & Sundar, S. S. The psychological appeal of personalized content in web portals: Does customization affect attitudes and behavior? Journal of Communication, 56(1), 110-132. 2006.
- Pimentel, D., Kalyanaraman, S., Fillingim, R., & Halan, S. The effects of VR use on pain experienced during a tattoo procedure: A pilot study. Frontiers in Virtual Reality, 2, 643938. 2021.
- Lim, S., Schmälzle, R., Bente, G. Artificial Intelligence (AI) Meets Virtual Reality (VR): Artificial Influence of Gender Matching in a Health-Related Conversation with an Embodied GPT-Agent. arXiv. 2024.
- Pfeiffer, U.J., Timmermans, B., Bente, G., Vogeley, K., Schilbach, L. A non-verbal Turing test: differentiating mind from machine in gaze-based social interaction. PLoS One. 2011;6:e27591.
- Pimentel, D., & Kalyanaraman, S. Customizing your demons: Anxiety reduction via anthropomorphizing and destroying an “anxiety avatar”. Frontiers in Psychology, 11, 566682. 2020.
- Schmälzle, R., Brook O’Donnell, M., Garcia, J.O., Cascio, C.N.C., Bayer, J., Vettel, J., Falk, E.B., Bassett, D.S.B. Brain connectivity dynamics during social interaction reflect social network structure. Proc Natl Acad Sci USA. 2017;114:5153–5158.
- Schmälzle, R., Häcker, F., Honey, C.J., Hasson, U. Engaged Listeners: Shared neural processing of powerful political speeches. Social, Cognitive, and Affective Neuroscience. 2015;1:168–169.
- Schmälzle, R., Lim, S., Cho, H.J., Wu, J., Bente, G. Examining the exposure-reception-retention link in realistic communication environments via VR and eye-tracking: The VR billboard paradigm. PLoS One. 2023;18:e0291924.
- Schmälzle, R., Meshi, D. Communication neuroscience: Theory, methodology and experimental approaches. Commun Methods Meas. 2020.
- Schmälzle, R., Wilcox, S. Harnessing Artificial Intelligence for Health Message Generation: The Folic Acid Message Engine. J Med Internet Res. 2022;24:e28858.
- Vogeley, K., Bente, G. Artificial humans: Psychology and neuroscience perspectives on embodiment and nonverbal communication. Neural Netw. 2010;23:1077–1090.
The Social Brain & Health Communication
Effective health outcomes rely on successful communication, from understanding instructions to building therapeutic trust.
Key Brain Regions
Critical Questions