Home
Resource
Multimodal attention network for trauma activity recognition from spoken language and environmental sound
PDF
PDF format is widely accepted and good for printing.
Plug-in required
PDF-1(1.25 MB)
Citation & Export
View Usage Statistics
Staff View

Citation & Export
Hide

Simple citation

Zhang, Ruiyu. Multimodal attention network for trauma activity recognition from spoken language and environmental sound. Retrieved from https://doi.org/doi:10.7282/t3-2hcr-tk25

Export

Click here for information about Citation Management Tools at Rutgers.

Statistics
Hide

Description

TitleMultimodal attention network for trauma activity recognition from spoken language and environmental sound
NameZhang, Ruiyu (author); Marsic, Ivan (chair); Gajic, Zoran (internal member); Chen, Yingying (internal member); Rutgers University; School of Graduate Studies
Date Created2020
Other Date2020-01 (degree)
SubjectElectrical and Computer Engineering, Medical informatics, Resuscitation
Extent1 online resource (vi, 21 pages) : illustrations
DescriptionTrauma activity recognition aims to detect, recognize, and predict the activities (or tasks) during a trauma resuscitation. Previous work has mainly focused on using various sensor data including image, RFID, and vital signals to generate the trauma event log. However, spoken language and environmental sound, which contain rich communication and contextual information necessary for trauma team cooperation, is still largely ignored. In this paper, we propose a multimodal attention network (MAN) that uses both verbal transcripts and environmental audio stream as input; the model extracts textual and acoustic features using a multi-level multi-head attention module, and forms a final shared representation for trauma activity classification. We evaluated the proposed architecture on 75 actual trauma resuscitation cases collected from a hospital. We achieved 72.4% accuracy with 0.705 F1 score, demonstrating that our proposed architecture is useful and efficient. These results also show that using spoken language and environmental audio indeed helps identify hard-to-recognize activities, compared to previous approaches. We also provide a detailed analysis of the performance and generalization of the proposed multimodal attention network.
NoteM.S.
NoteIncludes bibliographical references
NoteIncludes vita
Genretheses, ETD graduate
Persistent URLhttps://doi.org/doi:10.7282/t3-2hcr-tk25
LanguageEnglish
CollectionSchool of Graduate Studies Electronic Theses and Dissertations
Organization NameRutgers, The State University of New Jersey
RightsThe author owns the copyright to this work.
Version 8.5.5
Rutgers University Libraries - Copyright ©2024