田中 章浩

Introduction Individuals from Western cultures rely on facial expressions during the audiovisual emotional processing of faces and voices. In contrast, those from East-Asian cultures rely more on voices. This study aimed to investigate whether immigrants adopt the tendency of the host culture or whether common features of migration produce a similar modification regardless of the destination. Methods We examined how immigrants from Western countries to Japan perceive emotional expressions from faces and voices using MRI scanning. Results Immigrants behaviorally exhibited a decrease in the influence of emotions in voices with a longer stay in Japan. Additionally, immigrants with a longer stay showed a higher response in the posterior superior temporal gyrus, a brain region associated with audiovisual emotional integration, when processing emotionally congruent faces and voices. Discussion These modifications imply that immigrants from Western cultures tend to rely even less on voices, in contrast to the tendency of voice-dominance observed in native Japanese people. This change may be explained by the decreased focus on prosodic aspects of voices during second language acquisition. The current and further exploration will aid in the better adaptation of immigrants to a new cultural society.

We perceive and understand others’ emotional states from multisensory information such as facial expressions and vocal cues. However, such cues are not always available or clear. Can partial loss of visual cues affect multisensory emotion perception? In addition, the COVID-19 pandemic has led to the widespread use of face masks, which can reduce some facial cues used in emotion perception. Thus, can frequent exposure to masked faces affect emotion perception? We conducted an emotion perception task using audio-visual stimuli that partially occluded the speaker’s face. Participants were simultaneously shown a face and voice that expressed either congruent or incongruent emotions and judged whether the person was happy or angry. The stimuli included videos in which the eyes or mouth were partially covered and where the whole face was visible. Our findings showed that, when facial cues were partially occluded, participants relied more on vocal cues for emotion recognition. Moreover, when the mouth was covered, participants relied less on vocal cues after the pandemic compared to before. These findings indicate that partial face masking and prolonged exposure to masked faces can affect multisensory emotion perception. In unimodal emotion perception from only facial cues, accuracy also improved after the pandemic compared to before for faces with the mouth occluded. Therefore, changes in the reliance on vocal cues in multisensory emotion perception during the pandemic period could be explained by improved facial emotion perception from the eye region.

Previous research has revealed that several emotions can be perceived via touch. What advantages does touch have over other nonverbal communication channels? In our study, we compared the perception of emotions from touch with that from voice to examine the advantages of each channel at the emotional valence level. In our experiment, the encoder expressed 12 different emotions by touching the decoder's arm or uttering a syllable /e/, and the decoder judged the emotion. The results showed that the categorical average accuracy of negative emotions was higher for voice than for touch, whereas that of positive emotions was marginally higher for touch than for voice. These results suggest that different channels (touch and voice) have different advantages for the perception of positive and negative emotions.