Dynamic Local-Global Attention for Audio-Visual Speech Enhancement in Noisy Communication
DOI:
https://doi.org/10.66238/ijrss.v6i2.120Keywords:
Audio-Visual Fusion, Speech Enhancement, Attention Mechanism,, Multimodal Processing, Robust CommunicationAbstract
Speech enhancement in noisy communication environments remains a formidable challenge, particularly when relying solely on acoustic signals corrupted by severe background interference. Recent advancements have demonstrated that incorporating visual cues, such as lip movements and facial expressions, can significantly improve speech intelligibility and quality. However, effectively fusing these multimodal streams while simultaneously capturing fine-grained phonetic details and long-range semantic dependencies remains an unresolved problem. This paper proposes a novel framework utilizing a dynamic local-global attention mechanism for audio-visual speech enhancement. The proposed architecture distinctively processes local temporal dynamics, crucial for aligning short-term phonetic and visemic features, while simultaneously modeling global sequence dependencies to maintain context and speaker characteristics across the entire utterance. A dynamic gating mechanism is introduced to adaptively weight the contributions of local and global attention representations based on the estimated instantaneous signal-to-noise ratio and the reliability of the visual stream. Comprehensive evaluations are conducted across multiple challenging acoustic environments, demonstrating that the proposed method substantially outperforms existing single-modality and multimodal baselines in terms of objective metrics such as perceptual evaluation of speech quality and short-time objective intelligibility. The findings underscore the critical role of multi-scale temporal modeling and adaptive multimodal fusion in advancing the robustness of communication systems in real-world scenarios.
References
1. Peng, Qucheng, et al. "3d vision-language gaussian splatting." International Conference on Learning Representations. Vol. 2025. 2025.
2. Wu D, Zhang H, Chen J, et al. Mind-paced speaking: A dual-brain approach to real-time reasoning in spoken language models[J]. arXiv preprint arXiv:2510.09592, 2025.
3. Lv, Z., Wang, Q., Lyu, T., & Dong, J. (2026). A Deep Learning-Based Model for User Emotion Recognition and Behavior Guidance Design: EMoGuideNet-An Emotion-Aware Multimodal Framework for Enhancing Intelligent User Interaction. Journal of Organizational and End User Computing (JOEUC), 38(1), 1-30.
4. Zhang, Y., Zhang, Y., Zhao, M., Feng, S., & Cheung, Y. M. (2026). How to Ask the AI: A User Perspective Survey for Large Language Model Prompting.IEEE Transactions on Artificial Intelligence.
5. Pei, Zhengqi, Qingming Huang, and Shuhui Wang. "When LLMs develop languages: Symbolic communication for efficient multi-agent reasoning." arXiv preprint arXiv:2606.29354 (2026).
6. Ma, Y., Qu, D., & Wang, Y. (2026). HALO-GNN: hallucination-resistant temporal graph neural networks for dynamic community detection.Scientific Reports.
7. Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems.
8. Wang, S., Zhang, X., Wang, P., & Li, X. (2026). Design of Magnetic Sensor Array-Based PUFs for IoT Security. IEEE Transactions on Instrumentation and Measurement, 75, 1-9.
9. Qu, D., Ma, Y., Yan, J., & Pyrozhenko, M. (2026). TriMeta-BFNet: A Tri-Meta Stacked Atypical-Frequency Bayesian Fourier Neural Network for Hallucination-Resistant Community Detection. Mathematics, 14 (13), 2283.
10. Li, X., Yang, F., Chen, L., & Cai, H. (2016, July). Saliency transfer: An example-based method for salient object detection. In IJCAI (pp. 3411-3417).
11. Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). LibriSpeech: An ASR corpus based on public domain audio books. In Proceedings of ICASSP.
12. Xu, X., Wang, Y., Xu, D., Peng, Y., Zhang, C., Jia, J., & Chen, B. (2022, May). Vsegan: Visual speech enhancement generative adversarial network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)(pp. 7308-7311). IEEE.
13. Wang, X., Suzuki, Y., Li, C. M., & Tanaka, S. (2025, June). Micropatterned PDMS damping layer integration to enhance the fidelity of piezoelectric MEMS speakers without SPL penalty. In2025 23rd International Conference on Solid-State Sensors, Actuators and Microsystems (Transducers)(pp. 455-458). IEEE.
14. Huang, H., Zhang, J., Zhang, J., Xu, J., & Wu, Q. (2020). Low-rank pairwise alignment bilinear network for few-shot fine-grained image classification.IEEE Transactions on Multimedia,23, 1666-1680.
15. Ma, Y., & Qu, D. (2026, February). IMFM-GNN: Interpretable Multimodel GNN Via Fourier-Markov Dynamics for Community Detection in Sustainable Energy Networks. In 2026 International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) (pp. 1-6). IEEE.
16. Liang, R., Xu, S., Xie, C., Chen, J., Ren, F., Yang, S., & Yabe, T. (2026). Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency. Advances in Neural Information Processing Systems, 38, 99445-99471.
17. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.
18. Xiaoteng Ma, Yiqin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint arXiv:2110.09796, 2021b.
19. Xu, X., Tu, W., & Yang, Y. (2023). Pcnn: A lightweight parallel conformer neural network for efficient monaural speech enhancement. arXiv preprint arXiv:2307.15251.
20. Dai, S., Ullmer, B., & Newman, W. E. (2024, February). MorphMatrix: A toolkit facilitating shape-changing interface design. In Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 1-12).
21. Qu, D., & Ma, Y. (2026). Fourier–Transformer Mixer Network for Efficient Video Scene Graph Prediction.Engineering Proceedings,120(1), 16.
22. Qu, W., Wang, J., Gong, Y., Huang, X., & Xiao, L. (2025, June). An end-to-end robust point cloud semantic segmentation network with single-step conditional diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 27325-27335). IEEE.
23. Zhang H, Chen J, Wu D, et al. Duplexsla: A full-duplex spoken language model with synchronized speech, language, and action[J]. arXiv preprint arXiv:2605.20755, 2026.
24. Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., et al. (2016). Deep Speech 2: End-to-end speech recognition in English and Mandarin. In Proceedings of ICML.
25. Tao X, Wu X, Zhuo S, et al. Unveiling the sedentary epidemic through insights from college students in Guangdong[J]. Scientific Reports, 2025, 15(1): 35197.
26. Qu, D., Ma, Y., & Wang, Y. (2026). STR-GNN: stability-regularized graph neural networks for suppressing spurious temporal variations in dynamic community detection. Scientific Reports.
27. Liang, H., Huang, C., Xu, Y., Tang, C., Ye, W., Zhang, J., ... & Xu, L. (2024). Llava-slt: Visual language tuning for sign language translation. arXiv preprint arXiv:2412.16524.
28. Chen, C., Zhang, J., Lakshmikanth, S. K., Fang, Y., Shao, R., Wetzstein, G., ... & Adeli, E. (2025). The language of motion: Unifying verbal and non-verbal language of 3d human motion. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 6200-6211).
29. Huang, J., Patel, A. N., Narasimha, S. M., Mishne, G., & Gilja, V. (2025). Word-level error analysis in decoding systems: From speech recognition to brain-computer interfaces. In Proc. Interspeech 2025 (pp. 5563-5567).
30. Xu, X., Tu, W., & Yang, Y. (2023). CASE-Net: Integrating local and non-local attention operations for speech enhancement. Speech Communication, 148, 31-39.
31. Cao, X., Tao, J., Liu, Z., Lyu, R., & Li, J. (2026). Handling Missing Data in CALL: A Data Quality-Driven Imputation Framework for Learner Analytics. Future-Adaptive Intelligence and Lifelong Systems, 1 (1).
32. Li, X., Wang, P., Li, G., & Zhang, Y. (2023). Design of a novel self-test-on-chip interface ASIC for capacitive accelerometers. IEEE Transactions on Circuits and Systems I: Regular Papers, 70(7), 2834-2843.
33. Li, X., Wang, P., Li, G., Ni, L., & Zhang, Y. (2023). Design of interface circuits and lightweight PUF for TMR sensors. IEEE Sensors Journal, 23(11), 11754-11761.
34. Shneiderman, B. (1983). Direct manipulation: A step beyond programming languages. IEEE Computer, 16(8), 57–69.
35. Yu, H., Zhang, J., Chen, C., Xiang, T., Fang, Y., Niebles, J. C., & Adeli, E. (2026, March). Socialgen: Modeling multi-human social interaction with language models. In 2026 International Conference on 3D Vision (3DV) (pp. 1-17). IEEE.
36. Li, X., Hu, J., & Liu, X. (2020). Harmonic Distortion Optimization for Sigma-Delta Modulators Interface Circuit of TMR Sensors.Sensors,20(4), 1041.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Elizabeth C. Hughes
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles in the International Journal of Research & Social Sciences are published under the Creative Commons Attribution 4.0 (CC BY 4.0) license. Users may share and adapt the content, including for commercial purposes, provided proper credit is given to the authors and the journal.
