BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis

Huiqin Wu; Wenbing Lv; Dongyang Du; Hui Xu; Guoyu Lin; Zidong Zhou; Jianhua Ma; Lijun Lu

doi:10.1109/NSS/MIC44845.2022.10399186

BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis

Huiqin Wu
, Wenbing Lv
, Dongyang Du
, Hui Xu
, Guoyu Lin
, Zidong Zhou
, Jianhua Ma
, Lijun Lu

Southern Medical University
Guangdong Artificial Intelligence and Digital Economy Laboratory - Guangzhou

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In this work, we proposed a bidirectional attention-aware fluid pyramid features integrated fusion network (BAF-Net) with cross-modal interactions for multi-modality medical images prognosis. The network is composed of two identical branches to preserve the unimodal feature and one paralleled bidirectional attention-aware distillation stream to assimilate cross-modal complements progressively and learn supplementary refined features in both the bottom-up and the top-down processes. The fluid pyramid connections were adopted to integrate the hierarchical features at different levels of deep neural network, and channel-wise attention modules were exploited to mitigate the cross-modal cross-level incompatibility. Furthermore, the depth-wise separable convolution was introduced to fuse the cross-modal cross-level features to alleviate the increase of parameters to a great extent. Better performance was obtained by the proposed BAF-Net compared to unimodal network for cancer prognosis in a public TCIA dataset (head & neck cancer (HNC) dataset composed of 800 patients from nine centers).

Original language	English
Title of host publication	2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665488723
DOIs	https://doi.org/10.1109/NSS/MIC44845.2022.10399186
State	Published - 2022
Externally published	Yes
Event	2022 IEEE Nuclear Science Symposium, Medical Imaging Conference, and Room Temperature Semiconductor Detector Conference, IEEE NSS MIC RTSD 2022 - Milano, Italy Duration: 5 Nov 2022 → 12 Nov 2022

Publication series

Name	2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference

Conference

Conference	2022 IEEE Nuclear Science Symposium, Medical Imaging Conference, and Room Temperature Semiconductor Detector Conference, IEEE NSS MIC RTSD 2022
Country/Territory	Italy
City	Milano
Period	5/11/22 → 12/11/22

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Keywords

Bidirectional attention-aware
deep learning
diagnosis and prognosis
fluid pyramid feature integration
multi-modalities fusion

Access to Document

10.1109/NSS/MIC44845.2022.10399186

Cite this

Wu, H., Lv, W., Du, D., Xu, H., Lin, G., Zhou, Z., Ma, J., & Lu, L. (2022). BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis. In 2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference (2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/NSS/MIC44845.2022.10399186

Wu, Huiqin ; Lv, Wenbing ; Du, Dongyang et al. / BAF-Net : Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis. 2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference. Institute of Electrical and Electronics Engineers Inc., 2022. (2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference).

@inproceedings{6d54fa308aef4ed89009b9975507cd16,

title = "BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis",

abstract = "In this work, we proposed a bidirectional attention-aware fluid pyramid features integrated fusion network (BAF-Net) with cross-modal interactions for multi-modality medical images prognosis. The network is composed of two identical branches to preserve the unimodal feature and one paralleled bidirectional attention-aware distillation stream to assimilate cross-modal complements progressively and learn supplementary refined features in both the bottom-up and the top-down processes. The fluid pyramid connections were adopted to integrate the hierarchical features at different levels of deep neural network, and channel-wise attention modules were exploited to mitigate the cross-modal cross-level incompatibility. Furthermore, the depth-wise separable convolution was introduced to fuse the cross-modal cross-level features to alleviate the increase of parameters to a great extent. Better performance was obtained by the proposed BAF-Net compared to unimodal network for cancer prognosis in a public TCIA dataset (head \& neck cancer (HNC) dataset composed of 800 patients from nine centers).",

keywords = "Bidirectional attention-aware, deep learning, diagnosis and prognosis, fluid pyramid feature integration, multi-modalities fusion",

author = "Huiqin Wu and Wenbing Lv and Dongyang Du and Hui Xu and Guoyu Lin and Zidong Zhou and Jianhua Ma and Lijun Lu",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE Nuclear Science Symposium, Medical Imaging Conference, and Room Temperature Semiconductor Detector Conference, IEEE NSS MIC RTSD 2022 ; Conference date: 05-11-2022 Through 12-11-2022",

year = "2022",

doi = "10.1109/NSS/MIC44845.2022.10399186",

language = "英语",

series = "2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference",

}

Wu, H, Lv, W, Du, D, Xu, H, Lin, G, Zhou, Z, Ma, J & Lu, L 2022, BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis. in 2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference. 2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference, Institute of Electrical and Electronics Engineers Inc., 2022 IEEE Nuclear Science Symposium, Medical Imaging Conference, and Room Temperature Semiconductor Detector Conference, IEEE NSS MIC RTSD 2022, Milano, Italy, 5/11/22. https://doi.org/10.1109/NSS/MIC44845.2022.10399186

BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis. / Wu, Huiqin; Lv, Wenbing; Du, Dongyang et al.
2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference. Institute of Electrical and Electronics Engineers Inc., 2022. (2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - BAF-Net

T2 - 2022 IEEE Nuclear Science Symposium, Medical Imaging Conference, and Room Temperature Semiconductor Detector Conference, IEEE NSS MIC RTSD 2022

AU - Wu, Huiqin

AU - Lv, Wenbing

AU - Du, Dongyang

AU - Xu, Hui

AU - Lin, Guoyu

AU - Zhou, Zidong

AU - Ma, Jianhua

AU - Lu, Lijun

PY - 2022

Y1 - 2022

N2 - In this work, we proposed a bidirectional attention-aware fluid pyramid features integrated fusion network (BAF-Net) with cross-modal interactions for multi-modality medical images prognosis. The network is composed of two identical branches to preserve the unimodal feature and one paralleled bidirectional attention-aware distillation stream to assimilate cross-modal complements progressively and learn supplementary refined features in both the bottom-up and the top-down processes. The fluid pyramid connections were adopted to integrate the hierarchical features at different levels of deep neural network, and channel-wise attention modules were exploited to mitigate the cross-modal cross-level incompatibility. Furthermore, the depth-wise separable convolution was introduced to fuse the cross-modal cross-level features to alleviate the increase of parameters to a great extent. Better performance was obtained by the proposed BAF-Net compared to unimodal network for cancer prognosis in a public TCIA dataset (head & neck cancer (HNC) dataset composed of 800 patients from nine centers).

AB - In this work, we proposed a bidirectional attention-aware fluid pyramid features integrated fusion network (BAF-Net) with cross-modal interactions for multi-modality medical images prognosis. The network is composed of two identical branches to preserve the unimodal feature and one paralleled bidirectional attention-aware distillation stream to assimilate cross-modal complements progressively and learn supplementary refined features in both the bottom-up and the top-down processes. The fluid pyramid connections were adopted to integrate the hierarchical features at different levels of deep neural network, and channel-wise attention modules were exploited to mitigate the cross-modal cross-level incompatibility. Furthermore, the depth-wise separable convolution was introduced to fuse the cross-modal cross-level features to alleviate the increase of parameters to a great extent. Better performance was obtained by the proposed BAF-Net compared to unimodal network for cancer prognosis in a public TCIA dataset (head & neck cancer (HNC) dataset composed of 800 patients from nine centers).

KW - Bidirectional attention-aware

KW - deep learning

KW - diagnosis and prognosis

KW - fluid pyramid feature integration

KW - multi-modalities fusion

UR - https://www.scopus.com/pages/publications/85185383409

U2 - 10.1109/NSS/MIC44845.2022.10399186

DO - 10.1109/NSS/MIC44845.2022.10399186

M3 - 会议稿件

AN - SCOPUS:85185383409

T3 - 2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference

BT - 2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 5 November 2022 through 12 November 2022

ER -

Wu H, Lv W, Du D, Xu H, Lin G, Zhou Z et al. BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis. In 2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference. Institute of Electrical and Electronics Engineers Inc. 2022. (2022 IEEE NSS/MIC RTSD - IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference). doi: 10.1109/NSS/MIC44845.2022.10399186

BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this