BAF-Net: Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Prognosis

In this work, we proposed a bidirectional attention-aware fluid pyramid features integrated fusion network (BAF-Net) with cross-modal interactions for multi-modality medical images prognosis. The network is composed of two identical branches to preserve the unimodal feature and one paralleled bidirectional attention-aware distillation stream to assimilate cross-modal complements progressively and learn supplementary refined features in both the bottom-up and the top-down processes. The fluid pyramid connections were adopted to integrate the hierarchical features at different levels of deep neural network, and channel-wise attention modules were exploited to mitigate the cross-modal cross-level incompatibility. Furthermore, the depth-wise separable convolution was introduced to fuse the cross-modal cross-level features to alleviate the increase of parameters to a great extent. Better performance was obtained by the proposed BAF-Net compared to unimodal network for cancer prognosis in a public TCIA dataset (head & neck cancer (HNC) dataset composed of 800 patients from nine centers).