VAE-Based Generic Decoding via Subspace Partition and Priori Utilization

Generic decoding is a challenging problem in visual neural decoding. The existing methods based on generative models ignore the application of prior knowledge, which leads to poor interpretability, and few pay attention to fMRI (functional Magnetic Resonance Imaging) processing. To tackle these problems, a novel framework for generic decoding has been proposed named GD-VAE. GD-VAE is based on Variational Auto-Encoder (VAE) which is capable of meaningful latent space, and contains four modules: feature extractor, feature VAE, Prior Knowledge Network (PKN) and Latent Space Disentangling Network (LSDN). The feature extractors extract features of raw visual and cognitive data, and feature VAE implements decoding with a shared latent space for both modalities. The PKN and LSDN constrain the latent space of VAE with delicate structure, in order to apparently reveal the information in the subspace. Benefiting from these modules, the alignment between visual and cognitive modality can be achieved, and greater interpretability can be acquired. Experiments on Generic Decoding Dataset validate the effectiveness and interpretability of the proposed method.