MoEA: A Mixed-Precision Edge Accelerator for CNN-MSA Models with Fine-Tuning Support

Recent vision models integrating Convolutional Neural Networks (CNNs) and attention-based Transformers have achieved unprecedented accuracy, but face significant obstacles for edge deployment: sensitivity to quantization formats (especially for nonlinear functions in Transformers), high computational instructions complexity, and the inability to perform on-device fine-tuning for distribution shifts. To address these challenges, we propose Mixture-of-Edge-Architectures(MoEA), a RISC-V-based accelerator with three key innovations. First, the Mixed Fixed/Floating-Point (MFP) format can unify INT8, INT16, Shared Exponent Floating-Point (SFP8), FP8, and FP16 into a single data path, supporting the optimal mixed-precision strategy. Second, the typical VLIW instruction is condensed into a single direct memory access instruction to improve the computation performance. Third, a specialized engine integrates an FPGA-optimized Matrix Processing Unit(MPU), a Single Instruction Multi Data(SIMD)-based Vector Processing Unit(VPU), and an enhanced Direct Memory Access(DMA) for back-propagation. Implemented on FPGAs, MoEA delivers 420 GOPS (0.99 GOPS/DSP) on ZCU102, with ResNet18 inference at 16 ms and fine-tuning at 70ms. An 8-cluster variant on XCVU9P reduces ViT-Base latency to 23ms, 1.81 × ∼ 2.13 × faster than prior accelerators, supporting versatile model deployment and adaptive edge intelligence.