Skip to main navigation Skip to search Skip to main content

HIPU: A Hybrid Intelligent Processing Unit With Fine-Grained ISA for Real-Time Deep Neural Network Inference Applications

  • Xi'an Jiaotong University

Research output: Contribution to journal › Article › peer-review

9 Scopus citations

Abstract

Neural network algorithms have shown superior performance over conventional algorithms, leading to the designation and deployment of dedicated accelerators in practical scenarios. Coarse-grained accelerators achieve high performance but can support only a limited number of predesigned operators, which cannot cover the flexible operators emerging in modern neural network algorithms. Therefore, fine-grained accelerators, such as instruction set architecture (ISA)-based accelerators, have become a hot research topic due to their sufficient flexibility to cover the unpredefined operators. The main challenges for fine-grained accelerators include the undesired long delays of single-image inference when performing multibatch inference, as well as the difficulty of meeting real-time constraints when processing multiple tasks simultaneously. This article proposes a hybrid intelligent processing unit (HIPU) to address the aforementioned problems. Specifically, we design a novel conversion-free data format, expanding the single-instruction multiple-data (SIMD) instruction set and optimizing the microarchitecture design to improve the performance. We also arrange the inference schedule to guarantee scalability on multicores. The experimental results show that the proposed accelerator maintains high multiply-accumulation (MAC) utilization for all common operators and achieves high performance with 4- $7\times $ speedup against NVIDIA RTX2080Ti GPU. Finally, the proposed accelerator is manufactured using TSMC 28-nm technology, achieving 1 GHz for each core, with a peak performance of 13 TOPS.

Original languageEnglish
Pages (from-to)1980-1993
Number of pages14
JournalIEEE Transactions on Very Large Scale Integration (VLSI) Systems
Volume31
Issue number12
DOIs
StatePublished - 1 Dec 2023

Keywords

  • Network-on-chip (NoC)
  • neural network (NN) inference accelerating
  • out-of-order (OoO) superscalar processor
  • reduced instruction set architecture

Fingerprint

Dive into the research topics of 'HIPU: A Hybrid Intelligent Processing Unit With Fine-Grained ISA for Real-Time Deep Neural Network Inference Applications'. Together they form a unique fingerprint.

Cite this