Convolutional Neural Networks on Apache Storm

the performance of deep learning largely depends on the size of data. One data source is real-time streaming data, produced from mobile devices, sensors or social media, etc. Streaming data is high-speed and large-scale, which needs real-time processing. However, current mainstream frameworks are mainly designed for off-line data. To suit this, we first propose a deep learning framework based on Apache Storm, which is a distributed stream processing frame, fast and fault-tolerant. Our framework implements the distributed training of CNNs. which is different from MMLSpark or TensorFlowOnSpark that is a pure Java implementation. The design of message passing and synchronization is also suitable to other MapReduce-family distributed computing platforms. To validate our work, MNIST and Cifar -10 datasets are used for evaluation and comparison with similar architectures. The results show our framework, in resource-limited environment, realizes about 10 times speedup.