Parallel tempering with equi-energy moves for training of restricted boltzmann machines

Training RBMs is laborious due to the difficulty of sampling from model's distribution. Although using Parallel Tempering (PT) alleviates the problem to some extent, it will result in low swap acceptance ratio when the states' energies of neighboring chains are very different. In this paper, we propose a novel PT algorithm based on the principle of swapping between chains with the same level of energy. This new algorithm partitions the state space obtained by a population of Gibbs sampling chains into several energy rings. In each ring, states have similar energies and swapping of each pair of states are conducted with a probability. Experiments on a toy dataset as well as the MNIST dataset shown that the new algorithm keeps high swap acceptance ration and results in better likelihood scores compared to several training methods.