Learning kernel-based HMMs for dynamic sequence synthesis

In this paper we present an approach that synthesizes a dynamic sequence from another related sequence, and apply it to a virtual conductor: to synthesize linked figure animation from an input music track. We propose that the mapping between two dynamic sequences can be modeled with a Kernel-based Hidden Markov Model, or KHMM. A KHMM is an HMM for which the kernel-based functions are used to model the state observation density of the joint input and output distribution. Specifically, the state observation density is estimated by employing a likelihood-weighted sampling scheme. Our KHMM model is ideal for dynamic sequence synthesis because the global dynamics are learned by the HMM, and subtle details in the dynamic mapping are kept in the kernel-based state density. We demonstrate our virtual conductor by synthesizing extensive animation sequences from input music sequences with different styles and beat patterns.