A residual coefficients pruning algorithm in H.264

A pruning algorithm is proposed to reduce the encoding bits for the map information of residual coefficients in H.264. Video sequences are predicted, transformed, quantized, and scanned. Then residual coefficients obtained from scanning are divided into linear groups, and then the groups are pruned out using the rate-distortion optimization (RDO) sense. Furthermore, the pruning results of each macroblock with different quantization steplengths are calculated in a given range. Finally the remaining coefficients with the minimal rate-distortion cost are encoded and transformed. Experimental results show that the proposed algorithm has better RD performance than the residual coding method in H.264 at the relatively high bitrates. The proposed algorithm gains an 0.9 dB increase on the peak signal to noise ratio(PSNR) at the same bitrate, or equivalently, a 17% rate saving at the same PSNR. Moreover, the proposed algorithm does not change the bit-stream structure of H.264, and is compatible with the H.264 standard.