MSE-based regularization approach to direction estimation of coherent narrowband signals using linear prediction

This paper addresses the problem of directions-of-arrival (DOAs) estimation of coherent narrowband signals impinging on a uniform linear array (ULA) when the number of signals is unknown. By using an overdetermined linear prediction (LP) model with a subarray scheme, the DOAs of coherent signals can be estimated from the zeros of the corresponding prediction polynomial. Although the corrected least squares (CLS) technique can be used to improve the accuracy of the LP parameters estimated from the noisy array data, the inversion of the resulting matrix in the CLS estimation is ill-conditioned, and then, the CLS estimation becomes unstable. To combat this numerical instability, we introduce multiple regularization parameters into the CLS estimation and show that determining the number of coherent signals is closely related to the truncation of the eigenvalues. An analytical expression of the mean square error (MSE) of the estimated LP parameters is derived, and it is clarified that the number of signals can be determined by comparing the optimal regularization parameters with the corresponding eigenvalues. An iterative regularization algorithm is developed for estimating directions without any a priori knowledge, where the number of coherent signals and the noise variance are estimated from the noise-corrupted received data simultaneously.