Model-based clustering with nonconvex penalty

Nonconvex penalty functions, which include the smoothly clipped absolute deviation (SCAD) penalty, minimax concave penalty (MCP) and ℓ_q(0 ≤ q < 1) norm penalty, have been demonstrated to have attractive theoretical properties and excellent performance on experiment studies in the area of penalized regressions, compressive sensing and matrix completion. To take their advantages, we propose a penalized model-based clustering framework via the nonconvex penalty functions for dealing with high-dimensional data clustering problems. We establish an expectation-maximization (EM) algorithm to fit the suggested framework efficiently. To illustrate the general framework, we utilize four popular nonconvex penalties (SCAD, MCP, ℓ₀ and ℓ_1/2) to construct specific models. They are compared with the ℓ₁ penalty in the simulations and a real world application. Based on our experiments, the finite sample performance of the four proposed models is well exhibited. In particular, our numerical results suggest that the model-based clustering with the MCP or ℓ₀ penalty is the preferred approach.