Origin of zigzag antiferromagnetic order in XPS3 (X=Fe,Ni) monolayers

Recently, two monolayer magnetic materials, i.e., FePS3 and NiPS3, have been successfully fabricated. Despite them having the same atomic structure, the two monolayers exhibit distinct magnetic properties. FePS3 holds an out-of-plane zigzag antiferromagnetic (AFM-ZZ) structure, while NiPS3 exhibits an in-plane AFM-ZZ structure. However, there is no theoretical model that can properly describe its magnetic ground state due to the lack of a full understanding of its magnetic interactions. Here, by combining the first-principles calculations and the newly developed machine learning method, we construct an exact spin Hamiltonian of the two magnetic materials. Different from the previous studies that failed to fully consider the spin-orbit-coupling effect, we find that the AFM-ZZ ground state in FePS3 is stabilized by competing ferromagnetic nearest-neighbor and antiferromagnetic third-nearest-neighbor exchange interactions and combining single-ion anisotropy. In contrast, the often ignored nearest-neighbor biquadratic exchange is responsible for the in-plane AFM-ZZ ground state in NiPS3. We additionally calculate the spin-wave spectrum of the AFM-ZZ structure in the two monolayers based on the exact spin Hamiltonian, which can be directly verified by the experimental investigation. Our work provides a theoretical framework for the origin of the AFM-ZZ ground state in two-dimensional materials.