Spin liquid versus long-range magnetic order in the frustrated body-centered-tetragonal lattice

Abstract

We show how spin-liquid (SL) states can be stabilized in a realistic three-dimensional model as a result of frustration. SU(n)-symmetric generalization of the Heisenberg model for quantum spin S operators is used to investigate the frustrated body-centered tetragonal (BCT) lattice with antiferromagnetic interlayer coupling J1 and intralayer first and second-neighbor couplings J2 and J3. By using complementary representations of the spin operators, we study the phase diagram characterizing the ground state of this system. For small n, we find that the most stable solutions correspond to four different families of long-range magnetic orders that are governed by J1, J2, and J3. First, some possible instabilities of these phases are identified for n = 2, in large S expansions, up to the linear spin-wave corrections. Then, using a fermionic representation of the SU(n) spin operators for S = 1/2, we find that purely magnetic orders occur for n 3 while SL solutions are stabilized for n 10. The SL solution governed by J1 breaks the lattice translation symmetry. The modulated SL is associated with a commensurate ordering wave vector (1,1,1). For 4 n 9, we show how the competition between J1, J2, and J3 can turn the magnetically ordered ground state into a SL state. Finally, we discuss the relevance of this scenario for correlated systems with BCT crystal structure.