Nihal Rao, Ke Liu, Marc Machaczek et al.
We use a recently developed interpretable and unsupervised machine-learning method, the tensorial kernel support vector machine (TK-SVM), to investigate the low-temperature classical phase diagram of a generalized Heisenberg-Kitaev-$Γ$ ($J$-$K$-$Γ$) model on a honeycomb lattice. Aside from reproducing phases reported by previous quantum and classical studies, our machine finds a hitherto missed nested zigzag-stripy order and establishes the robustness of a recently identified modulated $S_3 \times Z_3$ phase, which emerges through the competition between the Kitaev and $Γ$ spin liquids, against Heisenberg interactions. The results imply that, in the restricted parameter space spanned by the three primary exchange interactions -- $J$, $K$, and $Γ$, the representative Kitaev material $α$-${\rm RuCl}_3$ lies close to the boundaries of several phases, including a simple ferromagnet, the unconventional $S_3 \times Z_3$ and nested zigzag-stripy magnets. A zigzag order is stabilized by a finite $Γ^{\prime}$ and/or $J_3$ term, whereas the four magnetic orders may compete in particular if $Γ^{\prime}$ is anti-ferromagnetic.