Summary: We propose a theory of three-dimensional (anti) de Sitter gravity carrying Chan-Paton color charges. We define the theory by Chern-Simons formulation with the gauge algebra \((\mathfrak{gl}_2\oplus\mathfrak{gl}_2)\otimes\mathfrak{u}(N)\), obtaining a color-decorated version of interacting spin-one and spin-two fields. We also describe the theory in metric formulation and show that, among \(N^2\) massless spin-two fields, only the singlet one plays the role of metric graviton whereas the rest behave as colored spinning matter that strongly interacts at large \(N\). Remarkably, these colored spinning matter acts as Higgs field and generates a non-trivial potential of staircase shape. At each extremum labelled by \(k=0,\dots,[\frac{N-1}{2}]\), the \(\mathfrak{u}(N)\) color gauge symmetry is spontaneously broken down to \({\mathfrak u}(N-k)\oplus_2{\mathfrak u}(k)\) and provides different (A)dS backgrounds with the cosmological constants \((\frac N{N-2k})^2\Lambda\). When this symmetry breaking takes place, the spin-two Goldstone modes combine with (or are eaten by) the spin-one gauge fields to become partially-massless spin-two fields. We discuss various aspects of this theory and highlight physical implications.