It is shown that the \(n\times n\) potential matrix \(Q(x)\) with zero diagonal entries in the first-order system \[ -iBy^\prime(x)+Q(x)y=\lambda y,\qquad 0<x<1, \] with \(B=\text{diag}(\lambda_1I_{n_1},\cdots,\lambda_rI_{n_r})\), \(n_1+\cdots+n_r=n\) and \(\sum_{j=1}^r n_j\text{sgn} \lambda_j=0\), is uniquely determined by its spectra under specified number of linearly independent separated boundary conditions. In particular, for \(n=2n_1\) and \(\lambda_1\lambda_2<0\), \(2n_1^2+1\) spectra suffice; in more general situations the potential \(Q(x)\) is also assumed to have an analytic continuation. Similar results are obtained for the uniqueness of the solution to the inverse spectral problem if the potential is known on part of \([0,1]\) and some spectra are given.