Harmony constitutes an essential aspect of a broad range of styles in Western music, and chords usually play a key role therein. Consequently, the generation or detection of chords is central to a wide range of computational models, for instance in chord estimation, chord sequence prediction, and harmonic structure detection. Such models are typically evaluated by comparing their outputs to ground-truth chord labels using a binary metric (“correct” or “incorrect”). As chord vocabularies continue to grow, binary metrics capture less information about the correctness of a given label, thus equating all labeling errors regardless of their severity. In this work, we present the chord-eval toolkit, which proposes three different metrics drawn, adapted, and generalized from previous work, addressing acoustic, perceptual, music-theoretical, and mechanical aspects of evaluation. We discuss use cases for which the metrics vary in appropriateness, depending on properties of the underlying music and the task at hand, and present an example of such differences.