Summary: According to common lore, massive elementary higher spin particles lead to inconsistencies when coupled to gravity. However, this scenario was not completely ruled out by previous arguments. In this paper, we show that in a theory where the low energy dynamics of the gravitons are governed by the Einstein-Hilbert action, any finite number of massive elementary particles with spin more than two cannot interact with gravitons, even classically, in a way that preserves causality. This is achieved in flat spacetime by studying eikonal scattering of higher spin particles in more than three spacetime dimensions. Our argument is insensitive to the physics above the effective cut-off scale and closes certain loopholes in previous arguments. Furthermore, it applies to higher spin particles even if they do not contribute to tree-level graviton scattering as a consequence of being charged under a global symmetry such as \(\mathbb{Z}_2\). We derive analogous bounds in anti-de Sitter space-time from analyticity properties of correlators of the dual CFT in the Regge limit. We also argue that an infinite tower of fine-tuned higher spin particles can still be consistent with causality. However, they necessarily affect the dynamics of gravitons at an energy scale comparable to the mass of the lightest higher spin particle. Finally, we apply the bound in de Sitter to impose restrictions on the structure of three-point functions in the squeezed limit of the scalar curvature perturbation produced during inflation.