We present novel symmetries of perturbation theory around rotating and non-rotating black holes in general relativity, and discuss their origins and implications for gravitational-wave astronomy. This is motivated by two special aspects of black hole perturbations in four dimensions: isospectrality of quasinormal modes and the vanishing of tidal Love numbers. There turn out to be off-shell symmetries underlying both of these phenomena. One is a duality, which on shell reproduces the famous Chandrasekhar duality and therefore underlies isospectrality, and can be thought of as an extension of electric-magnetic duality to black hole backgrounds. The other is an infinite set of "ladder symmetries" relating modes of different angular momentum or spin, which imply the vanishing of Love numbers. We further discuss the geometric origins of these symmetries.