If dark matter is asymmetric, bosonic, and non-self-interacting, or asymmetric, fermionic, and self-interacting, it could destroy pulsars at the galactic center by forming black holes inside of them. In this case, a measurable maximum attainable pulsar age would track the density of the dark matter halo, with the oldest pulsars being allowed in the least dense parts of the halo. This could explain a recent observation, that there are not as many pulsars in the galactic center as expected.