Axion-like particles with masses in the keV-GeV range have a profound impact on the cosmological evolution of our Universe, in particular on the abundance of light elements produced during Big Bang Nucleosynthesis. The resulting limits are complementary to searches in the laboratory and provide valuable additional information regarding the validity of a given point in parameter space. A potential drawback is that altering the cosmological history may potentially weaken or even fully invalidate these bounds. The main objective of this article is therefore to evaluate the robustness of cosmological constraints on axion-like particles in the keV-GeV region, allowing for various additional effects which may weaken the bounds of the standard scenario. Employing the latest determinations of the primordial abundances as well as information from the cosmic microwave background we find that while bounds can indeed be weakened, very relevant robust constraints remain.