In heavy-ion collisions, the highly relativistic ions act as a strong source of electromagnetic radiation, enhanced by the large proton charge number $Z$. Ultra-peripheral collisions (UPC) offer a natural environment in which to observe the photon-initiated production of Beyond Standard Model (BSM) processes with QED couplings. One such process sensitive to BSM effects is light-by-light scattering. This process was directly observed for the first time in UPC events at the LHC by ATLAS. In this talk, the final ATLAS measurements of light-by-light scattering will be presented. These measurements are performed using the full Run-2 dataset which results in substantially reduced uncertainties compared to the previous measurements. They provide a precise and unique opportunity to investigate extensions of the Standard Model, such as presence of axion-like particles. Measurements of tau-pair production via two-photon scattering that investigate the anomalous magnetic moment of the tau lepton are also presented. This process is also potentially sensitive to BSM effects.