If dark matter has strong self-interactions, future astrophysical and cosmological observations, together with a clearer understanding of baryonic feedback effects, might be used to extract the velocity dependence of the dark matter scattering rate. To interpret such data, we should understand what predictions for this quantity are made by various models of the underlying particle nature of dark matter. In this paper, we systematically compute this function for fermionic dark matter with light bosonic mediators of vector, scalar, axial vector, and pseudoscalar type. We do this by matching to the nonrelativistic effective theory of self-interacting dark matter and then computing the spin-averaged viscosity cross section nonperturbatively by solving the Schrodinger equation, thus accounting for any possible Sommerfeld enhancement of the low-velocity cross section. In the pseudoscalar case this requires a coupled-channel analysis of different angular momentum modes due to the structure of the effective potential experienced by the dark matter.