Primordial black holes can be seeded by large cosmological fluctuations produced during inflation. This happens if the potential for inflation is sufficiently flat in some regions. However, in such regions, the dynamics of the inflaton is dominated by quantum diffusion rather than by classical slow roll. This implies that the standard method to calculate the amplitude of the fluctuations, hence the abundance of black holes, breaks down. We show how a proper calculation of inflationary perturbations that incorporates the effect of quantum diffusion can be performed using the formalism of stochastic inflation. We discuss how the predictions for the primordial black holes abundance change, hence how the constraints on the inflationary potential coming from their non detection are modified.