We consider the hypothesis that Dark Matter (DM) has tree-level interactions only with leptons. Such a framework, where DM recoils against electrons bound in atoms, has been proposed as an explanation for the annually modulated scintillation signal in DAMA/LIBRA versus the absence of a signal for nuclear recoils in experiments like CDMS or XENON10. However, even in such a leptophilic DM scenario there are loop induced DM-hadron interactions, where the photons emitted from virtual leptons couple to the charge of a nucleus. Using a general effective field theory approach we show that, if such an interaction is induced at one or two-loop level, then DM-nucleus scattering dominates over DM-electron scattering. This is because the latter is suppressed by the bound state wave function. One obtains a situation similar to standard DM-nucleus scattering analyses with considerable tension between the results of DAMA and CDMS/XENON10. This conclusion does not apply in the case of pseudoscalar or axial vector coupling between DM and leptons, where the loop diagrams vanish. In this case the explanation of the DAMA signal in terms of DM-electron scattering is strongly disfavored by the spectral shape of the signal. Furthermore, if DM can annihilate into neutrinos or tau leptons, the required cross sections are excluded by many orders of magnitude using the Super-Kamiokande bound on neutrinos from DM annihilations in the Sun.