The lepton flavor violation is a clear signature of the new physics and has been searched for by many experiments. Such decays can be searched using large amount of $B$ and $\Upsilon$ mesons produced at Belle. We present our search for $B \to \ell\tau$, $\Upsilon(nS) \to \ell \ell', \ell \tau$ decays (\ell = e, \mu). We also present the final results on $\tau \to \ell \gamma$ at Belle. In addition, we present our search for $B \to K^*\tau\tau$, which could potentially test the lepton flavor universality together with $B \to K^*\ell\ell$. All these results are based on data collected by the Belle experiment at the KEKB asymmetric-energy $e^+e^-$ collider.