We calculate the spectrum of charmed baryons on $32^3\times64$, $2+1$-flavor lattice QCD ensembles generated by the PACS-CS Collaboration. Calculations are done with almost physical light quarks, $m_\pi$ ~ 156 MeV, and physical strange and charm quarks. A relativistic heavy-quark action is used for valance charm quarks to suppress the systematic errors. A two-fold variational analysis is employed in order to access the excited states by varying the interpolating operators and smearing parameters independently. In this talk, we report on the details and status of the current calculations and present some preliminary results for positive and negative parity, spin-$1/2$ and spin-$3/2$ states.