We report on a search for charged Higgs bosons in the mass range 80 < m_H+ < 155 GeV in decays of top quark pairs in ppbar collisions at sqrt{s}=1.96 TeV. It has been performed using ~1 fb-1 of data collected with the D0 detector at the Fermilab Tevatron collider. The search is based on the analysis of ttbar production rates in the final states with two isolated leptons (electrons or muons) and jets, one isolated lepton (electron or muon) and jets and isolated lepton (electron or muon) and tau decaying hadronically. We find no evidence for signal. The results are interpreted within a tauonic Higgs model where the charged Higgs is assumed to decay exclusively to taunu and in a leptophobic model where the charged Higgs is assumed to decay exclusively to csbar. We set upper limits on the branching fractions B(t -> H+ b -> taubar neutrino b)) and B(t -> H+ b -> c sbar b)) for low charged-Higgs masses. We also use ratios of ttbar cross sections in different final states to set upper limits on those branching fractions. Furthermore, we describe a search for quark fusion production qq' --> H+ of the charged Higgs boson reconstructed in the tb final state for a mass range 180 < m_H+ < 300 GeV. We find no evidence for heavy charged Higgs production and set limits on the production cross-section for a variety of theoretical models.