Heavy quarks are ideal probes of the QGP matter. To consistently describe the medium evolution and heavy-quark-medium interaction, we improve the linear Boltzmann Transport (LBT) model [Phys. Lett. B777 (2018) 255-259] by modeling QGP as a collection of quasi-particles with equation of state (EOS) fitted to the lattice QCD data. We call this QLBT model, in which the in-medium scatterings of heavy quarks are described using the linear Boltzmann transport that includes both elastic and inelastic energy loss of heavy quarks within the perturbative QCD framework. The temperature dependences of the quasi-particle masses, as well as the strong coupling parameter among them, are systematically extracted from two different sets of lattice EOS (Hot QCD and Wuppertal-Budapest) with a Bayesian statistical analysis method. With all model parameters fixed by the lattice QCD calculations, QLBT naturally provides a good description of the nuclear modification factors and elliptic flow coefficients of heavy mesons at high transverse momenta. Systematical uncertainties arising from applying two different lattice equations of states are evaluated and turn out to be small for heavy flavor observables.