We show how the increase in the Instanton-dyon density can explain both Confinement and Chiral symmetry breaking. We simulate an ensemble of 64 interacting Instanton-dyons for 2 colors and 0 or 2 quark flavors. We find that at low temperatures, the high density of dyons prefer a symmetric density, which leads to the confining value of the Polyakov Loop. At the same time the Chiral condensate is highly sensitive to the Polyakov Loop. As the Polyakov Loop gets close to the confining value, the Chiral condensate develops a non-zero expectation value, thus breaking Chiral symmetry.