Recent optical polarimetry monitoring programs have discovered optical polarization angle swings during blazar flares. Observations have shown that these swings are mostly simultaneous with Fermi gamma-ray flares. While angle swings are found in both flat spectrum radio quasars and BL Lac objects, blazars that have shown swings tend to be more active and brighter in gamma-rays. These features strongly suggest that at least some blazar flares may be driven by magnetic field. Magnetic reconnection is a very promising physical mechanism that can efficiently accelerate nonthermal particles by dissipating magnetic energy. We have performed detailed particle in cell simulations coupled with comprehensive polarization-dependent radiation transfer on magnetic reconnection in blazar emission environment. Our approach uniquely connects first principles with observations. Our results clearly show that magnetic reconnection can drive optical polarization angle swings with Fermi gamma-ray flares. The essential mechanism is the secondary reconnection due to large plasmoid mergers in the reconnection layer. For the first time, we have explicitly shown that the strength of the guide field in the reconnection layer is the key parameter that affects gamma-ray light curves and optical polarization signatures. Specifically, the frequently seen erratic fluctuations in optical polarization signatures during flares are due to reconnection with relatively large guide fields; but if the reconnecting magnetic field lines are almost anti-parallel, it will lead to strong angle swings and highly variable gamma-ray flares. We suggest that Fermi gamma-ray and optical polarimetry together can pinpoint the mechanism of blazar flares and the physical conditions in the emission region.