The origin of the diffuse isotropic gamma-ray back- ground measured by the Fermi gamma-ray satellite at energies between 100 MeV and 820 GeV remains largely uncertain. Population studies indicate that most of the emission originates in a large number of extragalactic objects such as active galactic nuclei, star-forming galaxies, or radio galaxies too faint to be resolved as individual sources. A detailed knowledge of how each population contributes to the overall observed flux is of key importance to understand the density and evolution of these sources with redshift, set constraints on a potential dark matter contribution, and study the link between the Isotropic Gamma-Ray Background (IGRB) and other cosmic backgrounds such as the diffuse neutrino flux discovered by the IceCube observatory. We present preliminary results from a Swift program of follow-up observations of unassociated Fermi-LAT very-high-energy photons observed at high Galactic latitudes. The purpose of this pilot program is to pinpoint the origin of individual VHE high-energy photons contributing to the IGRB to help identify X-ray and optical sources responsible for this flux. Apart from resolving new sources, these observations would be valuable in the identification of candidate VHE gamma-ray sources that could be further studied with ground-based VHE observatories.