We introduce a novel strategy for machine-learning-based predictive simulators, which can be trained in an unsupervised manner using observed data samples to learn a predictive model of the detector response and other difficult-to-model transformations. Particle physics detectors cannot directly probe fundamental particle collisions. Instead, statistical inference must be used to surmise information about the parameters of the latent theory space, ultimately determining the validity of the theory. High-fidelity simulations, which imitate detector response, are a crucial part of this process. However, these computationally intensive simulations have become a major bottleneck in the pursuit of discovery. Previous machine learning based solutions to this problem seek to replicate data for a fixed theory space, but do not attempt to learn a general mapping from the latent theory space to detected data. Such models will only be able to augment current simulations, limiting their scope and utility. Using Optimal Transport based machine learning techniques, we propose a method for a data-driven, physically meaningful, machine learned simulation which lays the framework for ultimately replacing the current computationally expensive simulations in particle physics.