State of the art atomic sensors operate near the standard quantum limit (SQL) of projection noise, where the precision scales as the square root of the particle number. Overcoming this limit by using atom-atom entanglement such as spin squeezing is a major goal in quantum metrology. Spin squeezing can be realized by coupling an atomic ensemble to a high-finesse optical resonator, where the resulting collective atom-light interaction allows for both measurement and cavity feedback squeezing. These methods for entangling the atoms are typically non-unitary and generate more anti-squeezing than the minimum prescribed by the uncertainty principle, due to a residual entanglement between the atomic ensemble and undetected probing photons. We find that non-unitarity significantly lowers the potential metrological gain from squeezing in atomic clocks and other quantum sensors.
To generate near-unitary spin squeezing experimentally, we couple an ensemble of approximately 1000 Yb-171 atoms to a high-finesse asymmetric micromirror cavity. A laser pulse induces an effective one-axis twisting Hamiltonian, producing the desired squeezed spin state, while detuning the probing light from atomic and cavity resonance by several linewidths limits the undesirable entanglement between atoms and light. We characterize the produced SSSs by state tomography, directly observing a variance reduction of 9.4(4) dB below the SQL, limited by detection noise. For this level of squeezing, we infer a state area only 30% higher than the limit set by the uncertainty principle, confirming the production of a nearly pure spin squeezed state. This experimental platform will allow for the creation of quantum states with metrologically useful entanglement on the clock transition of Yb-171.