We have analyzed four years of Automated Planet Finder (APF) data to understand the error sources in the measured radial velocities. Our approach combines two data sets, long term measurements on a few standard stars and an intense campaign observing simultaneously observing two stars with both the APF and the PFS on Magellan. We find that, despite the strong similarities between the APF's Levy spectrometer and the PFS, the PFS performs markedly better with a 5 minute intrinsic uncertainty of ~60 cm/s as opposed to the Levy's ~150 cm/s. This error source is likely from the star moving on the aperture and we detail our planned improvements to upgrade the spectrometer. We also find hints of a seasonal zero point shift in the Levy's performance, related to temperature control system of the instrument.