In recent years, SiPM photoelectric devices have drawn much attention in the domain of time-of-flight-based Positron emission tomography (TOF-PET). Using them to construct PET detectors with excellent coincidence time resolution (CTR) is always one of research focus. In this paper, a SiPM readout pre-amplifier based on common-base current amplifier structure followed by a Pole-Zero (PZ) compensation network is constructed, and the main factors that affect the timing performance of the PET detector are investigated. By experimental measurement, we found that the CTR is heavily related to the bandwidth of the amplifier, bias voltage of SiPM, comparator threshold, and PZ network parameter. The test setup has two detectors, one with LYSO crystal (3 mm x 3 mm x 10 mm) coupled with a Hamamatsu SiPM (S12642-0404), and the other with LaBr3 coupled to a PMT-R9800. After the optimization of the readout circuit with related factors, the CTR between the two detectors is measured as 266ps FWHM. The test result is a helpful guideline for the readout ASIC chip design in our next step.