A magnet composed by three coaxially nested coils has been developed to generate 100 Tesla pulsed magnetic field at the Wuhan National High Magnetic Field Center (WHMFC). The flywheel generator system has great energy storage density and can control the waveform of the magnet current, therefore the outer coil is energized by a 100 MVA/100 MJ flywheel pulse generator. To reduce the magnet heat and solve the generator output voltage overshoot, the power supply system works in the inverter state after rectification. As the pulse generator power system has rich harmonics and unbalanced three-phase voltage, the speed of adjusting inverse angle and the value of the maximum inverse angle is critical to avoid the inverter failure. In this paper, an appropriate speed of adjusting inverse angle and the optimal inverse angle are derived in detail, which is based on the simplified schematic and some assumptions. Considering that the spike voltage caused by the change of inverse angle affects the safety and stability of the power system, a method of reducing peak voltage by changing excitation voltage is proposed, which restrained peak voltage effectively. When the inverter time exceeds the protection value, the switchgear trips and cuts off the power supply with the magnet into the freewheeling state. simulations and experiments show that this proposed operation mode including rectification, de-excitation, inverter and tripping can meet the power supply needs of the outer coil, improve the efficiency of the pulse power and ensure the safe and stable operation of the system.