Characterizing the magnetic properties of industrial magnetic materials has great significance on the development of transformers, motors and other electrical equipment. Recently, high-frequency and high-power density electrical equipment has attracted more attention. As the frequency is getting higher and higher, the losses of the magnetic core are much higher than that in the power frequency situation. Nanocrystalline and amorphous alloys lead to a significant reduction in the core loss which gives the opportunity to develop high-efficiency devices in high-frequency applications under frequencies from 20 Hz to 20 kHz. For better using of these materials, it is very necessary to have a deep understanding of these materials’ loss behavior under rotational magnetization condition. In this paper, a new magnetizing structure for nanocrystalline alloys rotational core loss measurement is designed and optimized by the 3D-FEM method. The optimization goal is to achieve both the best homogeneity and the highest testing frequency. As the affecting factors are very complex and the 3D-FEM calculation is very time consuming, improved SVM and PSO algorithms are combined in the optimization process. The improvement of PSO is one of the significant parts of this paper. PSO is a kind of swarm intelligence algorithm. It is simple and easy to implement. But it also has the problem of low search accuracy and is easy to fall into local extrema. In this paper, an improved PSO called the velocity-controlled PSO (VCPSO), based on the analysis of the particles’ distribution has been developed and used in the optimization process. As the 3D FEM calculation is very complex, support vector machine (SVM) is used to establish a regression model between the designed parameters of the RSST during the optimizing process. The results well detailed discussed in the full paper.