CMS is a large, general-purpose experiment at the Large Hadron Collider (LHC) at CERN. For its simulation, triggering, data reconstruction and analysis needs, CMS collaborators have developed many millions of lines of C++ code, which are used to create applications run in computer centers around the world. Maximizing the performance and efficiency of the software is highly desirable in order to maximize the physics results obtained from the available computing resources. In the past the code optimization effort in CMS has focused on improving algorithms, basic C++ issues, excessive dynamic memory use and memory footprint. Optimizing software today, on modern multi-/many-core 64bit CPU's and their memory architectures, requires however a more sophisticated approach. This presentation will summarize efforts in CMS to understand how to properly optimize our software for maximum performance on modern hardware. Experience with various tools, lessons learned and concrete results achieved will be described.