The Careful Puppet Master: Reducing risk and fortifying acceptance testing with Jenkins CI

Apr 14, 2015, 4:30 PM
B250 (B250)



oral presentation Track4: Middleware, software development and tools, experiment frameworks, tools for distributed computing Track 4 Session


Mr Jason Alexander Smith (Brookhaven National Laboratory)


Using centralized configuration management, including automation tools such as Puppet, can greatly increase provisioning speed and efficiency when configuring new systems or making changes to existing systems, reduce duplication of work, and improve automated processes. However, centralized management also brings with it a level of inherent risk: a single change in just one file can quickly be pushed out to thousands of computers and, if that change is not properly and thoroughly tested and contains an error, could result in catastrophic damage to many services, potentially bringing an entire computer facility offline. Change management procedures can -- and should -- be formalized in order to prevent such accidents. However, like the configuration management process itself, if such procedures are not automated, they can be difficult to enforce strictly. Therefore, to reduce the risk of merging potentially harmful changes into our production Puppet environment, we have created an automated testing system, which includes the Jenkins CI tool, to manage our Puppet testing process. This system includes the proposed changes and runs Puppet on a pool of dozens of RHEV VMs that replicate most of our important production services. This paper describes our automated test system and how it hooks into our production approval process for automatic acceptance testing. All pending changes that have been pushed to production must pass this validation process before they can be approved and merged into production.

Primary authors

Mr Gabriel Richman (Oberlin College) Mr Jason Alexander Smith (Brookhaven National Laboratory)


Mr James Pryor (Brookhaven National Laboratory) John Steven De Stefano Jr (Brookhaven National Laboratory) Mr Tejas Rao (Brookhaven National Laboratory) Dr Tony Wong (Brookhaven National Laboratory) Mr William Strecker-Kellogg (Brookhaven National Laboratory)

Presentation materials