Speaker
Mr
Jason Alexander Smith
(Brookhaven National Laboratory)
Description
Using centralized configuration management, including automation tools such as
Puppet, can greatly increase provisioning speed and efficiency when configuring
new systems or making changes to existing systems, reduce duplication of work,
and improve automated processes. However, centralized management also brings
with it a level of inherent risk: a single change in just one file can quickly
be pushed out to thousands of computers and, if that change is not properly and
thoroughly tested and contains an error, could result in catastrophic damage to
many services, potentially bringing an entire computer facility offline.
Change management procedures can -- and should -- be formalized in order to
prevent such accidents. However, like the configuration management process
itself, if such procedures are not automated, they can be difficult to enforce
strictly. Therefore, to reduce the risk of merging potentially harmful changes
into our production Puppet environment, we have created an automated testing
system, which includes the Jenkins CI tool, to manage our Puppet testing process.
This system includes the proposed changes and runs Puppet on a pool of dozens of RHEV
VMs that replicate most of our important production services. This paper
describes our automated test system and how it hooks into our production
approval process for automatic acceptance testing. All pending changes that have
been pushed to production must pass this validation process before they can be
approved and merged into production.
Primary authors
Mr
Gabriel Richman
(Oberlin College)
Mr
Jason Alexander Smith
(Brookhaven National Laboratory)
Co-authors
Mr
James Pryor
(Brookhaven National Laboratory)
John Steven De Stefano Jr
(Brookhaven National Laboratory)
Mr
Tejas Rao
(Brookhaven National Laboratory)
Dr
Tony Wong
(Brookhaven National Laboratory)
Mr
William Strecker-Kellogg
(Brookhaven National Laboratory)
