AARNet's CloudStor is one of the most successful collaboration tools across the Australian research data ecosystem. We operate a significant software stack across multiple geographically dispersed nodes. Our service end points are exponentially increasing, along with the complexity of the interacting components, and an ever increasing number of end users.
As the AARNet Cloud Services team have been busily scaling our systems up, we've noticed that its been harder and harder to test our changes in a way that accurately reflects our production environments. Complexity introduces challenges with scale, and we have to scale up some of our processes to cope with these challenges.
This talk will cover the road we have started to walk down for quality control, specifically in how we are testing changes, and a mindshift in how we are looking at monitoring. This will including testing environments, chaos monkey environments, infernal testing environments and client focused monitoring and eventually full continuous integration.
The aim with our QA platform is to provide an environment we can rapidly spin up full deployments using Openstack, Kubernetes, Terraform and Ansible. The combination of this rapid deployment, improved monitoring and ability to test multiple complex environments will facilitate the addition of new features and capabilities to our systems. This agility will benefit our interoperability testing with the CS3Mesh, and our deployment testing for the new ownCloud Infinite Scalability deployment.