Introduction

• Marwan Khelif (CERN)
• Jose Carlos Luna Duran (CERN)

Room: 31/3-004 - IT Amphitheatre

QA and Testing in CERNBOX: the cornerstone of service development and operation

• Jakub Moscicki (CERN)

Room: 31/3-004 - IT Amphitheatre QA and Testing in CERNBOX (https://cernbox.cern.ch) presents a serious challenge and is critically important: - the service deals with user data directly on their local computers (synchronization clients), at present ~1000 clients connecting daily - the very nature of synchronization is to propagate changes across computers which also means propagating problems if they occur - there is a wide range of supported platforms (MacOSX, Windows, major Linux distributions, mobile platforms) - OS semantics are often incompatible or conflicting: for example HFS is case preserving, NTFS supports the legacy of 8.3 DOS file format, etc. - the operational environment varies enormously, for example: from fast, reliable network inside computing center to unreliable, high-latency, ad-hoc connections from airports - etc. From service development perspective, CERNBOX integrates several complex components which development cycles are disjoint and geographically distributed: - PB-range storage backend (EOS) developed and operated in-house - synchronization clients and web-access layer (ownCloud) developed externally To handle some of the development, integration and operational challenges we have developed a testing framework called Smashbox. In this presentation I will summarize how we use Smashbox for QA and Testing of critical data handling paths of CERNBOX with some real-life examples. References: - Source code: https://github.com/cernbox/smashbox - Mysteries of Dropbox: https://indico.cern.ch/event/336753/session/1/contribution/28 - Smashbox in action: https://owncloud.org/blog/smashbox-in-action

Automated Test Case Generation

• Stefan Klikovits (CERN)

Room: 31/3-004 - IT Amphitheatre I would like to present the concept of automated test case generation. I work on it as part of my PhD and I think it would be interesting also for other people. It is also the topic of a workshop paper that I am introducing in Paris. (abstract below) Please note that the talk itself would be more general and not about the specifics of my PhD, but about the broad field of Automated Test Case Generation. I would introduce the main approaches (combinatorial testing, symbolic execution, adaptive random testing) and their advantages and problems. (oracle problem, combinatorial explosion, ...) Abstract of the paper: Over the last decade code-based test case generation techniques such as combinatorial testing or dynamic symbolic execution have seen growing research popularity. Most algorithms and tool implementations are based on finding assignments for input parameter values in order to maximise the execution branch coverage. Only few of them consider dependencies from outside the Code Under Test’s scope such as global variables, database values and subroutine calls as influences to the execution path. In order to fully test all possible scenarios these dependencies have to be taken into account for the test input generation. This paper introduces ITEC, a tool for automated test case generation to support execution environment resilience in large-scaled, complex systems. One of ITEC’s corner stones is a technique called semi-purification, a source code transformation technique to overcome limitations of existing tools and to set up the required system state for software testing.

DB On Demand 2.0 (alpha)

• Ignacio Coterillo Coz (CERN)

Room: 31/3-004 - IT Amphitheatre The presentation would cover the ongoing efforts and struggles we are facing in our attempt to modernize an existing codebase with large portions of legacy code in the Database On Demand project. It would be an exposition of what we are trying to do, with lots of open questions... Some topics: - Working with legacy code - Refactoring - Automated testing - New tools, new workflows (github,gitlab, jenkins...) - Open sourcing - Compromises Some characteristics of our team that can probably be common around CERN: - Small team, easy to have 'experts' owning sections of code. - Lots of code was written by 'eventual' people not longer around

The quality and testing PH-SFT infrastructure for the external LHC software packages deployment

• Pere Mato Vila (CERN)
• Patricia Mendez Lorenzo (CERN)
• Benedikt Hegner (CERN)

Room: 31/3-004 - IT Amphitheatre ------------------------------------------------------------------------------------------------------------------------------------------------------------ The PH-SFT group is responsible for the build, test, and deployment of the set of external software packages used by the LHC experiments. This set includes ca. 170 packages including Grid packages and Montecarlo generators provided for different versions. A complete build structure has been established to guarantee the quality of the packages provided by the group. This structure includes an experimental build and three daily nightly builds, each of them dedicated to a specific ROOT version including v6.02, v6.04, and the master. While the former build is dedicated to the test of new packages, versions and dependencies (basically SFT internal used), the three latter ones are the responsible for the deployment to AFS of the set of stable and well tested packages requested by the LHC experiments so they can apply their own builds on top. In all cases, a complete build of these packages is provided for different platforms and compilers. In this talk, we will show the build and test infrastructure we provide to the experiments including a description of the specific tools used for this aim.

The certification process of the LHCb distributed computing software

• Christophe Haen (CERN)

Room: 31/3-004 - IT Amphitheatre DIRAC is a software framework for distributed computing, integrating Workload and Data Management. LHCb created DIRAC in 2003. It was then split-off between a VO-agnostic part, released as an open source software in 2009, and and LHCb-specific extension LHCbDIRAC. DIRAC/LHCbDIRAC is able to handle different types of computing, storage, and catalog resources. It has also been adopted by several other Virtual Organizations who are developing their own extensions. Each LHCbDIRAC version has a strict dependency from a DIRAC version and from a set of externally-developed software packages. Before deploying an LHCbDIRAC version in production, the LHCb computing team has to make sure that this version will not disrupt the ongoing activities. DIRAC contains around 200 thousand lines of python code, and LHCbDIRAC around 120 thousand. The testing process for each release consists of a number of steps, that includes static code analysis, unit tests, integration tests, regression tests, system tests. We dubbed the full process as the LHCbDIRAC certification process. We use Jenkins for continuous software integration, but we also submit pilots, jobs, and data to the Grid computing and storage resources. This talk will highlight LHCb achievements obtained with running the certification process. It will also talk about the issues encountered, as well as give hints about the next steps.

Who's gonna test your tests ? Mutants !

• Sebastian Witowski (CERN)

Room: 31/3-004 - IT Amphitheatre This will be a short presentation explaining the idea of *mutation testing* - an interesting approach to testing tests. I will go through the general concept of mutation testing, some simple examples of mutations and point out a few libraries that you can use. Examples will be in Python, however no prior knowledge of Python is required (examples will be dead simple). Presentation should take between 10 and 15 minutes.

Quality assurance for CORAL and COOL within the LCG software stack for the LHC experiments

• Andrea Valassi (CERN)

Room: 31/3-004 - IT Amphitheatre CORAL and COOL are software packages used by the LHC experiments for managing different categories of physics data using a variety of relational database technologies. The core components are written in C++, but Python bindings are also provided. CORAL is a generic relational access layer, while COOL includes the implementation of a specific relational data model and optimization of SQL queries for "conditions data". The software is the result of more than 10 years of development in colaboration between the IT department and the LHC experiments. The packages are built and released within the LCG software stack, for which automatic nightly builds and release installations are provided by PH-SFT (cmake, jenkins, cdash) for many different platforms, compilers and software version configurations. Test-driven development and functional tests of both C++ and Python components (CppUnit, unittest) have been key elements in the success of the projects. Dedicated test suites have also been prepared to commission and maintain the integration of these packages with the LHC experiment framework software. Memory profilers (valgrind, IgProf, gperftools) and static code analyzers (Coverity) are also routinely used for proactive quality assurance. Performance test suites for SQL readback queries also exist for COOL and have been essential for the early detection of issues across Oracle server versions. This talk will give an overview of all these different areas of the CORAL and COOL software quality assurance.

Managing the CMS Online Software integrity through development and production cycles

• Luciano Orsini (CERN)

Room: 31/3-004 - IT Amphitheatre The Data Acquisition system of the Compact Muon Solenoid experiment at CERN is a distributed system made of several different network technologies and computers to collect data from more than 600 custom detector Front-End Drivers. It assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GByte/s. The architecture takes advantage of the latest developments in the computing industry. For data concentration, 10/40 Gbit Ethernet technologies are used while a 56Gbps Infiniband FDR CLOS network has been chosen for the event builder with a throughput of ~4 Tbps. The CMS Online Software (CMSOS) infrastructure is a complex product created specifically for the development of large distributed data acquisition systems as well as all application components to achieve the CMS data acquisition task. It is designed to benefit from different networking technologies, parallelism available on a processing platform such as multi-core or multi-processor systems. It provides platform independent framework that builds upon industrial standards, open protocols and libraries, tools for local and remote inter-process communication, configuration and control. It is designed according to the object-oriented model, and is implemented using the C++ programming language. The processing infrastructure is made scalable by the ability to partition applications into smaller functional units that can be distributed over multiple processing units. The CMSOS is highly modular. It is organized in several interdependent packages that can be individually deployed for operations. The CMSOS product development and the maintenance such as repairing problems and deficiencies, enhancement and new requirements occur in a well-established environment. Achieving a high level of the CMSOS product quality during its lifetime is an iterative process that consists of collective activities like technical and administrative direction and surveillance, whose purpose is to maintain the integrity of the software products as these products evolve from specification through design, development and production cycles (e.g. releases, milestones, baselines, versions, etc.) A cost-effective implementation of an application development process and technology environment is described to address the product evolution with the required quality over time. The provided environment has shown benefits at accounting the complexity issues associated with the software itself, accommodating the problem of transient developers and overcoming different development cultural values. The approach based on Software Configuration Management practices attempts to achieve the right balance between the support to a visible and traceable development and the overhead due to shuffling paperwork. This work is based on the experience of the authors at CMS DAQ group and the experiences of others in different organizations who have been faced with similar concerns.

Experience on QA in the CernVM File System

• Rene Meusel (CERN)
• Jose Molina Colmenero (CERN)

Room: 31/3-004 - IT Amphitheatre The CernVM-File System (CVMFS) delivers experiment software installations to thousands of globally distributed nodes in the WLCG and beyond. In recent years it became a mission-critical component for offline data processing of the LHC experiments and many other collaborations. From a software engineering perspective, CVMFS is a medium-sized C++ system-level project. Following the growth of the project, we introduced a number of measures to improve the code quality, testability, and maintainability. In particular, we found very useful code reviews through github pull requests and automated unit- and integration testing. We are also transitioning to a test-driven development for new features and bug fixes. These processes are supported by a number of tools, such as Google Test, Jenkins, Docker, and others. We would like to share our experience on problems we encountered and on which processes and tools worked well for us.

Experience with C++ Code Quality in ATLAS

• Shaun Roe (CERN)

Room: 31/3-004 - IT Amphitheatre Explanation of tools used for code quality checking in Atlas offline software, including results and a description of developers' responses. Principle tools used: Coverity cppcheck 'include what you use' Undefined Behaviour Sanitizer

Verification steps for the CMS event-builder software

• Remi Mommsen (Fermi National Accelerator Lab. (US))

Room: 31/3-004 - IT Amphitheatre The CMS event-builder software is used to assemble event fragments into complete events at 100 kHz. The data originates at the detector front-end electronics, passes through several computers and is transported from the underground to the high-level trigger farm on the surface. I will present the testing and verifications steps a new software version has to pass before it is deployed in production. I will discuss the current practice and possible improvements.

Continuous Integration and Code Review: how IT can help

• Alexandre Lossent (CERN)

Room: 31/3-004 - IT Amphitheatre We will present recent additions to the IT service portfolio that can help developers implement Continuous Integration and Code Review workflows. The talk will cover Jenkins, GitLab and future plans.

Clean Code - Why you should care

• Benjamin Wolff (CERN)

Room: 31/3-004 - IT Amphitheatre *"Any fool can write code that a computer can understand. Good programmers write code that humans can understand."* - Martin Fowler Writing code is communication, not solely with the computer that executes it, but also with other developers and with oneself. A developer spends a lot of his working time reading and understanding code that was written by other developers or by himself in the past. The readability of the code plays an important factor for the time to find a bug or add new functionality, which in turn has a big impact on the productivity. Code that is difficult to undestand, hard to maintain and refactor, and offers many spots for bugs to hide is not considered to be "clean code". But what could considered as "clean code" and what are the advantages of a strict application of its guidelines? In this presentation we will take a look on some typical "code smells" and proposed guidelines to improve your coding skills to write cleaner code that is less bug prone and better to maintain.

Automated Smoke Testing on the JVM with Geb

• Joao Silva (CERN)

Room: 31/3-004 - IT Amphitheatre The presentation will focus on automated smoke testing in the Java world, and how to prevent catastrophic failures from reaching the production environment. We will show how we have leveraged Geb -- a browser automation tool built on top of Selenium -- to develop a full suite of smoke tests for CERN's Electronic Document Handling (EDH) system -- a large web application with over 1 million lines of Java code, in production since 1998. Though the tool is aimed for use on the JVM, the concepts that will be presented can be applied in other languages that have wrappers for Selenium. In particular, patterns that can be used to better describe and structure functional/smoke tests, as well as how to integrate these tests with continuous integration systems, such as Atlassian Bamboo, using headless browsers (e.g. PhantomJS) or virtual frame buffers.

Achieving dependable software through Continuous Delivery and Quality Monitoring

• Kamil Henryk Krol (CERN)

Room: 31/3-004 - IT Amphitheatre The TE-MPE-MS Software team is a small group of people responsible for developing software for machine protection. Main projects include mostly software for the equipment supervision and testing. The usage of the Scrum methodology involves getting clients’ feedback as soon as possible which pushed the requirement to integrate all the time and deliver software in small chunks. To meet these requirements the team successfully implemented Continuous Integration and Delivery processes using multiple open source and proprietary products including: Gradle, Flyway, JUnit+Mockito, SonarQube and Bamboo. The idea for the presentation is to present our implementation of the CI/CD paradigms and explain on real live examples advantages and drawbacks of the current solution. During the presentation we will try to cover all the required steps which should automatically triggered by a developer’s commit. The presentation should give users a good hands-on experience on basic CI/CD principles and allow them to design and implement simple software delivery platform. ![CI/CD][1] [1]: https://wikis.cern.ch/download/attachments/77562499/continuous%20delivery%20diagram.jpg?version=1&modificationDate=1413356546000&api=v2

The benefits of SOLID in software development

• Ruben Agudo Santos (Universidad de Oviedo (ES))

Room: 31/3-004 - IT Amphitheatre In the talk, the SOLID principles will be presented, every one of them will be explained and some examples will be given and I I will tell why should we use every one of them to improve the quality of our software as well its maintainability and extensibility.

Formal verification of industrial control systems

• Daniel Darvas (CERN / Budapest University of Technology and Economics (HU))

Room: 31/3-004 - IT Amphitheatre Verification of critical software is a high priority but a challenging task for industrial control systems. For many kinds of problems, testing is not an efficient method. Formal methods, such as model checking appears to be an appropriate complementary method. However, it is not common to use model checking in industry yet, as this method needs typically formal methods expertise and huge computing power. In the EN-ICE-PLC section, we are working on a [methodology][1] and a tool ([PLCverif][2]) to overcome these challenges and to integrate formal verification in the development process of our PLC-based control systems. [1]: http://cern.ch/project-plc-formalmethods [2]: http://cern.ch/plcverif

FESA Quality Assurance

• Frederic William Hoguin (CERN)

Room: 31/3-004 - IT Amphitheatre FESA is a framework used by 100+ developers at CERN to design and implement the real-time software used to control the accelerators. Each new version must be tested and qualified to ensure that no backward compatibility issues have been introduced and that there is no major bug which might prevent accelerator operations. Our quality assurance approach is based on code review and a two-level testing process. The first level is made of unit-test (Python unittest & Google tests for C++). The second level consists of integration tests running on an isolated test environment. We also use a continuous integration service (Bamboo) to ensure the tests are executed periodically and the bugs caught early. In the presentation, we will explain the reasons why we took this approach, the results and some thoughts on the pros and cons.

Writing robust C++ code for critical applications

• Miguel Ojeda Sandonis (CERN)

Room: 31/3-004 - IT Amphitheatre **C++** is one of the most **complex**, expressive and powerful languages out there. However, its complexity makes it hard to write **robust** code. When using C++ to code **critical** applications, ensuring **reliability** is one of the key topics. Testing, debugging and profiling are all a major part of this kind of work. In the BE department we use C++ to write a big part of the controls system for beam operation, which implies putting a big focus on system stability and ensuring smooth operation. This talk will try to: - Highlight potential problems when writing C++ code, giving guidelines on writing defensive code that could have avoided such issues - Explain how to avoid common pitfalls (both in writing C++ code and at the debugging & profiling phase) - Showcase some tools and tricks useful to C++ development The attendees' proficiency in C++ should not be a concern. Anyone is free to join, even people that do not know C++, if only to learn the pitfalls a language may have. This may benefit future decisions on the design phase of a project.

How we test the CERN Power Converter Control Libraries (CANCELLED)

• Quentin King (CERN)
• Krzysztof Tomasz Lebioda (AGH University of Science and Technology (PL))

Room: 31/3-004 - IT Amphitheatre The Converter Control Software (CCS) section of the CERN Electrical Power Converter Group (TE-EPC) is responsible for the software that controls the power converters for most of CERNs magnet circuits. In 2010, a project was started to create a set of C libraries that implement the core functions needed for the regulation of current in a magnet circuit, and the generation of reference functions. The first versions of these libraries have been operational in the LHC and the PSB since 2012. Now, with new accelerators such as HIE-ISOLDE and Linac4 being commissioned, additional functionality is needed and a major upgrade to the converter control libraries is ongoing. Two test programs have been developed in parallel with the libraries, one is single-threaded and the other multi-threaded. Both can work interactively or in a batch mode. The multi-threaded program supports commands that allow assertions to be tested, with results reported using Test Anything Protocol. This enables test scripts to be written, which are run automatically when commits are pushed to git as part of the section’s global continuous integration system. After a brief introduction to the libraries, this presentation will show how a simple set of assertions and other supporting commands have allowed an investment in test scripts that will assure the quality of these vital libraries in the long term.