# 20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013)

Europe/Amsterdam
Amsterdam, Beurs van Berlage

#### Amsterdam, Beurs van Berlage

Damrak 243 1012 ZJ AMSTERDAM
Description

The International Conference on Computing in High Energy and Nuclear Physics (CHEP) is a major series of international conferences for physicists and computing professionals from the high energy and nuclear physics community, computer science, and information technology. CHEP provides an international forum to exchange the experiences and needs of the community, and to review recent, ongoing, and future activities.

CHEP2013 is organized by Nikhef, the Dutch National Institute for Sub-atomic Physics, in collaboration with partners.

CHEP2013 Logistics Management
• Monday, October 14
• Plenaries Effectenbeurszaal

### Effectenbeurszaal

#### Amsterdam, Beurs van Berlage

Conveners: David Groep (NIKHEF (NL)), Dr Jerome LAURET (BROOKHAVEN NATIONAL LABORATORY)
• 1
Welcome
• 2
CHEP in Amsterdam: from 1985 to 2013
Speaker: David Groep (NIKHEF (NL))
• 3
Nikhef, the national institute for subatomic physics
Speaker: Frank Linde (NIKHEF (NL))
• 4
C++ evolves!
High Energy Physics is unthinkable without C++. But C++ is not the language it used to be: today it evolves continuously to respond to new requirements, and to benefit from the streamlined delivery process of new language features to compilers. How should HEP react? After a short, subjective overview of parallel languages and extensions, the main features of C++11 will be presented, including the new concurrency model. A simple migration strategy for HEP will be discussed. A second theme will focus on structs-of-arrays and limits of auto-vectorization. The evolution of C++ including vectorization and concurrency will be presented.
Speaker: Axel Naumann (CERN)
• 10:30 AM
Coffee Grote zaal

### Grote zaal

#### Amsterdam, Beurs van Berlage

• Plenaries: II Effectenbeurszaal

### Effectenbeurszaal

#### Amsterdam, Beurs van Berlage

Convener: Frederic Hemmer (CERN)
• 5
Software engineering for science at the LSST
Many of the scientific computing frameworks used in 'big science' have several million lines of source code, and software engineering challenges are amongst the most prominent challenges, be it in high-energy physics, astronomy, or other sciences. Dr Robert Lupton of Princeton University will talk the software engineering challenges that face scientific computing and how large scale systems like the Large Synoptic Survey Telescope LSST and others deal with these challenges.
Speaker: Dr Robert Lupton (Princeton)
• 6
Horizon 2020: an EU perspective on data and computing infrastructures for research
Through joint efforts between the HEP community in the early days of the EU DataGrid project, through EGEE, and via EGI-InSPIRE today, the European Commission has had a profound impact in the way computing and data management for high energy physics is done. Kostas Glinos, Head of Unit eInfrastructures of the European Commission, has been with the European Commission since 1992. He leads the eInfrastructures unit of the Directorate General for Communications, Networks, Content and Technology since 1 January 2009. From 2003 to 2008 he was Head of the Embedded Systems and Control unit and interim Executive Director of the ARTEMIS Joint Undertaking. Previously he was deputy head of Future and Emerging Technologies. Before joining the Commission Kostas worked with multinational companies and research institutes in the U.S., Greece and Belgium. He holds a diploma in Chemical Engineering from the University of Thessaloniki, a PhD from the University of Massachusetts and a MBA in investment management from Drexel University. As identified e.g. by the High Level Expert Group in their "Riding the Wave" report (October 2010), the emergence of "big data" in research triggered by ever more powerful instruments (in HEP, bioinformatics, astronomy, etc.) demands advanced computing resources and software to increase the available capacity to manage, store and analyse extremely large, heterogeneous and complex datasets. We should support integrated, secure, permanent, on-demand service-driven and sustainable e-infrastructures to tackle this challenge. Our vision is a pan-European e-infrastructure that will cover the whole lifecycle of scientific data and provide the necessary computing resources needed by researchers to process information. All components of the e-infrastructure should expose standard interfaces to enable interoperation with other relevant e-infrastructures.
Speaker: Dr Kostas Glinos (European Commission)
• 7
The KPMG Challenge
Speaker: Sander Klous (N)
• 12:35 PM
Lunch Grote zaal

### Grote zaal

#### Amsterdam, Beurs van Berlage

• Data Acquisition, Trigger and Controls: I Verwey Kamer

### Verwey Kamer

#### Amsterdam, Beurs van Berlage

Convener: Niko Neufeld (CERN)
• 8
The LHCb Data Acquisition during LHC Run 1
The LHCb Data Acquisition system reads data from over 300 read-out boards and distributes them to more than 1500 event-filter servers. It uses a simple push-protocol over Gigabit Ethernet. After filtering, the data is consolidated into files for permanent storage using a SAN-based storage system. Since the beginning of data-taking many lessons have been learned and the reliability and robustness of the system has been greatly improved. We report on these changes and improvements, their motivation and how we intend to develop the system for Run 2. We also will report on how we try to optimise the usage of CPU resources during the running of the LHC ("deferred triggering") and the implications on the data acquisition.
Speaker: Rainer Schwemmer (CERN)
• 9
The artdaq Data Acquisition Software Toolkit
The artdaq data acquisition software toolkit has been developed within the Fermilab Scientific Computing Division to meet the needs of current and future experiments. At its core, the toolkit provides data transfer, event building, and event analysis functionality, the latter using the art event analysis framework. In the last year, functionality has been added to the toolkit in the areas of state behavior, process control, data quality monitoring, and system monitoring with the goal of providing a complete DAQ software toolkit for future experiments. In addition, the toolkit has been used in the construction of the DAQ system of the DarkSide-50 dark matter experiment and the prototype DAQ for the Mu2e rare decay experiment. We will present the design and features of the toolkit, the advantages of using the toolkit to construct the DAQ software for an experiment, representative performance results, and future plans.
Speaker: Jim Kowalkowski (Fermilab)
• 10
The IceCube Neutrino Observatory DAQ and Online System
The IceCube Neutrino Observatory is a cubic kilometer-scale neutrino detector built into the ice sheet at the geographic South Pole. The online system for IceCube comprises subsystems for data acquisition, online filtering, supernova detection, and experiment control and monitoring. The observatory records astrophysical and cosmic ray events at a rate of approximately 3 kHz and selects the most interesting events for transmission via satellite to the experiment’s data warehouse in the northern hemisphere. The system has been designed to run in a remote environment with minimal operator intervention. Its user interface permits remote control and monitoring of the experiment both locally and via satellite. Despite the remote location and complexity of the various subsystems interoperating, the system as a whole achieves an uptime in excess of 98%. We describe the design and implementation of the core detector online systems: the Data Acquisition Software (DAQ), including the in-ice and surface optical modules, the triggering system, and event builder; the distributed Processing and Filtering (PnF) system; the IceCube Live control and monitoring system; and SPADE, the data archival and transport system.
Speaker: Kael Hanson (Université Libre de Bruxelles)
• 11
Algorithms, performance, and development of the ATLAS High-level trigger
The ATLAS trigger system has been used for the online event selection for three years of LHC data-taking and is preparing for the next run. The trigger system consists of a hardware level-1 (L1) and a software high-level trigger (HLT). The high-level trigger is currently implemented in a region-of-interest based level-2 (L2) stage and a event filter (EF) operating after even building with offline-like software. During the past three years, the luminosity and pile-up (number of collisions per beam crossing) has increased significantly placing escalating demands on the rejection and timing performace. The HLT algorithms advanced during this period to maintain and even improve performance. For the next run, the boundary between the L2 and EF will be removed, so that there is only one high-level trigger which can operate either on regions of interest or on the full event depending on the objects found in the event either by the L1 or by the HLT itself. This talk will discuss the algorithms, performance and ongoing development work on the reconstruction of calorimeter objects electrons, photons, taus, jets, and missing energy), inner detector tracking, and muon reconstruction. Among the improvements is a new missing energy trigger which uses specialized sums of the calorimeter cells to access the calorimeter readout earlier than was previously possible with strict region-of-interest only L2 system. Another improvement is a jet scan algorithm which operates at L2 using the information from the L1 digitization, but applies a clustering algorithm (anti-kT ) similar to that used in the offline software. The jet and b-jet algorithms have been further developed to more closely resemble and include improvements from the offline software. Also discussed will be the work towards the merging of the two HLT levels in to a single level HLT, as well as operational experiences from the first LHC run.
Speaker: Kunihiro Nagano (High Energy Accelerator Research Organization (JP))
• Data Stores, Data Bases, and Storage Systems: I Administratiezaal

#### Amsterdam, Beurs van Berlage

• 12
ECFS: A decentralized, distributed and fault-tolerant FUSE filesystem for the LHCb online farm
The LHCb experiment records millions of proton collisions every second, but only a fraction of them are useful for LHCb physics. In order to filter out the "bad events" a large farm of x86-servers (~2000 nodes) has been put in place. These servers boot from and run from NFS, however they use their local disk to temporarily store data, which cannot be processed in real-time ("data-deferring"). These events are subsequently processed, when there are no live-data coming in. The effective CPU power is thus greatly increased. This gain in CPU power depends critically on the availability of the local disks. For cost and power-reasons, mirroring (RAID-1) is not used, leading to a lot of operational headache with failing disks and disk-errors or server failures induced by faulty disks. To mitigate these problems and increase the reliability of the LHCb farm, while at same time keeping cost and power-consumption low, an extensive research and study of existing highly available and distributed file systems has been done. While many distributed file systems are providing reliability by "file replication", none of the evaluated one supports erasure algorithms. A decentralised, distributed and fault-tolerant "write once read many" file system has been designed and implemented as a proof of concept providing fault tolerance without using expensive - in terms of disk space - file replication techniques and providing a unique namespace as a main goals. This paper describes the design and the implementation of the Erasure Codes File System (ECFS) and presents the specialised FUSE interface for Linux. Depending on the encoding algorithm ECFS will use a certain number of target directories as a backend to store the segments that compose the encoded data. When target directories are mounted via nfs/autofs - ECFS will act as a file-system over network/block-level raid over multiple servers.
Speaker: Tomasz Rybczynski (AGH University of Science and Technology (PL))
• 13
Fuzzy Pool Balance: An algorithm to achieve two dimensional balances in distribute storage systems
The limitation of scheduling modules and the gradual addition of disk pools in distributed storage systems often result in imbalances among their disk pools in terms of both available space and number of files. This can cause various problems to the storage system such as single point of failure, low system throughput and imbalanced resource utilization and system loads. An algorithm named Fuzzy Pool Balance (FPB) is proposed here to solve this problem. The input of FPB is the current file distribution among disk pools and the output is a file migration plan indicating what files are to be migrated to which pools. FPB uses an array to classify the files by their sizes. The file classification array is dynamically calculated with a defined threshold named Tmax which defines the allowed available space deviations of disk pools. File classification is the basis of file migration. FPB also defines the Immigration Pool (IP) and Emigration Pool (EP) according to the available space of the disk pools and File Quantity Ratio (FQR) which indicates the percentage of each category of files in each disk pool, so files with higher FQR in an EP will be migrated to IP(s) with a lower FQR of this file category. To verify this algorithm, we implemented FPB on an ATLAS Tier2 dCache production system which hosts 12 distributed disk pools with 300TB of storage space. The results show that FPB can achieve a very good balance among the disk pools, and a tradeoff between available space and file quantity can be achieved by adjusting the threshold value Tmax and the correction factor to the average FQR.
Speakers: Prof. Gang CHEN (INSTITUTE OF HIGH ENERGY PHYSICS), Dr Wenjing Wu (IHEP, CAS)
• 14
Disk storage at CERN: handling LHC data and beyond
Data Storage and Services (DSS) group at CERN stores and provides access to the data coming from the LHC and other physics experiments. We implement specialized storage services to provide tools for an optimal data management, based on the evolution of data volumes, the available technologies and the observed experiment and users usage patterns. Our current solutions are CASTOR for highly-reliable tape-backed storage for heavy-duty Tier-0 workflows and EOS for disk-only storage for full-scale analysis activities. CASTOR has been the main physics storage system at CERN since 2001 and successfully caters for the LHC experiments' needs, storing 90 PB of data and more than 350 M files. During the last LHC run CASTOR was routinely storing 1 PB/week of data to tape. CASTOR is currently evolving towards a simplified disk layer in front of the tape robotics, focusing on recording the primary data from the detectors. EOS is now a well established storage service used intensively by the four big LHC experiments, holding over 15 PB of data and more than 130M files (30 PB usable disk space expected at the end of the year). Its conceptual design based on multi-replica and in-memory namespace make it the perfect system for data intensive workflows and its usage will expand via a shared instance for non-LHC experiments. In the short term EOS usage will absorb most of the newly installed capacity at CERN and expand via a shared instance for the non-LHC experiments. An additional challenge will be to run this service across two geographically different sites (CERN Geneva and Budapest). LHC-Long Shutdown 1 presents a window of opportunity to shape up both of our storage services and validate against the ongoing analysis activity in order to successfully face the new LHC data taking period in 2015. Besides summarizing the current state and foreseen evolutions, the talk will focus on the detailed analysis of the operational experience of both systems, in particular service efficiency, performance and reliability.
Speaker: Xavier Espinal Curull (CERN)
• 15
Forming an ad-hoc nearby storage framework, based on the IKAROS architecture and social networking services
Speaker: Christos Filippidis (Nat. Cent. for Sci. Res. Demokritos (GR))
• Distributed Processing and Data Handling A: Infrastructure, Sites, and Virtualization: I Graanbeurszaal

### Graanbeurszaal

#### Amsterdam, Beurs van Berlage

• 16
Towards a Global Service Registry for the World-wide LHC Computing Grid
The WLCG information system is just one of the many information sources that are required to populate a VO configuration database. Other sources include central portals such as the GOCDB and the OIM from EGI and OSG respectively. Providing a coherent view of all this information that has been synchronized from many different sources is a challenging activity and has been duplicated to various extents by each of the LHC experiments. The WLCG Global Service Registry address these issues by aggregating information from multiple information sources and presenting a consolidated view of both pledged and available resources. It aims to help the LHC experiments populate their own VO configuration databases, used for job submission and storage management, by providing them with a single point for obtaining information on WLCG resources. In addition, in-depth validation checks are incorporated into a wider system-wide strategy to ensure the information is of the highest quality. It is hoped that this service will decouple the LHC experiments from the underlying blocks of the WLCG information system, making it easier to evolve the system in future. This paper presents the WLCG Global Service Registry architecture, its advantages compared to the current approach and how can be used to populate a VO configuration database.
Speaker: Alessandro Di Girolamo (CERN)
• 17
Reliability Engineering analysis of ATLAS data reprocessing campaigns
During three years of LHC data taking, the ATLAS collaboration completed three petascale data reprocessing campaigns on the Grid, with up to 2 PB of data being reprocessed every year. In reprocessing on the Grid, failures can occur for a variety of reasons, while Grid heterogeneity makes failures hard to diagnose and repair quickly. As a result, Big Data processing on the Grid must tolerate a continuous stream of failures, errors and faults. While ATLAS fault-tolerance mechanisms improve the reliability of Big Data processing in the Grid, their benefits come at costs and result in delays making the performance prediction difficult. Reliability Engineering provides a framework for fundamental understanding of the Big Data processing on the Grid, which is not a desirable enhancement but a necessary requirement. In ATLAS, cost monitoring and performance prediction became critical for the success of the reprocessing campaigns conducted in preparation for the major physics conferences. In addition, our Reliability Engineering approach supported continuous improvements in data reprocessing throughput during LHC data taking. The throughput doubled in 2011 vs. 2010 reprocessing, then quadrupled in 2012 vs. 2011 reprocessing. We present the Reliability Engineering analysis of ATLAS data reprocessing campaigns providing the foundation needed to scale up the Big Data processing technologies beyond the petascale.
Speaker: Dmytro Karpenko (University of Oslo (NO))
• 18
Evaluating Google Compute Engine with PROOF
The advent of private and commercial cloud platforms has opened the question of evaluating the cost-effectiveness of such solution for computing in High Energy Physics . Google Compute Engine (GCE) is a IaaS product launched by Google as an experimental platform during 2012 and now open to the public market. In this contribution we present the results of a set of CPU-intensive and I/O-intensive tests we have run with PROOF on a GCE resources made available by Google for test purposes. We have run tests on large scale PROOF clusters (up to 1000 workers) to study the overall scalability of coordinated multi-process jobs. We encountered the known scalability limitation of the PROOF technology in single master mode and we have investigated solution to lift this limitation. We have studied and compared the performance of ephemeral and persistent storage with PROOF-Lite on the single machines and of standard PROOF on the whole cluster. We will discuss our results in perspective, in particular with respect to the typical analysis needs of an LHC experiment.
Speaker: Gerardo Ganis (CERN)
• 19
User Centric Job Monitoring – a redesign and novel approach in the STAR experiment
User Centric Monitoring (or UCM) has been a long awaited feature in STAR, whereas programs, workflows and system “events” could be logged, broadcast and later analyzed. UCM allows to collect and filter available job monitoring information from various resources and present it to users in a user-centric view rather than and administrative-centric point of view. The first attempt and implementation of ”a” UCM approach was made in STAR 2004 using a log4cxx plug-in back-end and then further evolved with an attempt to push toward a scalable database back-end (2006) and finally using a Web-Service approach (2010, CSW4DB SBIR). The latest showed to be incomplete and not addressing the general (evolving) needs of the experiment where streamlined messages for online (data acquisition) purposes as well as the continuous support for the data mining needs and event analysis need to coexists and unified in a seamless approach. The code also revealed to be hardly maintainable. This work will present the next evolutionary step of the UCM toolkit, a redesign and redirection of our latest attempt acknowledging and integrating recent technologies and a simpler, maintainable and yet scalable manner. The extended version of the job logging package is built upon three-tier approach based on Task, Job and Event, and features a Web-Service based logging API, responsive AJAX-powered user interface, and database back-end relying on MongoDB, which seems to be uniquely suited for STAR needs. In addition, we present details on integration of this logging package with STAR offline and online software frameworks. Leveraging on the reported experience and work from the ATLAS and CMS experience on using the ESPER engine, we will discuss and show how such approach has been implemented in STAR for meta-data event triggering stream processing and filtering. An ESPER based solution seems to fit well into the online data acquisition system where many systems are monitored.
Speaker: Dr Jerome LAURET (BROOKHAVEN NATIONAL LABORATORY)
• Distributed Processing and Data Handling B: Experiment Data Processing, Data Handling and Computing Models: I Keurzaal

### Keurzaal

#### Amsterdam, Beurs van Berlage

• 20
Recent and planned changes to the LHCb computing model
The LHCb experiment has taken data between December 2009 and February 2013. The data taking conditions and trigger rate have been adjusted several times to make optimal use of the luminosity delivered by the LHC and to extend the physics potential of the experiment. By 2012, LHCb was taking data at twice the instantaneous luminosity and 2.5 times the high level trigger rate than originally foreseen. This represents a considerable increase in the amount of data to be handled compared to the original Computing Model from 2005, both in terms of compute power and in terms of storage. In this paper we describe the changes that have taken place in the LHCb computing model during the last 2 years of data taking to process and analyse the increased data rates within limited computing resources. In particular a quite original change was introduced at the end of 2011 when LHCb started to use for reprocessing compute power that was not co-located with the RAW data, namely using Tier2 sites and private resources. The flexibility of the LHCbDirac Grid interware allowed to easily include these additional resources that in 2012 provided 40% of the compute power for the end-of-year reprocessing. Several changes were also implemented on the Data Management model in order to limit the need for accessing data from tape, as well as in the data placement policy in order to cope with a large imbalance in storage resources at Tier1 sites. We also discuss changes that are being implemented during the LHC Long Shutdown 1 to prepare for a further doubling of the data rate when the LHC restarts at a higher energy in 2015.
Speaker: Marco Cattaneo (CERN)
• 21
First Production with the Belle II Distributed Computing System
The Belle II experiment, a next-generation B factory experiment at KEK, is expected to record a two orders of magnitude larger data volume than its predecessor, the Belle experiment. The data size and rate are comparable to or more than the ones of LHC experiments and requires to change the computing model from the Belle way, where basically all computing resources were provided by KEK, to a more distributed scheme. We exploit existing grid technologies, like DIRAC for the management of jobs and AMGA for the metadata catalog, to build a distributed computing infrastructure for the Belle II experiment. The system provides an abstraction layer for collections of jobs, called a project, and collections of files in a dataset. This year we could demonstrate for the first time the viability of our system in a generation, simulation, and reconstruction of 60M events on several grid sites. The results of this Monte-Carlo production campaign and the further plans for the distributed computing system of the Belle II experiment are presented in this talk.
Speaker: Thomas Kuhr (KIT)
• 22
Evolution of the ATLAS Distributed Computing system during the LHC Long Shutdown
The ATLAS Distributed Computing project (ADC) was established in 2007 to develop and operate a framework, following the ATLAS computing model, to enable data storage, processing and bookkeeping on top of the WLCG distributed infrastructure. ADC development has always been driven by operations and this contributed to its success. The system has fulfilled the demanding requirements of ATLAS, daily consolidating worldwide up to 1PB of data and running more than 1.5 million payloads distributed globally, supporting almost one thousand concurrent distributed analysis users. Comprehensive automation and monitoring minimized the operational manpower required. The flexibility of the system to adjust to operational needs has been important to the success of the ATLAS physics program. The LHC shutdown in 2013-2015 affords an opportunity to improve the system in light of operational experience and scale it to cope with the demanding requirements of 2015 and beyond, most notably a much higher trigger rate and event pileup. We will describe the evolution of the ADC software foreseen during this period. This includes consolidating the existing Production and Distributed Analysis framework (PanDA) and ATLAS Grid Information System (AGIS), together with the development and commissioning of next generation systems for distributed data management (DDM/Rucio) and production (PRODSYS2). We will explain how new technologies such as Cloud Computing and NoSQL databases, which ATLAS investigated as R&D projects in past years, will be integrated in production. Finally, we will describe more fundamental developments such as breaking job-to-data locality by exploiting storage federations and caches, and event level (rather than file or dataset level) workload engines.
Speaker: Simone Campana (CERN)
• 23
CMS Computing Model Evolution
The CMS Computing Model was developed and documented in 2004. Since then the model has evolved to be more flexible and to take advantage of new techniques, but many of the original concepts remain and are in active use. In this presentation we will discuss the changes planned for the restart of the LHC program in 2015. We will discuss the changes planning in the use and definition of the computing tiers, that were defined with the MONARC project. We will present how we intend to use new services and infrastructure to provide more efficient and transparent access to the data. We will discuss the computing plans to make better use of the computing capacity by scheduling more of the processor nodes, making better use of the disk storage, and more intelligent use of the networking. Speakers
Speaker: Claudio Grandi (INFN - Bologna)
• Event Processing, Simulation and Analysis: I Berlagezaal

### Berlagezaal

#### Amsterdam, Beurs van Berlage

• 24
Track Reconstruction at the ILC
One of the key requirements for Higgs physics at the International Linear Collider ILC is excellent track reconstruction with very good momentum and impact parameter resolution. ILD is one of the two detector concepts at the ILC. Its central tracking system comprises of a highly granular TPC, an intermediate silicon tracker and a pixel vertex detector, and it is complemented by silicon tracking discs in the forward direction. Large hit densities from beam induced incoherent electron-positron pairs at the ILC pose an additional challenge to the pattern recognition algorithms. We present the ILD tracking algorithms that are using clustering techniques, cellular automatons and kalman filter based track extrapolation. The performance of the tracking reconstruction is evaluated using a realistic geant4 simulation including dead material, gaps and imperfections, that recently has been used for a large Monte Carlo production for the Detailed Baseline Design of the ILD detector concept. The algorithms are written to a large extend in a framework independent way, with the eventual goal of providing a generic tracking toolkit. Parallelization techniques for some of the algorithms are under investigation.
Speaker: Frank-Dieter Gaede (Deutsches Elektronen-Synchrotron (DE))
• 25
Track extrapolation and muon identification using GEANT4E in event reconstruction in the Belle II experiment
I will describe the charged-track extrapolation and the muon identification modules in the Belle II data analysis code library. These modules use GEANT4E to extrapolate reconstructed charged tracks outward from the Belle II Central Drift Chamber into the outer particle-identification detectors, the electromagnetic calorimeter, and the K-long and muon (KLM) detector embedded in the iron yoke surrounding the Belle II solenoid. using the detailed detector geometry that was developed for the simulation module. The extrapolation module propagates the position, momentum, 6-dimensional covariance matrix, and time of flight from the interaction point to permit comparison of the extrapolated track with the hits detected in the outer detectors. In the course of track extrapolation into the KLM, a Kalman fitting procedure is applied that adjusts the track parameters using the matching hits in each of the crossed detectors. The muon identification procedure then compares the longitudinal and transverse profiles of the extrapolation and the matched hits in the KLM and, for the low-momentum tracks, the extrapolated and matched crystals in the electromagnetic calorimeter, to distinguish between the muon and hadron-like hypotheses. Several modifications were made to permit GEANT4E to interoperate with GEANT4 and to expand the number of particle species that can be extrapolated.
Speaker: Leo Piilonen (Virginia Tech)
• 26
The Role of Effective Event Reconstruction in the Higgs Boson Discovery at CMS
In 2012 the LHC increased both the beam energy and intensity. The former made obsolete all of the simulation data generated for 2011; the latter increased the rate of multiple proton-proton collisions (piluep) in a single event, significantly increasing the complexity of both the reconstructed and matching simulated events. Once the pileup surpassed 10, the resources needed for the software to function created significant strain on CMS computing facilities. Problems with increasing memory and CPU use had to be alleviated in a way that did not sacrifice the physics performance of the reconstruction. In 2012 this was particularly important as the prompt calibration system was fully commissioned, making the data produced in the prompt reconstruction, the primary datasets used in 2012 physics publications on 8Tev data. This paper summarizes the changes applied to the CMS data reconstruction software, which was deployed successfully and delivered high quality data used in the Higgs boson discovery and many other physics results from CMS.
Speaker: Slava Krutelyov (Texas A & M University (US))
• Facilities, Infrastructures, Networking and Collaborative Tools: I Veilingzaal

### Veilingzaal

#### Amsterdam, Beurs van Berlage

• 27
ATLAS Cloud Computing R&D
The computing model of the ATLAS experiment was designed around the concept of grid computing and, since the start of data taking, this model has proven very successful. However, new cloud computing technologies bring attractive features to improve the operations and elasticity of scientific distributed computing. ATLAS sees grid and cloud computing as complementary technologies that will coexist at different levels of resource abstraction, and two years ago created an R&D working group to investigate the different integration scenarios. The ATLAS Cloud Computing R&D has been able to demonstrate the feasibility of offloading work from grid to cloud sites and, as of today, is able to integrate transparently various cloud resources into the PanDA workload management system. The ATLAS Cloud Computing R&D is operating various PanDA queues on private and public resources and has provided several hundred thousand CPU days to the experiment. The HammerCloud grid site testing framework is used to evaluate the performance of cloud resources and, where appropriate, compare it with the performance of bare metal to measure virtualization penalties. As a result, the ATLAS Cloud Computing R&D group has gained a deep insight into the cloud computing landscape and has identified points that still need to be addressed in order to fully profit from this young technology. This contribution will explain the cloud integration models that are being evaluated and will discuss ATLAS' learning during the collaboration with leading commercial and academic cloud providers.
Speaker: Dr Randy Sobie (University of Victoria (CA))
• 28
Nagios and Arduino integratition for monitoring
The Datacenter at the Galician Institute of High Energy Physics(IGFAE) of the Santiago de Compostela University (USC) is a computing cluster with about 150 nodes and 1250 cores that hosts the LHCb Tiers 2 and 3. In this small datacenter, and of course in similar or bigger ones, it is very important to keep optimal conditions of temperature, humidity and pressure. Therefore, it is a necessity to monitor the environment and be able to trigger alarms when operating outside the recommended settings. There are currently a plenty of tools and systems developed for Datacenter monitorization, but until recent years all of them were of comercial nature and expensive. In recent years there has been and increasing interest in the use of technologies based on Arduino due to its open hardware licensing and the low cost of this type of components. In this article we describe the system developed to monitorize IGFAE's Datacenter, which integrates an Arduino controlled sensor network with the Nagios monitoring software. Sensors of several types, temperature, humidity or pressure, are connected to the Arduino board. The nagios software is in charge of monitoring the different sensors, with the help of nagiosgraph to keep track of the historic data and to produce the plots. An arduino program, developed in house, provides the nagios sensor with the readout of one or several sensors depending on the sensor's request. The nagios temperature sensor also broadcasts an SNMP trap when the temperature gets out of the allowed operating range.
Speakers: Marcos Seco Miguelez (Universidade de Santiago de Compostela (ES)), Victor Manuel Fernandez Albor (Universidade de Santiago de Compostela (ES))
• 29
SynapSense Wireless Environmental Monitoring System of the RHIC & ATLAS Computing Facility at BNL
RHIC & ATLAS Computing Facility (RACF) at BNL is a 15000 sq. ft. facility hosting the IT equipment of the BNL ATLAS WLCG Tier-1 site, offline farms for the STAR and PHENIX experiments operating at the Relativistic Heavy Ion Collider (RHIC), BNL Cloud installation, various Open Science Grid (OSG) resources, and many other small physics research oriented IT installations. The facility originated in 1990 and grew steadily up to the present configuration with 4 physically isolated IT areas with a combined rack capacity of about 2000 racks and the total peak power consumption of 1.5 MW. Since the infrastructural components of the RACF were deployed over such a long period of time (the oldest parts of physical infrastructure were built in late 1960s while the newest ones were added in 2010) a multitude of various environmental monitoring systems were eventually inherited in different areas. These various groups of equipment that in the end required costly maintenance and support were lacking a high level integration mechanism as well as a centralized web interface. In June 2012 a project was initiated with the primary goal to replace all these environmental monitoring systems with a single commercial hardware and software solution by SynapSense Corp. based on wireless sensor groups and proprietary SynapSense MapSense (TM) software that offers a unified solution for monitoring the temperature and humidity within the rack/CRAC units as well as pressure distribution underneath the raised floor across the entire facility. The new system also supports a set of additional features such as capacity planning based on measurements of total heat load, power consumption monitoring and control, CRAC unit power consumption optimization based on feedback from the temperature measurements and overall power usage efficiency estimations that are not currently implemented within RACF but may be deployed in the future. This contribution gives a detailed review of all the stages of deployment of the system and its integration with the existing personnel notification mechanisms and emergency management/disaster recovery protocols of the RACF. The experience gathered while operating the system is summarized and a comparative review of functionality/maintenance costs for the RACF environmental monitoring system before and after transition is given.
Speakers: Mr Alexandr Zaytsev (Brookhaven National Laboratory (US)), Mr Kevin CASELLA (Brookhaven National Laboratory (US))
• 30
Operating dedicated data centers - Is it cost-effective?
The advent of cloud computing centers such as Amazon's EC2 and Google's Computing Engine has elicited comparisons with dedicated computing clusters. Discussions on appropriate usage of cloud resources (both academic and commercial) and costs have ensued. This presentation discusses a detailed analysis of the costs of operating and maintaining the RACF (RHIC and ATLAS Computing Facility) compute cluster at Brookhaven National Lab and compares them with the cost of cloud computing resources under various usage scenarios. An extrapolation of likely future cost effectiveness of dedicated computing resources is also presented.
Speaker: Dr Tony Wong (Brookhaven National Laboratory)
• Software Engineering, Parallelism & Multi-Core: I Effectenbeurszaal

### Effectenbeurszaal

#### Amsterdam, Beurs van Berlage

• 31
Monte Carlo Simulations of the IceCube Detector with GPUs
The IceCube Neutrino Observatory is a cubic kilometer-scale neutrino detector built into the ice sheet at the geographic South Pole. Light propagation in glacial ice is an important component of IceCube detector simulation that requires a large number of embarrassingly parallel calculations. The IceCube collaboration recently began using GPUs in order to simulate direct propagation of Cherenkov photons in the antarctic ice as part of our detector simulation. GPU computing is now being utilized in large scale Monte Carlo productions involving computing centers distributed across the world. We discuss practical issues of our implementation involving mixed CPU and GPU resources in the simulation chain and our efforts to optimize the utilization of these resources in a grid environment.
Speaker: Claudio Kopper
• 32
High Energy Electromagnetic Particle Transportation on the GPU
We will present massively parallel high energy electromagnetic particle transportation through a finely segmented detector in the Graphic Processor Unit (GPU). Simulating events of energetic particle decay in a general-purpose high energy physics (HEP) detector requires intensive computing resources, due to the complexity of the geometry as well as physics processes applied to particles copiously produced by primary collisions and secondary interactions. The recent advent of hardware architectures of many-core or accelerated processors provides the variety of concurrent programming models applicable not only for the high performance parallel computing, but also for the conventional computing intensive application such as the HEP detector simulation. The component of the transportation prototype consists of a transportation process under a non-uniform magnetic field, a geometry navigation with a set of solid shapes and materials, electromagnetic physics processes for electrons and photons, and an interface to a framework that dispatches bundles of tracks in a highly vectorized manner optimizing for spatial locality and throughput. Core algorithms and methods are excerpted from the Geant4 toolkit, and are modified and optimized for the GPU application. Programs written in C/C++ are designed to be compatible with CUDA and openCL and generic enough for future variations of programming models and hardware architectures. Used with multiple streams, asynchronous kernel executions are overlapped with concurrent data transfers of streams of tracks to balance arithmetic intensity and memory bandwidth. Issues with floating point accuracy, random number generation, data structure, kernel divergences and register spills are also considered. Performance evaluation for the relative speedup compared to the corresponding sequential execution on CPU will be presented as well.
Speaker: Philippe Canal (Fermi National Accelerator Lab. (US))
• 33
Synergia-CUDA: GPU Accelerated Accelerator Modeling Package (video conference)
Synergia is a parallel, 3-dimensional space-charge particle-in-cell code that is widely used by the accelerator modeling community. We present our work of porting the pure MPI-based code to a hybrid of CPU and GPU computing kernels. The hybrid code uses the CUDA platform, in the same framework as the pure MPI solution. We have implemented a lock-free collaborative charge-deposition algorithm for the GPU, as well as other optimizations, including local communication avoidance for GPUs, customized FFT, and fine-tuned memory access patterns. On a small GPU cluster (up to 4 Tesla C1070 GPUs), our benchmarks exhibit both superior peak performance and better scaling, when compared to a CPU cluster with 16 nodes and 128 cores. We have further compared the code performance on different GPU architectures, including C1070 Tesla, M2070 Fermi, and K20 Kepler. We show 10 to 20% performance increases with optimizations addressing each specific hardware architectures.
Speaker: Qiming Lu (Fermi National Accelerator Laboratory)
• 34
The Telescope Array Fluorescence Detector Simulation on GPUs
The Telescope Array Cosmic Rays Detector located in the Western Utah Desert is used for the observation of ultra-high energy cosmic rays. The simulation of a fluorescence detector response to cosmic rays initiated air showers presents many opportunities for parallelization. In this presentation we report on the Monte Carlo program used for the simulation of the Telescope Array fluorescence detector located at the Middle Drum site. The program makes extensive use of GPU acceleration (CUDA) to achieve a 50x speed-up compared to running on a single CPU core. The main design criteria for the simulation code is that it can be run with/without acceleration while eliminating code duplication as much as possible. This is seen as critical for long term maintainability of the source code. A CMake based build system allows the user to easily select to compile/run the code with or without CUDA, and to include external package, e.g. ROOT, dependent code in the build. All of the physics simulation from shower development, light production and propagation with atmospheric attenuation, as well as, the realistic detector optics and electronics simulations are done on the GPU. A detailed description of the code implementation is given, and results on the accuracy and performance of the simulation are presented as well. Detector event reconstruction is performed using an inverse Monte Carlo method. We are in the process of porting that code to make use of the GPU based simulation. Results will be presented on this effort as well.
Speaker: Dr Tareq AbuZayyad (University of Utah)
• Poster presentations: I Grote zaal

### Grote zaal

#### Amsterdam, Beurs van Berlage

• 35
A browser based multi-user working environment for physicists
Many programs in experimental particle physics do not yet have a graphical interface, or demand strong platform and software requirements. With the most recent development of the VISPA project, we provide graphical interfaces to existing software programs and access to multiple computing clusters through standard web browsers. The scalable client-server system allows analyses to be performed in sizable teams, and disburdens the individual physicist from installing and maintaining a software environment. The VISPA graphical interfaces are implemented in HTML, JavaScript and extensions to the Python webserver. The webserver uses SSH and RPC to access user data, code and processes on remote sites. As example applications we present graphical interfaces for steering the reconstruction framework OFFLINE of the Pierre-Auger experiment, and the analysis development toolkit PXL. The browser based VISPA system was field-tested in bi-weekly homework of a third year physics course by more than 100 students. We discuss the system deployment and the evaluation by the students.
Speaker: Gero Müller (RWTH Aachen University)
• 36
A Comprehensive Approach to Tier-2 Administration
Liverpool is consistently amongst the top Tier-2 sites in Europe in terms of efficiency and cluster utilisation. This presentation will cover the work done at Liverpool over the last six years to maximise and maintain efficiency and productivity at their Tier 2 site, with an overview of the tools used (including established, emerging, and locally developed solutions) for monitoring, testing, installation, configuration, ticketing, logging, and administration, along with the techniques for management, operations, and optimisation that tie them together into a comprehensive, scalable, and sustainable approach to Tier 2 administration.
Speaker: John Bland (University of Liverpool)
• 37
A data parallel digitizer for a time-based simulation of CMOS Monolithic Active Pixel Sensors with FairRoot
CMOS Monolithic Active Pixel Sensors (MAPS) have demonstrated excellent performances as tracking detectors for charged particles. Their outstanding spatial resolution (few µm), ultra-light material budget (50 µm) and advanced radiation tolerance (> 1Mrad, >1e13 neq/cm²). They were therefore chosen for the vertex detectors of STAR and CBM and are foreseen to equip the upgraded ALICE-ITS. They also constitute a valuable option for tracking devices at future e+e- colliders. MAPS were initially developed as sensors for photographic devices and the data is readout with a rolling shutter. The readout time of an individual frame lasts typically 10-100µs. In high rate experiments like CBM, the pixels matrix may sum particle signals generated by several particle collisions during this integration time. Powerful tracking codes are needed to disentangle those collisions based on the data obtained from the faster tracking detectors located more downstream the collision point. Developing this code requires a realistic and fast digitizer software, which represents the properties of the sensors within GEANT-based simulation frameworks like FairRoot. We introduce the challenges related to representing collision pile-up in an event based simulation environment and discuss our simulation strategy. Moreover, we introduce our concept for data parallelism, which aims to allow for a parallel code execution in a near future.
Speaker: Philipp Sitzmann (Goethe University Frankfurt)
• 38
A flexible monitoring infrastructure for the simulation requests
Running and monitoring simulations usually involves several different aspects of the entire workflow: the configuration of the job, the site issues, the software deployment at the site, the file catalogue, the transfers of the simulated data. In addition, the final product of the simulation is often the result of several sequential steps. This project tries a different approach to monitoring the simulation requests. All the necessary data are collected from the central services which lead the submission of the requests and the data management, and stored by a backend into a NoSQL-based data cache; those data can be queried through a Web Service interface, which returns JSON responses, and allows users, sites, physics groups to easily create their own web frontend, aggregating only the needed information. As an example, it will be shown how it is possible to monitor the CMS services (ReqMgr, DAS/DBS, PhEDEx) using a central backend and multiple customized cross-language frontends.
Speakers: Vincenzo Spinoso (Universita e INFN (IT)), Vincenzo Spinoso (Universita e INFN (IT))
• 39
The LHCb Software Infrastructure is built around a flexible, extensible, single-process, single-threaded framework named Gaudi. One way to optimise the overall usage of a multi-core server, which is used for example in the Online world, is running multiple instances of Gaudi-based applications concurrently. For LHCb, this solution has been shown to work well up to 32 cores and is expected to scale up a bit further. The appearance of many-core architectures such as GPGPUs and the Intel Xeon/Phi poses a new challenge for LHCb. Since the individual data sets are so small (about 60 kB raw event size), many events must be processed in parallel for optimum efficiency. This is, however, not possible with the current framework, which allows only a single event at a time. Exploiting the fact that we always have many instances of the same application running, we have developed an offloading mechanism, based on a client-server design. The server runs outside the Gaudi framework and thus imposes no additional dependencies on existing applications. It asynchronously receives event data from multiple client applications, coalesces them, and returns the computed results back to callers. While this incurs additional inter-process communication overhead, it allows co-processors to be used within the existing large framework with minimal changes. We present our solution and describe the achieved performance, both at the single-instance and the server levels.
Speaker: Daniel Hugo Campora Perez (CERN)
• 40
A J2EE based server for Muon Spectrometer Alignment monitoring in the ATLAS detector
The ATLAS muon alignment system is composed of about 6000 optical sensors for the Barrel muon spectrometer and the same number for the 2 Endcaps wheels. The system is acquiring data from every sensor continuously , with a whole read-out cycle of about 10 minutes. The read-out chain stores data inside an Oracle DB. These data are used as input from the alignment algorithms (C++ based) in order to compute alignment corrections that are then used by ATLAS muon reconstruction software. An application deployed inside a J2EE server takes care of interactions between the DB and the alignment algorithms, delivering also functions for monitoring tools, handling scheduled tasks to launch alignment reconstruction software and checking and validating results before their migration to the official ATLAS Condition DB (COOL). The same application allows access to COOL database information and to another database containing Conditions and Configurations Metadata for ATLAS (COMA), giving thus the possibility to follow the full chain of the data flow of the Muon Alignment system. We will describe the architecture of the J2EE application and the monitoring tools that have been developed.
Speaker: Andrea Formica (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
• 41
A method to improve the electron momentum reconstruction for the PANDA experiment
The PANDA (AntiProton ANnihilation at DArmstadt) experiment is one of the key projects at the future Facility for Antiproton and Ion Research (FAIR), which is currently under construction at Darmstadt. This experiment will perform precise studies of antiproton-proton and antiproton-nucleus annihilation reactions. The aim of the rich experimental program is to improve our knowledge of the strong interaction and of the structure of hadrons. In particular, the study of electromagnetic processes, like (pbar p->e+e-, pbarp->e+e-π0, etc..), gives access to the proton structure (electric and magnetic form factors, Transition Distribution Amplitudes, etc..). In such channels, the electron and positron signal needs to be separated from the hadronic background, which is six orders of magnitude larger than the signal. Excellent electron particle identification and momentum reconstruction are therefore crucial for such studies. The PandaRoot software, based on ROOT and Virtual MonteCarlo, is used as the simulation and analysis framework for the future PANDA experiment. A Kalman Filter provides the particle momenta deduced from the central tracker, with GEANE as track follower. This method is not well suited for electrons, for which the highly non-gaussian Bremsstrahlung process yields a tail in the momentum resolution distribution. A new method was developed to solve this problem at least partially, in an event by event procedure, taking advantage of the possible detection of the Bremsstrahlung photon as a separate cluster in the Electromagnetic Calorimeter. We will show that this is possible for tracks with transverse momentum up to 1 GeV/c. The shape of the electron shower is also used to identify the Bremsstrahlung photon at higher electron momenta region. The improvement of electron momentum reconstruction will be shown, as well as the gain on the signal to background ratio. In the presentation, the details about the technical implementation of the method in PANDARoot will also be given.
Speaker: Mr MA Binsong (IPN Orsay France)
• 42
A New Nightly Build System for LHCb
The nightly build system used so far by LHCb has been implemented as an extension on the system developed by CERN PH/SFT group (as presented at CHEP2010). Although this version has been working for many years, it has several limitations in terms of extensibility, management and ease of use, so that it was decided to develop a new version based on a continuous integration system. In this paper we describe a new implementation of the LHCb Nightly Build System based on the open source continuous integration system Jenkins and report on the experience on the configuration of a complex build workflow in Jenkins.
Speaker: Marco Clemencic (CERN)
• 43
A novel dynamic event data model using the Drillbit column store
The focus in many software architectures of the LHC experiments is to deliver a well-designed Event Data Model (EDM). Changes and additions to the stored data are often very expensive, requiring large amounts of CPU time, disk storage and man-power. At the ATLAS experiment, such a reprocessing has only been undertaken once for data taken in 2012. However, analysts have to develop and apply corrections and do computations after the final official data processing or re-processing has taken place. The current practice at ATLAS is to distribute software tools to apply these corrections at analysis runtime, requiring any necessary input data to be present and manually supplied to the tool, and requiring manual application of the tool output to the event data by each analyst. This approach has proven to be very expensive in terms of man-power, especially since verifying that the tools have been applied correctly (or at least consistently) is very time consuming. Drillbit enables dynamic addition of event data, stored in and read from external files. This would make it possible to forego a fixed EDM class structure and instead collate validated variables and objects in a dynamically defined event. Corrections could be computed once or twice by experts, versioned, and then made available to others directly. The technical basis for this architecture is currently being prototyped, and initial versions of the underlying Drillbit column store are available.
Speaker: Mr Peter Waller (University of Liverpool (GB))
• 44
A quasi-online distributed data processing on WAN: the ATLAS muon calibration system.
In the Atlas experiment, the calibration of the precision tracking chambers of the muon detector is very demanding, since the rate of muon tracks required to get a complete calibration in homogeneous conditions and to feed prompt reconstruction with fresh constants is very high (several hundreds Hz for 8-10 hours runs). The calculation of calibration constants is highly CPU consuming. In order to fulfill the requirement of completing the cycle and having the final constants available within 24 hours, distributed resources at Tier-2 centers have been allocated. The best place to get muon tracks suitable for detector calibration is the second level trigger, where the pre-selection of data sitting in a limited region by the first level trigger via the Region of Interest mechanism allows selecting all the hits from a single track in a limited region of the detector. Online data extraction allows calibration data collection without performing special runs. Small event pseudo-fragments (about 0.5 kB) built at the muon level-1 rate (2-3 kHz at the beginning of 2012 run, to become 10-12 kHz at maximum LHC luminosity) are then collected in parallel by a dedicated system, without affecting the main data taking, and sent to the Tier-0 computing center at CERN. The computing resources needed to calculate the calibration constants are distributed through three calibration centers (Rome, Munich, Ann Arbor) for the tracking device and one (Napoli) for the trigger chambers. From Tier-0, files are directly sent to the calibration centers through the ATLAS Data Distribution Manager. At the calibration centers, data is split per trigger tower and distributed to computing nodes for concurrent processing (~250 cores are currently used at each center). A two-stage processing is performed, the first stage reconstructing tracks and creating ntuples, the second one calculating constants. The calibration parameters are then stored in the local calibration database and replicated to the main condition database at CERN, which makes them available for data analysis within 24 hours from data extraction. The architecture and performance of this system during the 2011-2012 data taking will be presented. This system will evolve in the next future to comply with the new stringent requirements of the LHC and ATLAS upgrade. If for the WAN distribution part the availability of bandwidth is already much larger than needed for this task and the CPU power can be increased according to our need, the online part will follow the evolution of the ATLAS TDAQ architecture. In particular, the current model foresees the merging of the level-2 and event filtering processes on the same nodes, allowing the simplification of the system and a more flexible and dynamic resource distribution. Two possible architectures are possible to comply with this model; possible implementation will be discussed.
Speaker: Alessandro De Salvo (Universita e INFN, Roma I (IT))
• 45
A Scalable Infrastructure for CMS Data Analysis Based on Openstack Cloud and Gluster File System
The challenge of providing a resilient and scalable computational and data management solution for massive scale research environments, such as the CERN HEP analyses, requires continuous exploration of new technologies and techniques. In this article we present a hybrid solution of an open source cloud with a network file system for CMS data analysis. Our aim has been to design a scalable and resilient infrastructure for CERN HEP data analysis. The infrastructure is based on Openstack components for structuring a private cloud together with the Gluster filesystem. The Openstack cloud platform is one of the fastest growing solutions in the cloud world. The Openstack components provide solutions for computational resource (NOVA), instance (GLANCE), network (QUANTUM) and security (Keystone) management all underneath an API layer that supports global applications, web clients and large ecosystems. NOVA and GLANCE manage Virtual Machines (VMs) and image repository respectively. QUANTUM provides network as a service (NAAS) and advanced network management capabilities. The virtual network layer provided by QUANTUM supports seamless migration of the VMs while preserving the network configuration. One important component that is currently not part of the Openstack suite is a network file system. To overcome this limitation we have used GlusterFS, a network-based file system for high availability. GlusterFS uses FUSE and may be scale to petabytes. In our experiments, 1TB is used for instance management and 2TB for the data related to CMS jobs within GlusterFS. We integrate the above state-of-the-art cloud technologies with the traditional Grid middleware infrastructure. This approach implies no changes for the end-user, while the production infrastructure is enriched by the high-end resilient and scalable components. To achieve this, we have run Advance Resource Connector (ARC) as a meta-scheduler. Both Computing Elements (CE) and Worker Nodes (WN) are running on VM instances inside the Openstack cloud. Currently we consider our approach as semi-static, as the instance management is manual yet provides scalability and performance. In near future we are aiming for a comprehensive elastic solution by including the EMI authorization service (Argus) and the Execution Environment Service (Argus-EES). In order to evaluate the strength of the infrastructure, four test cases have been selected for experimentation and analysis. (i) The first test case is based on instance performance, the boot time of customized images using different hypervisors and the performance of multiple instances with different configurations. (ii) The second test case focuses on I/O-related performance analysis based on GlusterFS. This test also presents the performance of instances running on GlusterFS compared with the local file system. (iii) The third test case examines system stability with live migration of VM instances based on QUANTUM. (iv) In the fourth test we will present long-term system performance both at the level of VMs running CMS jobs and physical hosts running VMs. Our test results show that the adopted approach provides a scalable and resilient solution for managing resources without compromising on performance and high availability.
Speaker: Dr Salman Toor (Helsinki Institute of Physics (FI))
• 46
A tool for Conditions Tag Management in ATLAS
ATLAS Conditions data include about 2 TB in a relational database and 400 GB of files referenced from the database. Conditions data is entered and retrieved using COOL, the API for accessing data in the LCG Conditions Database infrastructure. It is managed using an ATLAS-customized python based tool set. Conditions data are required for every reconstruction and simulation job, so access to them is crucial for all aspects of ATLAS data taking and analysis, as well as by preceding tasks to derive optimal corrections to reconstruction. Ensuring that the optimal alignment and calibration information is used in the reconstruction is a complex task: variations can occur even within a run and independently in time from one subsystem to the other. Optimized sets of conditions for processing are accomplished using strict version control on those conditions: a process which assigns COOL Tags to sets of conditions, and then unifies those conditions over data-taking intervals into a COOL Global Tag. This Global Tag identifies the set of conditions used to process data so that the underlying conditions can be uniquely identified with 100% reproducibility should the processing be executed again. Understanding shifts in the underlying conditions from one tag to another and ensuring interval completeness for all detectors for a set of runs to be processed is a complex task, requiring tools beyond the above mentioned python utilities. Therefore, a Java/php based utility called the Conditions Tag Browser (CTB) has been developed. CTB gives detector and conditions experts the possibility to navigate through the different databases and COOL folders; explore the content of given tags and the differences between them, as well as their extent in time; visualize the content of channels associated with leaf tags. This report describes the structure and implementation of the CTB, demonstrates its use during LHC Run 1, and describes plans for expanding its functionality in preparation for LHC Run 2.
Speaker: Andrea Formica (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
• 47
A Validation Framework to faciliatate the Long Term Preservation of High Energy Physics Data (The DESY-DPHEP Group)
In a future-proof data preservation scenario, the software and environment employed to produce and analyse high energy physics data needs to be preserved, rather than just the data themselves. A software preservation system will be presented which allows analysis software to be migrated to the latest software versions and technologies for as long as possible, substantially extending the lifetime of the software, and hence also the data. Contrary to freezing the environment and relying on assumptions about future virtualisation standards, we propose a rolling model of preservation of the software. Technically, this is realised using a virtual environment capable of hosting an arbitrary number of virtual machine images, built with different configurations of operating systems and the relevant software, including any necessary external dependencies. A significant fraction of the work involved requires a deep level of validation of the experimental software and environment, and in particular the progress made by the participating experiments will be presented. Such a system is by design expandable and able to host and validate the requirements of multiple experiments, and can be thought of as a tool to aid migration that will detect problems and incoherence, helping to identify and solve them by the join efforts of experiments and computer experts.
Speaker: Dmitry Ozerov (D)
• 48
A Voyage to Arcturus
With the current trend towards "On Demand Computing" in big data environments it becomes crucial that the deployment of services and resources becomes increasingly automated. With opensource projects such as Canonicals MaaS and Redhats Spacewalk; automated deployment is available for large scale data centre environments but these solutions can be too complex and heavyweight for smaller, resource constrained WLCG Tier-2 sites. Along with a greater desire for bespoke monitoring and the collection of more Grid related metrics, a more lightweight and modular approach is desired. In this paper work carried out on the test cluster environment at the Scotgrid site of the University of Glasgow is presented. Progress towards a lightweight automated framework for building WLCG grid sites is presented, based on "off the shelf" software components such as Cobbler and Puppet, the building blocks of the larger open source projects mentioned before. Additionally the test cluster is used to investigate these components in a mixed IPv4/IPv6 environment, as well as using emerging OpenFlow technologies for software service provisioning. As part of the research into an automation framework the use of IPMI and SNMPv2 for physical device management will be included, as well as the possibility of SNMPv2 as a monitoring/data sampling layer such that more comprehensive decision making can take place and potentially be automated. This could lead to reduced down times and better performance as services are recognised to be in a non-functional state by autonomous systems. Finally, through the use of automated service provisioning and automated device management the building blocks of a fully automated expert system will be touched upon.
Speakers: Gareth Roy (U), Mark Mitchell (University of Glasgow)
• 49
Accessing opportunistic resources with Bosco
Bosco is a software project developed by the Open Science Grid to help scientists better utilize their on-campus computing resources. Instead of submitting jobs through a dedicated gatekeeper, as most remote submission mechanisms use, it uses the built-in SSH protocol to gain access to the cluster. By using a common access method, SSH, we are able to simplify the interaction with the cluster, making the submission process more user friendly. Additionally, it does not add any extra software to be installed on the cluster making Bosco an attractive option for the cluster administrator. In this paper, we will describe Bosco, the personal supercomputing assistant, and how Bosco is used by researchers across the U.S. to manage their computing workflows. In addition, we will also talk about how researchers are using it, including an unique use of Bosco to submit CMS reconstruction jobs to an opportunistic XSEDE resource.
Speaker: Derek John Weitzel (University of Nebraska (US))
• 50
AGIS: The ATLAS Grid Information System
In this paper we describe the ATLAS Grid Information System (AGIS), the system designed to integrate configuration and status information about resources, services and topology of the computing infrastructure used by ATLAS Distributed Computing (ADC) applications and services. The Information system centrally defines and exposes the topology of the ATLAS computing infrastructure including various static, dynamic and configuration parameters collected both from independent sources like gLite BDII (Berkley Database Information Index), Grid Operations Centre Database (GOCDB), the Open Science Grid Information services (MyOSG), and from ATLAS specific ones like the ATLAS Distributed Data Management (DQ2) system and the ATLAS Production and Distributed Analysis (PanDA) workload management system. Being an intermediate middleware system between clients and external information sources, AGIS automatically collects and keeps data up to date, caching information required by ATLAS, removing the source as a direct dependency for end-users, but without duplicating the source information itself: AGIS represents data objects in the way more convenient for ATLAS services, introduces additional object relations required by ATLAS applications, and exposes the data via the REST style API and WEB front end services. For some types of information AGIS itself has become the primary repository. We describe the evolution and new functionalities of WEB and API services implemented to integrate AGIS with ADC applications and provide user-friendly native WEB interface to manage data. We will explain how the AGIS flexibility allows the definition of new services still not integrated in production in WLCG but already used by ATLAS for specific use cases. In particular, it concerns the definitions of the FAX storage federation, which is based on XrootD storage services, and redirectors organized with a precise ATLAS specific topology, stored in AGIS. Special attention is given also to the implementation of user roles and privileges allowed to separate access within user groups using AGIS in production. Introducing various user access groups makes the Information System a sustainable way to keep a global and coherent view of the whole computing infrastructure used by ATLAS.
Speaker: Alexey Anisenkov (Budker Institute of Nuclear Physics (RU))
• 51
An Agile Service Deployment Framework and its Application
In this paper we shall introduce the service deployment framework based on Quattor and Microsoft HyperV at the RAL Tier 1. As an example, we will explain how the framework has been applied to CASTOR in our test infrastructure and outline our plans to roll it out into full production. CASTOR is a relatively complicated open source hierarchical storage management system in production use at RAL for both WLCG and large scale scientific facilities data. Finally, we will examine different approaches of virtualizing CASTOR using HyperV and we will present an optimal approach along with the advantages and disadvantages of each approach.
Speakers: Ian Collier (UK Tier1 Centre), Mr Matthew James Viljoen (STFC - Science & Technology Facilities Council (GB))
• 52
An efficient data protocol for encoding preprocessed clusters of CMOS Monolithic Active Pixel Sensors
CBM aims to measure open charm particles from 15-40 AGeV/c heavy ion collisions by means of secondary vertex reconstruction. The measurement concept includes the use of a free-running DAQ, real time tracking, primary and secondary vertex reconstruction and a tagging of open charm candidates based on secondary vertex information. The related detector challenge will be adressed with an ultra-light and highly granular Micro Vertex Detector (MVD) based on CMOS Monolithic Active Pixel Sensors (MAPS). Performing the real time vertex reconstruction at collision rates of ~1e5 coll./s will introduce a substantial CPU-load to the computing system (FLES) of CBM. To reduce this load, we consider to perform pre-processing steps like cluster finding already in DAQ-instances upstream the FLES. A successful pre-processing concept should be FPGA-compatible. Moreover, it should provide a lossless encoding of the original information as much as the newly computed information on the cluster position and shape without extending the data volume. To fulfill those requirements, we developped a cluster encoding concept which may encode the above mentioned information in a single 32-bit word. This concept is introduced and its validity is discussed based on data from the recent beam test of the MVD-prototype at CERN-SPS.
Speaker: Qiyan Li (Goethe University Frankfurt)
• 53
An HTTP Ecosystem for HEP Data Management
In this contribution we present a vision for the use of the HTTP protocol for data management in the context of HEP, and we present demonstrations of the use of HTTP-based protocols for storage access & management, cataloguing, federation and transfer. The support of HTTP/WebDAV, provided by frameworks for scientific data access like DPM, dCache, STORM, FTS3 and foreseen for XROOTD, can be seen as a coherent ensemble – an ecosystem – that is based on a single, standard protocol, where the HEP-related features are covered, and the door is open to standard solutions and tools provided by third parties, in the context of the Web and Cloud technologies. The application domain for such an ecosystem of services goes from large scale Cloud and Grid-like computing to the data access from laptops, profiting from tools that are shared with the Web community, like browsers, clients libraries and others. Particular focus was put into emphasizing the flexibility of the frameworks, which can interface with a very broad range of components, data stores, catalogues and metadata stores, including the possibility of building high performance dynamic federations of endpoints that build on the fly the feeling of a unique, seamless and efficient system. The overall goal is to leverage standards and standard practices, and use them to provide the higher level functionalities that are needed to fulfil the complex problem of Data Access in HEP. In this context we explain how the subset of SRM functionality relevant to disk system could be offered over HTTP. Other points of interest are about harmonizing the possibilities given by the HTTP/WebDAV protocols with existing frameworks like ROOT and already existing Storage Federations based on the XROOTD framework. We also provide quantitative evaluations of the performance that is achievable using HTTP for remote transfer and remote I/O in the context of HEP data, with reference to the recent implementation of HTTP support in FTS3.
Speaker: Fabrizio Furano (CERN)
• 54
An Infrastructure in Support of Software Development
The success of a scientific endeavor depends, often significantly, on the ability to collect and later process large amounts of data in an efficient and effective way. Despite the enormous technological progress in areas such as electronics, networking and storage, the cost of the computing factor remains high. Moreover the limits reached by some historical directions of hardware development, such as the saturation of the CPU clock speed with the consequent strong shift towards hardware parallelization, has made the role of software more and more important. In order to support and facilitate the daily activities of software developers within INFN, an integrated infrastructure has been designed, comprising several tools, each providing a function: project tracking, code repository, continuous integration, quality control, knowledge base, dynamic provisioning of virtual machines, services, or clusters thereof. When applicable, access to the services is based on the INFN-wide Authentication and Authorization Infrastructure. The system is being installed and progressively made available to INFN users belonging to tens of INFN sites and laboratories and will represent a solid foundation to the software development efforts of the many experiments and projects that see the involvement of INFN. The infrastructure will be beneficial especially for small- and medium-size collaborations, which often cannot afford the resources, in particular in terms of know-how, needed to set up such services. This contribution describes the design of the infrastructure, the components that implement it, how they integrate with each other, in particular from the Authentication point of view, and how it is expected to evolve in the future.
Speaker: Francesco Giacomini (INFN CNAF)
• 55
An integrated framework for the data quality assessment and database management for the ATLAS Tile Calorimeter
The Tile calorimeter is one of the sub-detectors of ATLAS. In order to ensure its proper operation and assess the quality of data, many tasks are to be performed by means of many tools which were developed independently to satisfy different needs. Thus, these systems are commonly implemented without a global perspective of the detector and lack basic software features. Besides, in some cases they overlap in the objectives and resources with another one. It is therefore evident the necessity of an infrastructure to allow the implementation of any functionality without having to duplicate the effort while being possible to integrate with an overall view of the detector status. Tile-in-ONE is intended to meet these needs, by providing a unique system, which integrates all the web systems and tools used by the Tile calorimeter, with a standard development technology and documentation. It also intends to abstract the user from knowing where and how to get the wanted data by providing a user friendly interface. It is based in a server containing a core, which represents the basic framework that loads the configuration, manages user settings and loads plugins at run-time; a set of services, which provide common features to be used by the plug-ins, such as connectors to different databases and resources; and the plug-ins themselves which provide features at the top level layer for the users. Moreover, a web environment is being designed to allow collaborators develop their own plug-ins, test them and add them to the system. To make it possible, an API is used allowing any kind of application to be interpreted and displayed in a standard way.
Speaker: Carlos Solans Sanchez (CERN)
• 56
An SQL-based approach to Physics Analysis
As part of the CERN Openlab collaboration, an investigation has been made into the use of an SQL-based approach for physics analysis with various up-to-date software and hardware options. Currently physics analysis is done using data stored in customised root-ntuples that contain only the variables needed for a specific analysis. Production of these ntuples is mainly done by accessing the centrally produced analysis data through the LHC computing grid and can take several days to complete. We'll present an alternative approach to physics data analysis where analysis data is stored in a database, removing the need for customized ntuple production, and allowing calculations that are part of the analysis to be done on the database side. An example implementation of such a database will be shown, demonstrating how physics analysis in a database can be done via ROOT. The use of an Oracle database in this setup is compared to use of a Hadoop-based data structure. The advantages and drawbacks of the different database setups are presented, with a detailed analysis of the CPU and I/O usage, in particular for the case when many users need to access the database at the same time.
Speaker: Maaike Limper (CERN)
• 57
An Xrootd Italian Federation for CMS
The italian community in CMS has built a geographically distributed network in which all the data stored in the italian region are available to all the users for their everyday work. This activity involves at different level all the CMS centers: the Tier1 at CNAF, all the four Tier2s (Bari, Rome, Legnaro and Pisa), and few Tier3s (Trieste, Perugia, etc). The federation uses the new network connections as provided by our NREN, GARR, which provides a minimum of 10 Gbit/s to all the sites via the GARR-X project. The federation is currently based on Xrootd technology, and on a redirector aimed to seamlessly connect all the sites, giving the logical view of a single entity. A special configuration has been put in place for the Tier1, CNAF, where ad-hoc Xrootd changes have been implemented in order to protect the tape system from excessive stress, by not allowing WAN connections to access tape only files, on a file-by-file basis. We will describe in details the test carried on the authentication and authorization capabilities in the Xrootd code, in order to achieve a better fine grained authorization criteria. For example with this authentication mechanism it is possible to implement a protection on the base of VOMS attributes and the group and roles. In order to improve the overall performance while reading files, both in terms of bandwidth and latency, it is implemented a hierarchical solution for the xrootd redirectors. The solution implemented provides a dedicated redirector where all the INFN sites are registered, without considering their status (T1, T2, or T3 sites). This redirector is the first option for the end user where they can read files both in the CMSSW framework and using bare ROOT Macros. This redirector is used also to publish information of sites that do not provide official Service Level Agreement to CMS, so that could not join the official CMS redirector. An interesting use case were able to cover via the federation are disk-less Tier3s. For sites where local manpower and/or funding does not allow the operation of a storage system, CMS analysis is still allowed by serving all the input files via WAN; the option of a local frontend cache protects the infrastructure from excessive data transfers in this case. The caching solution allows to operate a local storage with minimal human intervention: transfers are automatically done on a single file basis, and the cache if maintained operational by automatic removal of old files.
Speaker: Dr Giacinto Donvito (INFN-Bari)
• 58
Analysis and improvement of data-set level file distribution in Disk Pool Manager
Of the three most widely used implementations of the WLCG Storage Element specification, Disk Pool Manager (DPM) has the simplest implementation of file placement balancing (StoRM doesn't attempt this, leaving it up to the underlying filesystem, which can be very sophisticated in itself). DPM uses a round-robin algorithm (with optional filesystem weighting), for placing files across filesystems and servers. This does a reasonable job of evenly distributing files across the storage array provided to it. However, it does not offer any guarantees of the evenness of distribution of that subset of files associated with a given "dataset" (which often maps onto a "directory" in the DPM namespace (DPNS) ). It is useful to consider a concept of 'balance', where an optimally balanced set of files indicates that the files are distributed evenly across all of the pool nodes. The best case performance of the round robin algorithm is to maintain balance, it has no mechanism to improve balance. In the past year or more, larger DPM sites have noticed load spikes on individual disk servers, and suspected that these were exacerbated by excesses of files from popular datasets on those servers. We present here a software tool which analyses file distribution for all datasets in a DPM SE, providing a measure of the poorness of file location in this context. Further, the tool provides a list of file movement actions which will improve dataset-level file distribution, and can action those file movements itself. We present results of such an analysis and movement on the UKI-SCOTGRID-GLASGOW DPM.
• 59
Analysis of Alternative Storage Technologies for the RAL Tier 1
At the RAL Tier 1 we have successfully been running a CASTOR HSM instance for a number of years. While it performs well for disk-only storage for analysis and processing jobs, it is heavily optimised for tape usage. We have been investigating alternative technologies which could be used for online storage for analysis. We present the results of our preliminary selection and test results for selected storage technologies.
Speaker: Shaun De Witt (STFC - Science & Technology Facilities Council (GB))
• 60
Arby, a general purpose, low-energy spectroscopy simulation tool
Measurements of radioactive sources, in order to reach an optimum level of accuracy, require an accurate determination of the detection efficiency of the experimental setup. In gamma ray spectroscopy, in particular, the high level of sensitivity reached nowadays implies a correct evaluation of the detection capability of source emitted photons. The standard approach, based on an analytical extension of calibrated source measurements usually introduces large uncertainties related to shapes, geometrical setup, compositions and details of the nuclear decay emissions. To overcome the limitations intrinsic in the standard approach a different and more appropriate methodology is needed, specifically the possibility to simulate a virtual experiment featuring the same characteristics as the real measurements (geometry, materials, physical process involved). The GEANT4 toolkit offers all the ingredients needed to build such a simulator: the standard approach is to write specialized code to simulate a specific problem using the toolkit components and assembling them in a compiled program that contains the full specification of the experimental setup. Our approach, at INFN Milano-Bicocca has been, instead, to build a general purpose program capable of tackling a wide range of use-cases by reading the complete specification of the experiment to simulate from external files. This decoupling between simulation algorithm and experimental setup description allows to maintain and validate a single program, making it easy to add features and components by leveraging on an already existing body of functionality. This code, called Arby, was designed based on our experience in very low-background experiments: it's generality stems from the complete decoupling between its simulation capabilities and the actual specification of the experiment's setup. Different materials in different geometries (source and detector) can be specified externally, as well as a wide variety of different physical process to describe the interaction of radiation with matter. Even pile-up phenomena are correctly taken into account by the code, allowing for a fine-tuning and an arbitrary level of accuracy in determining the efficiency of the experimental setup. In this talk we will describe the architecture of the system and show some applications to real radioactive data analysis.
Speaker: Dr Massimiliano Nastasi (INFN Milano-Bicocca)
• 61
ARC SDK: A toolbox for distributed computing and data applications
Grid middleware suites provide tools to perform the basic tasks of job submission and retrieval and data access, however these tools tend to be low-level, operating on individual jobs or files and lacking in higher-level concepts. User communities therefore generally develop their own application-layer software catering to their specific communities' needs on top of the Grid middleware. It is thus important for the Grid middleware to provide a friendly, well documented and simple to use interface for the applications build on. The Advanced Resource Connector (ARC), developed by NorduGrid, provides a Software Development Kit (SDK) which enables applications to use the middleware for job and data management. This paper presents the architecture and functionality of the ARC SDK along with an example graphical application developed with the SDK. The SDK consists of a set of libraries accessible through Application Programming Interfaces (API) in several languages. It contains extensive documentation and example code and is available on multiple platforms. The libraries provide generic interfaces and rely on plugins to support a given technology or protocol and this modular design makes it easy to add a new plugin if the application requires supporting additional technologies. The ARC Graphical Clients package is a graphical user interface built on top of the ARC SDK and the Qt toolkit and it is presented here as a fully functional example of an application. It provides a graphical interface to enable job submission and management at the click of a button, and allows data on any Grid storage system to be manipulated using a visual file system hierarchy, as if it were a regular file system.
Speaker: David Cameron (University of Oslo (NO))
• 62
Architectural improvements and 28nm FPGA implementation of the APEnet+ 3D Torus network for hybrid HPC systems
Modern Graphics Processing Units (GPUs) are now considered accelerators for general purpose computation. A tight interaction between the GPU and the interconnection network is the strategy to express the full potential on capability computing of a multi-GPU system on large HPC clusters; that is why an efficient and scalable interconnect is a key technology to finally deliver GPUs for scientific HPC. In this paper we show the latest architectural and performance improvement of the APEnet+ network fabric, a FPGA-based PCIe board with 6 fully bidirectional off-board links with 34 Gbps of raw bandwidth per direction, and X8 Gen2 bandwidth towards the host PC. The board implements a Remote Direct Memory Access (RDMA) protocol that leverages upon peer-to-peer (P2P) capabilities of Fermi and Kepler-class NVIDIA GPUs to obtain real zero-copy, low-latency GPU-to-GPU transfers. Finally we report on the development activities for 2013 focusing on the adoption of the latest generation 28 nm FPGAs and the preliminary results achieved with synthetic benchmarks exploiting the implementation of state-of-the-art signalling capabilities of PCI-express Gen3 host interface.
Speaker: Dr Roberto Ammendola (INFN Roma Tor Vergata)
• 63
Archival Services and Technologies for Scientific Data
After analysis and publication, there is no need to keep experimental data online on spinning disks. For reliability and costs inactive data is moved to tape and put into a data archive. The data archive must provide reliable access for at least ten years following a recommendation of the German Science Foundation (DFG), but many scientific communities wish to keep data available much longer. Data archival is on the one hand purely a bit preservation activity in order to ensure the bits read are the same as those written years before. On the other hand enough information must be archived to be able to use and interpretate the content of the data. The latter is depending on many also community specific factors and remains an areas of much debate among archival specialists. The paper describes the current practice of archival and bit preservation in use for different science communities at KIT for which a combination of organisational services and technical tools are required. The special monitoring to detect tape related errors, the software infrastructure in use as well as the service certification are discussed. Plans and developments at KIT also in the context of the Large Scale Data Management and Analysis (LSDMA) project are presented. The technical advantages of the T10 SCSI Stream Commands (SSC-4) and the Linear Tape File System (LTFS) will have a profound impact on future long term archival of large data sets.
Speaker: Dr Jörg Meyer (KIT - Karlsruher Institute of Technology)
• 64
Speaker: Mario Lassnig (CERN)
• 65
ATLAS Distributed Computing Monitoring tools during the LHC Run I
The ATLAS Distributed Computing (ADC) Monitoring targets three groups of customers: ADC Operations, ATLAS Management, and ATLAS sites and ATLAS funding agencies. The main need of ADC Operations is to identify malfunctions early and then escalate issues to an activity or a service expert. The ATLAS Management use visualisation of long-term trends and accounting information about the ATLAS Distributed Computing resources. The ATLAS sites and the ATLAS funding agencies utilize both real-time monitoring and long-term measurement of the performance of the provided computing resources. During the LHC Run I a significant development effort has been invested in standardization of the monitoring and accounting applications in order to provide an extensive monitoring and accounting suite. ADC Monitoring applications separate the data layer and the visualisation layer. The data layer exposes data in a predefined format. The visualisation layer is designed bearing in mind visual identity of the provided graphical elements, and reusability of the visualisation elements across the different tools. A rich family of filtering and searching options enhancing available user interfaces comes naturally with the data and visualisation layer separation. With a variety of reliable monitoring data accessible through standardized interfaces, the possibility of automating actions under well defined conditions, correlating multiple data sources, has become feasible. In this contribution we also discuss the automated exclusion of degraded resources and their automated recovery in different activities.
Speaker: Jaroslava Schovancova (Brookhaven National Laboratory (US))
• 66
ATLAS Distributed Computing Operation Shift Teams experience during the discovery year and beginning of the Long Shutdown 1
ATLAS Distributed Computing Operation Shifts were evolved to meet new requirements. New monitoring tools as well as new operational changes led to modifications in organization of shifts. In this paper we describe the roles and the impacts of the shifts to smooth operation of complex computing grid employed in ATLAS, the influence of Discovery of Higgs like particle on shift operations, the achievements in monitoring and automation that made possible to focus more on tasks that leaded to the Discovery, as well as influence of the Long Shutdown 1 and operational changes related to no beam period.
Speaker: Alexey Sedov (Universitat Autònoma de Barcelona)
• 67
ATLAS DQ2 to Rucio renaming infrastructure
The current ATLAS Distributed Data Management system (DQ2) is being replaced by a new one called Rucio. The new system has many improvements, but it requires a number of changes. One of the most significant ones is that no local file catalog like the LFC, which was a central component in DQ2, will be used by Rucio. Instead of querying a file catalogue that stores the association of files with their corresponding locations, in Rucio the physical path of a file can be ascertained by deriving it from the file's Logical File Name (LFN) via a deterministic function. Therefore, all file replicas produced by ATLAS have to be renamed before migrating to the new system. It represents about 300M files split between about 120 sites with six different storage technologies. An infrastructure to perform this is needed: it should be automated, robust, transparent for the users, fault tolerant, storage technology agnostic and require as little work as possible from site administrative personnel. It needs also to be fast enough to rename everything before the final switch to Rucio in 2014. An infrastructure following all these requirements has been developed and is described here. The technologies that have been used are also presented as well as the performance of the system.
Speaker: Cedric Serfon (CERN)
• 68
ATLAS Experience with HEP Software at the Argonne Leadership Computing Facility
A number of HEP software packages used by the ATLAS experiment, including GEANT4, ROOT and ALPGEN, have been adapted to run on the IBM Blue Gene supercomputers at the Argonne Leadership Computing Facility. These computers use a non-x86 architecture and have a considerably less rich operating environment than in common use in HEP, but also represent a computing capacity an order of magnitude beyond what ATLAS is presently using via the LCG. The status and potential for making use of leadership-class computing, including the status of integration with the ATLAS production system, is discussed.
Speaker: Tom Uram (ANL)
• 69
The ATLAS Nightly Build System is a facility for automatic production of software releases. Being the major component of ATLAS software infrastructure, it supports more than 50 multi-platform branches of nightly releases and provides vast opportunities for testing new packages, for verifying patches to existing software, and for migrating to new platforms and compilers. The Nightly System testing framework runs several hundred integration tests of different granularity and purposes. The nightly releases are distributed and validated, and some are transformed into stable releases used for data processing worldwide. The first LHC long shutdown (2013-2015) activities will elicit increased load on the Nightly System as additional releases and builds are needed to exploit new programming techniques, languages, and profiling tools. This talk describes the program of the ATLAS Nightly Build System Long Shutdown upgrade. It brings modern database and web technologies into the Nightly System, improves monitoring of nightly build results, provides new tools for offline release shifters. We will also outline our long term plans for distributed nightly releases builds and testing.
Speaker: Dr Alexander Undrus (Brookhaven National Laboratory (US))
• 70
ATLAS software configuration and build tool optimisation
The ATLAS software code base is over 7 million lines organised in about 2000 packages. It makes use of some 100 external software packages, is developed by more than 400 developers and used by more than 2500 physicists from over 200 universities and laboratories in 6 continents. To meet the challenge of configuration and building of this software, the Configuration Management Tool (CMT) is used. CMT expects each package to describe its build targets, build and environment setup parameters, dependencies on other packages in a text file called requirements, and each project (group of packages) to describe its policies and dependencies on other projects in a text project file. Based on the effective set of configuration parameters read from the requirements files of dependent packages and project files, CMT commands build the packages, generate the environment for their use, or query the packages. The main focus was on build time performance that was optimised within several approaches: - reduction of the number of reads of requirements files that are now read once per package by a CMT build command that generates cached requirements files for subsequent CMT build commands; - introduction of more package level build parallelism, i.e., dependent applications and libraries are compiled in parallel; - code optimisation of CMT commands used for build; - introduction of project level build parallelism, i.e., parallelise the build of independent packages. By default, CMT launches NUMBER-OF-PROCESSORS build commands in parallel. The other focus was on CMT commands optimisation in general that made them about 2 times faster. CMT can generate a cached requirements file for the environment setup command, which is especially useful for deployment on distributed file systems like AFS or CERN VMFS. The use of parallelism, caching and code optimisation significantly - by several times - reduced software build time, environment setup time, increased the efficiency of multi-core computing resources utilisation, and considerably improved software developer and user experience.
Speaker: Grigori Rybkin (Universite de Paris-Sud 11 (FR))
• 71
Automated Cloud Provisioning Using Puppet & MCollective
Public clouds are quickly becoming cheap and easy methods for dynamically adding more computing resources to your local site to help handle peak computing demands. As cloud use continues to grow, the HEP community is looking to run more than just simple homogeneous VM images, which run basic data analysis batch jobs. The growing demand for heterogeneous server configurations demands better provisioning systems that can quickly and automatically instantiate various service VMs. In this paper, we discuss how we extend our local Puppet configuration management system to help us easily provision various service instances using the Marionette Collective framework.
Speaker: Jason Alexander Smith (Brookhaven National Laboratory (US))
• 72
Automated Configuration Validation with Puppet & Nagios
Running a stable production service environment is important in any field. To accomplish this, a proper configuration management system is necessary along with good change management policies. Proper testing and validation is required to protect yourself against software or configuration changes to production services that can cause major disruptions. In this paper, we discuss how we extended our Puppet configuration management system to automatically run all configuration changes through a validation environment, which replicates all critical services and warns us of potential problems before applying these changes to our production servers.
Speaker: Jason Alexander Smith (Brookhaven National Laboratory (US))
• 73
Automatic Tools for Enhancing the Collaborative Experience in Large Projects
With the explosion of big data in many fields, the efficient management of knowledge about all aspects of the data analysis gains in importance. A key feature of collaboration in large scale projects is keeping a log of what and how is being done - for private use and reuse and for sharing selected parts with collaborators and peers, often distributed geographically on an increasingly global scale. Even better if this log is automatic, created on the fly while a scientist or software developer is working in a habitual way, without the need for extra efforts. This saves human time and enables a team to do more with the same resources. The CODESH - COllaborative DEvelopment SHell - and CAVES - Collaborative Analysis Versioning Environment System projects address this problem in a novel way. They build on the concepts of virtual states and transitions to enhance the collaborative experience by providing automatic persistent virtual logbooks. CAVES is designed for sessions of distributed data analysis using the popular ROOT framework, while CODESH generalizes the same approach for any type of work on the command line in typical UNIX shells like bash or tcsh. Repositories of sessions can be configured dynamically to record and make available the knowledge accumulated in the course of a scientific or software endeavor. Access can be controlled to define logbooks of private sessions or sessions shared within or between collaborating groups. A typical use case is building working scalable systems for analysis of Petascale volumes of data as encountered in the LHC experiments. Our approach is general enough to find applications in many fields.
Speaker: Dr Jorge Luis Rodriguez (UNIVERSITY OF FLORIDA)
• 74
AutoPyFactory and the Cloud: Flexible, scalable, and automatic management of virtual resources for ATLAS
AutoPyFactory (APF) is a next-generation pilot submission framework that has been used as part of the ATLAS workload management system (PanDA) for two years. APF is reliable, scalable, and offers easy and flexible configuration. Using a plugin-based architecture, APF polls for information from configured information and batch systems (including grid sites), decides how many additional pilot jobs are needed, and submits them. With the advent of cloud computing, providing resources goes beyond submitting pilots to grid sites. Now, the resources on which the pilot will run also need to be managed. Handling both pilot submission and controlling the virtual machine life cycle (creation, retirement, and termination) from the same framework allows robust and efficient management of the process. In this paper we describe the design and implementation of these virtual machine management capabilities of APF. Expanding on our plugin-based approach, we allow cascades of virtual resources associated with a job queue. A single workflow can be directed first to a private, facility-based cloud, then a free academic cloud, then spot-priced EC2 resources, and finally on-demand commercial clouds. Limits, weighting, and priorities are supported, allowing free or less expensive resources to be used first, with costly resources only used when necessary. As demand drops, resources are drained and terminated in reverse order. Performance plots and time series will be included, showing how the implementation handles ramp-ups, ramp-downs, and spot terminations.
Speaker: John Hover (Brookhaven National Laboratory (BNL)-Unknown-Unknown)
• 75
CDS Multimedia Services and Export
The volume of multimedia material produced by CERN is growing rapidly, fed by the increase of dissemination activities carried out by the various outreach teams, such as the central CERN Communication unit and the Experiments Outreach committees. In order for this multimedia content to be stored digitally for the long term, to be made available to end-users in the best possible conditions and finally to be easily re-usable in various contexts e.g. by web site creators or by media professionals, various new services and export mechanisms have been set up in the CERN Document Server Multimedia collections. This talk will explain which technologies have been selected, the challenges satisfying all these needs together, and how users can take advantage of the newly developed facilities. In particular we will present the developments in video and image embedding, image slideshow generation, embedding via oEmbed, image processing, and image and video retrieval and ranking. The underlying architecture of the CDS multimedia platform (Invenio based), responsible for ensuring a robust, flexible and scalable service will also be described.
Speaker: Ludmila Marian (CERN)
• 76
CernVM-FS - Beyond LHC Computing
In the last three years the CernVM Filesystem (CernVM-FS) has transformed the distribution of experiment software to WLCG grid sites. CernVM-FS removes the need for local installations jobs and performant network fileservers at sites, in addition it often improves performance at the same time. Furthermore the use of CernVM-FS standardizes the computing environment across the grid and removes the need for software tagging at sites. Now established and proven to work at scale, CernVM-FS is beginning to perform a similar role for non-LHC computing. We discuss the deployment of a Stratum 0 'master' CernVM-FS repository at the RAL Tier 1 and the development of a network of Stratum 1 replicas somewhat modeled upon the infrastructure developed to support WLCG computing. We include a case study of one small non-LHC virtual organisation, describing their use of the CernVM-FS Stratum 0 service. We examine the impact of using CernVM-FS upon their ability to utilize a wider range of resources across the UK GridPP network.
Speaker: Ian Peter Collier (STFC - Science & Technology Facilities Council (GB))
• 77
Changing the batch system in a Tier 1 computing center: why and how
At the Italian Tier1 Center at CNAF we are evaluating the possibility to change the current production batch system. This activity is motivated mainly because we are looking for a more flexible licensing model as well as to avoid vendor lock-in. We performed a technology tracking exercise and among many possible solutions we chose to evaluate Grid Engine as an alternative because its adoption is increasing in the HEPiX community and because it's supported by the EMI middleware that we currently use on our computing farm. Another INFN site evaluated Slurm and we will compare our results in order to understand pros and cons of the two solutions. We will present the results of our evaluatin of Grid Engine, in order to understand if it can fit the requirements of a Tier 1 center, compared to the solution we adopted long ago. We performed a survey and a critical re-evaluation of our farming infrastructure: many production softwares (accounting and monitoring on top of all) rely on our current solution and changing it required us to write new wrappers and adapt the infrastructure to the new system. We believe the results of this investigation can be very useful to other Tier-1s and Tier-2s centers in a similar situation, where the effort of switching may appear too hard to stand. We will provide guidelines in order to understand how difficult this operation can be and how long the change may take.
Speaker: Stefano Dal Pra (Unknown)
• 78
Cloud flexibility using Dirac Interware
Communities of different locations are running their computing jobs on dedicated infrastructures without the need to worry about software, hardware or even the site where their programs are going to be executed. Nevertheless, this usually implies that they are restricted to use certain types or versions of an Operating System because either their software needs an definite version of a system library or a specific platform is required by the collaboration to which they belong. On this scenario, if a data center wants to service software incompatible communities, it has to split its physical resources among those communities. This splitting will inevitably lead to an underuse of resources because the data centers are bound to have periods where one or more of its subclusters are idle. It is in this situation where Cloud Computing provides the flexibility and reduction in computational cost that data centers are searching for. This paper describes a set of realistic tests that we ran on one of such implementations. The test comprise software from three different HEP communities (Auger, LHCb and QCD phenomelogists) and the Parsec Benchmark Suite running on one or more of three Linux flavors (SL5, Ubuntu 10.04 and Fedora 13). The implemented infrastructure has, at the cloud level, CloudStack that manages the virtual machines (VM) and the hosts on which they run, and, at the user level, the DIRAC framework along with a VM extension that will submit, monitorize and keep track of the user jobs and also requests CloudStack to start or stop the necessary VM's. In this infrastructure, the community software is distributed via the CernVM-FS, which has been proven to be a reliable and scalable software distribution system. With the resulting infrastructure, users are allowed to send their jobs transparently to the Data Center. The main purpose of this system is the creation of flexible cluster, multiplatform with an scalable method for software distribution for several VOs. Users from different communities do not need to care about the installation of the standard software that is available at the nodes, nor the operating system of the host machine, which is transparent to the user.
Speaker: Victor Manuel Fernandez Albor (Universidade de Santiago de Compostela (ES))
• 79
CMS Data Analysis School Model
To impart hands-on training in physics analysis, CMS experiment initiated the concept of CMS Data Analysis School (CMSDAS). It was born three years ago at the LPC (LHC Physics Center), Fermilab and is based on earlier workshops held at the LPC and CLEO Experiment. As CMS transitioned from construction to the data taking mode, the nature of earlier training also evolved to include more of analysis tools, software tutorials and physics analysis. This effort epitomized as CMSDAS has proven to be a key for the new and young physicists to jump start and contribute to the physics goals of CMS by looking for new physics with the collision data. With over 400 physicists trained in six CMSDAS around the globe , CMS is trying to engage the collaboration discovery potential and maximize the physics output. As a bigger goal, CMS is striving to nurture and increase engagement of the myriad talents of CMS, in the development of physics, service, upgrade, education of those new to CMS and the career development of younger members. An extension of the concept to the dedicated software and hardware schools is also planned, keeping in mind the ensuing upgrade phase.
Speaker: Kenneth Bloom (University of Nebraska (US))
• 80
CMS experience of running glideinWMS in High Availability mode
The CMS experiment at the Large Hadron Collider is relying on the HTCondor-based glideinWMS batch system to handle most of its distributed computing needs. In order to minimize the risk of disruptions due to software and hardware problems, and also to simplify the maintenance procedures, CMS has set up its glideinWMS instance to use most of the attainable High Availability (HA) features. The setup involves running services distributed over multiple nodes, which in turn are located in several physical locations, including Geneva, Switzerland, Chicago, Illinois and San Diego, California. This paper describes the setup used by CMS, the HA limits of this setup, as well as a description of the actual operational experience spanning many months.
Speaker: Mr Igor Sfiligoi (University of California San Diego)
• 81
CMS geometry through 2020
CMS faces real challenges with upgrade of the CMS detector through 2020. One of the challenges, from the software point of view, is managing upgrade simulations with the same software release as the 2013 scenario. We present the CMS geometry description software model, its integration with the CMS event setup and core software. The CMS geometry configuration and selection is implemented in Python. The tools collect the Python configuration fragments into a script used in CMS workflow. This flexible and automated geometry configuration allows choosing either transient or persistent version of the same scenario and specific version of the same scenario. We describe how the geometries are integrated and validated, how we define and handle different geometry scenarios in simulation and reconstruction. We discuss how to transparently manage multiple incompatible geometries in the same software release. Several examples are shown based on current implementation assuring consistent choice of scenario conditions. The consequences and implications for multiple/different code algorithms are discussed.
Speaker: Mrs Ianna Osborne (Fermi National Accelerator Lab. (US))
• 82
CMS Space Monitoring
During the first LHC run, CMS saturated one hundred petabytes of storage resources with data. Storage accounting and monitoring help to meet the challenges of storage management, such as efficient space utilization, fair share between users and groups, and further resource planning. We present newly developed CMS space monitoring system based on the storage dumps produced at the sites. Storage contents information is aggregated and uploaded to the central database. Web based data service is provided to retrieve the information for a given time interval and a range of sites, so it can be further aggregated and presented in the desired format. The system has been designed based on the analysis of CMS monitoring requirements and experiences of the other LHC experiments. In this paper, we demonstrate how the existing software components of the CMS data placement system PhEDEx have been re-used, reducing dramatically the development effort.
Speaker: Dr Tony Wildish (Princeton University (US))
• 83
CMS users data management service integration and first experiences with its NoSQL data storage
The distributed data analysis workflow in CMS assumes that jobs run in a different location to where their results are finally stored. Typically the user outputs must be transferred from one site to another by a dedicated CMS service, AsyncStageOut. This new service is originally developed to address the inefficiency in using the CMS computing resources when transferring the analysis job outputs, synchronously, once they are produced in the job execution node to the remote site. The AsyncStageOut is designed as a thin application relying only on the NoSQL database (CouchDB) as input and data storage. It has progressed from a limited prototype to a highly adaptable service which manages and monitors the whole user files steps, namely file transfer and publication. The AsyncStageOut is integrated with the Common CMS/Atlas Analysis Framework. It foresees the management of nearly 200k users files per day of close to 1000 individual users per month with minimal delays, and providing a real time monitoring and reports to users and service operators, while being highly available. The associated data volume represents a new set of challenges in the areas of database scalability and service performance and efficiency. In this paper, we present an overview of the AsyncStageOut model and the integration strategy with the Common Analysis Framework. The motivations for using the NoSQL technology are also presented, as well as data design and the techniques used for efficient indexing and monitoring of the data. We describe deployment model for the high availability and scalability of the service. We also discuss the hardware requirements and the results achieved as they were determined by testing with actual data and realistic loads during the commissioning and the initial production phase with the Common Analysis Framework.
Speaker: Marco Mascheroni (Universita & INFN, Milano-Bicocca (IT))
• 84
Common accounting system for monitoring the ATLAS Distributed Computing resources
The ATLAS Experiment at the Large Hadron Collider has been collecting data for three years. The ATLAS data are distributed, processed and analysed at more than 130 grid and cloud sites across the world. The total throughput of transfers is more than 5 GB/s and data occupies more than 120 PB on disk and tape storage. At any given time, there are more than 100,000 concurrent jobs running and more than a million jobs are submitted on a daily basis. The large scale activity of the ATLAS Distributed Computing (ADC) increases the level of complexity of the system and, thus, increases the probability of failures or inefficiencies in the involved components. Effective monitoring provides a comprehensive way to identify and address any issues with the infrastructure. It is also a key factor in the effective utilisation of the system. A significant effort has been invested over the last three years within the Experiment Dashboard project to assure effective and flexible monitoring. The Experiment Dashboard system provides generic solutions that cover all areas of ADC activities, such as data distribution and data processing over a large number of sites, and these solutions are extensively used by different categories of ATLAS users ranging from daily operations to resource management. This talk covers a common accounting system used to monitor the utilisation of the available computational and storage resources of ATLAS. This system provides quality and scalable solutions that are flexible enough to support the constantly evolving requirements of the ATLAS user community.
Speaker: Dr Edward Karavakis (CERN)
• 85
Compute Farm Software for ATLAS IBL Calibration
In 2014 the Insertable B-Layer (IBL) will extend the existing Pixel Detector of the ATLAS experiment at CERN by 12 million additional pixels. As with the already existing pixel layers, scanning and tuning procedures need to be employed for the IBL to account for aging effects and guarantee a unified response across the detector. Scanning the threshold or time-over-threshold of a front-end module consists of two main steps: gathering histograms of the detector response for different configurations and then fitting a target function to these histograms. Despite of the currently used method of performing the computationally demanding fits on DSPs located on the read-out hardware, it was decided to abandon this approach for IBL and realize the functionality on an external computing farm for easier development and greater flexibility. This not only requires the fast transfer of histogram data from the read-out hardware to the computing farm via Ethernet, but also the integration of the fit farm software and hardware into the already existing data-acquisition and calibration framework (TDAQ and PixelDAQ) of the ATLAS experiment and the current Pixel Detector. It is foreseen to implement the software running on the compute cluster with an emphasis on modularity, allowing for flexible adjustment of the infrastructure and a good scalability with respect to the number of network interfaces, available CPU cores, and deployed machines. By using a modular design we are able to not only employ CPU based fitting algorithms, but also have the possibility to take advantage of the performance offered by a GPU-based approach to fitting. We present the compute farm software architecture and report on the status of the implementation of the IBL calibration architecture into the ATLAS hardware and software framework. We discuss used test methods and point out obstacles that were encountered along the way.
Speaker: Moritz Kretz (Ruprecht-Karls-Universitaet Heidelberg (DE))
• 86
Computing challenges in the certification of ATLAS Tile Calorimeter front-end electronics during maintenance periods
After two years of operation of the LHC, the ATLAS Tile Calorimeter is undergoing the consolidation process of its front-end electronics. The first layer of certification of the repairs is performed in the experimental area with a portable test-bench which is capable of controlling and reading out all the inputs and outputs of one front-end module through dedicated cables. This testbench has been redesigned to improve the quality assessment of the data until the end of Phase I. It is now possible to identify low occurrence errors due to its increased read-out bandwidth and perform more sophisticated quality checks due to its enhanced computing power. Improved results provide fast and reliable feedback to the user.
Speaker: Carlos Solans Sanchez (CERN)
• 87
Continuous service improvement
Using the framework of ITIL best practises, the service managers within CERN-IT have engaged into a continuous improvement process, mainly focusing on service operation. This implies an explicit effort to understand and improve all service management aspects in order to increase efficiency and effectiveness. We will present the requirements, how they were addressed and share our experiences. We will describe how we measure, report and use the data to continually improve both the processes and the services being provided. The focus is not the tool or the process, but the results of the continuous improvement effort from a large team of IT experts providing services to thousands of users, supported by the tool and its local team.
Speaker: Line Everaerts (CERN)
• 88
Control functionality of DAQ-Middleware
DAQ-Middleware is a software framework for a network-distributed data acquisition (DAQ) system that is based on the Robot Technology Middleware (RTM). The framework consists of a DAQ-Component and a DAQ-Operator. The basic functionalities such as transferring data, starting and stopping the system, and so on, are already prepared in the DAQ-Components and DAQ-Operator. The DAQ-Component is especially used as a core component to read, transfer, and record data. And the DAQ-Operator is used as controlling the DAQ-Components. The system works as a state machine and it has four states, i.e., LOADED, CONFIGURED, RUNNING, PAUSED states. The states changed by the command to be transferred from the DAQ-Operator. The DAQ system can be easily implemented with using the framework. But the control functionalities, such as transferring parameters from DAQ-Operator to DAQ-Components, were not prepared, except when the state changes from LOADED to CONFIGRED. Then we developed the functionality to transfer data from DAQ-Operator to DAQ-Components at any time. The new functionality enables us to transfer any parameters that are stored in a configuration file on the DAQ-Operator to each DAQ-Component. In order to add this functionality, the new state, SET, is introduced. The DAQ-Operator transfers the specified configuration file to the specified DAQ-Components in this SET state. On the other hand, the DAQ-Components receive the configuration data from the DAQ-Operator in the SET state. Any parameters can be transferred and set with the DAQ-Middleware without restarting the system with using this functionality.
Speaker: Mr Hiroyuki Maeda (Hiroshima Institute of Technology)
• 89
CORAL and COOL during the LHC long shutdown
CORAL and COOL are two software packages that are widely used by the LHC experiments for the management of conditions data and other types of data using relational database technologies. They have been developed and maintained within the LCG Persistency Framework, a common project of the CERN IT department with ATLAS, CMS and LHCb. The project used to include the POOL software package, providing a hybrid store for C++ objects, which is now maintained by ATLAS. This presentation will report on the status of CORAL and COOL at the time of CHEP2013. It will cover the new features in CORAL and COOL (such as Kerberos authentication to Oracle and the prototyping of CORAL server monitoring), as well as the changes and improvements in the software process infrastructure (better integration with various tools, new code repository, new platforms). It will also review the usage of the software in the experiments and the outlook for the project during the LHC long shutdown (LS1) and beyond.
Speaker: Dr Andrea Valassi (CERN)
• 90
DAQ Architecture for the LHCb Upgrade
LHCb will have an upgrade of its detector in 2018. After the upgrade, the LHCb experiment will run at a high luminosity of 2x 10^33 cm^-2 . s^-1. The upgraded detector will be read out at 40 MHz with a highly flexible software-based triggering strategy. The Data Acquisition (DAQ) system of HCb reads out the data fragments from the Front-End Electronics and transports them to the High-Lever Trigger farm at an aggregate throughput of ~32 Tbit/s. The DAQ system will be based on high speed network technologies such as InfiniBand and/or 10/40/100 Gigabit Ethernet. Independent of the network technology, there are different possible architectures for the DAQ system. In this paper, we present our studies on the DAQ architecture, where we analyze size, complexity and (relative) cost. We evaluate and compare several data-flow schemes for a network-based DAQ: push, pull and push with barrel-shifter traffic shaping. We also discuss the requirements and overall implications of the data-flow schemes on the DAQ system.
Speaker: Niko Neufeld (CERN)
• 91
Data Acquisition of A totally Active Scintillator Calorimeter of the Muon Ionization Cooling Experiment
The Electron-Muon Ranger (EMR) is a totally active scintillator detector which will be installed in the muon beam of the Muon Ionization Cooling Experiment (MICE), the main R&D project for a future neutrino factory. It is designed to measure the properties of a low energy beam composed of muons, electrons and pions, and to perform an identification on a particle by particle basis. The EMR is made up of 48 intersecting layers, each of which is made of 59 triangular bars. Wavelength shifting fibers incorporated into the bars trap and transfer light, generated by particles traversing the detector, to PMTs located at the ends of the bars. One side is read out by single-anode PHILIPS XP2972 PMTs and the other by 64-ch. HAMAMATSU R7600 PMTs. Signals from 48 single-anode PMTs are read out by 8 fast ADCs (CAEN V1731) which store pulse shapes with 2ns time resolution. Each 64-ch. PMT is interfaced to a front-end-board which hosts a MAROC ASIC that amplifies, discriminates, and shapes all input signals. Pulse height information can be extracted at low rate and will be used during calibration and tests with cosmic rays. Fast discriminated signals from the front-end-board are directed to a piggy-back buffer board which stores all the signals created during one duty cycle of the accelerator. Six buffer boards are connected in a chain and read out by a dedicated VME card. Communication between the buffer and VME board is made through TLK chip to allow for fast data transfer. The front-end-boards also have corresponding VME readout boards which at the same time configure the MAROC ASICs. All electronics boards employ FPGA chips which allow for great customization of the detector behavior. The complete read out system of the EMR will be described, including the hardware, firmware, and results of a full system test using cosmic rays.
Speakers: Ruslan Asfandiyarov (Universite de Geneve (CH)), Yordan Ivanov Karadzhov (Universite de Geneve (CH))
• 92
Data Driven Trigger Algorithms to Search for Exotic Physics in the NOvA Detector
The NOvA experiment at Fermi National Accelerator Lab, due to its unique readout and buffering design, is capable of accessing physics beyond the core neutrino oscillations program for which it was built. In particular the experiment is able to search for evidence of relic cosmic magnetic monopoles and for the signs of the neutrino flash from a near by supernova through uses of a specialized triggering system that is able to be self driven off of the raw readout stream. These types of “data driven triggers” require fast track reconstruction and rejection over a wide range of time windows and detector conditions. The amount of data that the NOvA detector produces and the continuous streaming nature of the data make this type of real time reconstruction is a challenge for modern computing. To meet these challenges a fast track reconstruction algorithm has been developed based on N-dimensional Hough Transform which is able to meet the latency requirements of the NOvA data acquisition system. This algorithm forms the basis of both the searches for magnetic monopoles, where it is able to separate out the slow moving monopoles from the prompt cosmic backgrounds, as well as for the supernova detection trigger, where it is used to subtract out the cosmic ray background and leave the low energy EM shower candidates. In this paper, we will discuss in details of this algorithm and show how it can be used to distinguish and reconstruct slow magnetic monopole tracks verse fast cosmic ray tracks through an expansion of the Hough space to include velocity and pulse height information. We will discuss the scaling of the algorithm and its performance when run on real data. Examples of cosmic rate rejection and the improvements to supernova detection will be discussed and examples of the algorithm and trigger performance will shown for the early running of the NOvA far detector.
Speakers: Evan Niner (Indiana University), Mr Zukai Wang (University of Virginia)
• 93
Data Preservation activities at DESY (The DESY-DPHEP Group)
The data preservation project at DESY was established in 2008, shortly after data taking ended at the HERA ep collider, soon after coming under the umbrella of the DPHEP global initiative. All experiments are implementing data preservation schemes to allow long term analysis of their data, in cooperation with the DESY-IT division. These novel schemes include software validation and migration techniques as well as archival storage of the data themselves. In addition to the preservation of data and software, a consolidation programme of all digital and non-digital documentation, some of which dates back to the 1980s, is being performed, including projects with the INSPIRE initiative. The activities of the group and their relevance and portability to other HEP experiments will be presented.
Speaker: katarzyna wichmann (DESY)
• 94
Data Preservation at the CDF Experiment
The Fermilab Tevatron collider's data-taking run ended in September 2011, yielding a dataset with rich scientific potential. The CDF experiment has nearly 9 PB of collider and simulated data stored on tape. A large computing infrastructure consisting of tape storage, disk cache, and distributed grid computing for physics analysis with the CDF data is present at Fermilab. The Fermilab Run II data preservation project intends to keep this analysis capability sustained through the year 2020 or beyond. To achieve this, we are implementing a system that utilizes virtualization, automated validation, and migration to new standards in both software and data storage technology as well as leveraging resources available from currently-running experiments at Fermilab. These efforts will provide useful lessons in ensuring long-term data access for numerous experiments throughout high-energy physics, and provide a roadmap for high-quality scientific output for years to come. We will present a talk on the status, benefits, and challenges of data preservation efforts within the CDF collaboration at Fermilab.
Speaker: Dr Bodhitha Jayatilaka (Fermilab)
• 95
Data Preservation at the D0 Experiment
The Tevatron experiments have entered their post-data-taking phases but are still producing physics output at a high rate. The D0 experiment has initiated efforts to preserve both data access and full analysis capability for the collaboration members through at least 2020. These efforts will provide useful lessons in ensuring long-term data access for numerous experiments throughout high-energy physics, and provide a roadmap for high-quality scientific output for years to come. D0 is making a number of changes to its computing infrastructure to retain analysis capability and maintain long-term data access. These changes include transitioning to newer versions of data handling tools, virtualizing database servers and batch worker nodes, modifying job submission scripts, migrating to new data storage technology, and developing tools for automated validation of physics software on future OS platforms. We will present a talk describing the benefits of long-term data preservation, the status of changes to the D0 computing infrastructure, the challenges of data preservation efforts, and plans for the next several years within the D0 collaboration at Fermilab.
Speaker: Dr Michael Kirby (Fermi National Accelerator Laboratory)
• 96
Dataset-based High-Level Data Transfer System in BESDIRAC
Data Transfer is an essential part in grid. In the BESIII experiment, the result of Monte Carlo Simulation should be transfered back from other sites to IHEP and the DST files for physics analysis should be tranfered from IHEP to other sites. A robust transfer system should make sure all data are transfered correctly. DIRAC consists of cooperation distributed services and light-weight agents delivering the workload to the Grid Resources. We reuse the most basic functionalities supplied by DIRAC. BESDIRAC is an extension to DIRAC for BES specified. In BESDIRAC, a Dataset-based Data Transfer System is developed. In this paper, we present the design of this system and its implementation details. A Transfer Request Service is used for creating and monitoring the transfer requests. A Transfer Agent is used for transfering data from one SE to another. For flexibility and reuse of the current low-level transfer systems, we have designed a transfer worker factory to create transfer workers with different protocols. A transfer worker is the wrapper of the low-level file transfer commands. The Transfer Agent uses the Async I/O to manage the transfer workers.
Speaker: Mr Tao Lin (Institute of High Energy Physics)
• 97
dCache Billing data analysis with Hadoop
The dCache storage system writes billing data into flat files or a relational database. For a midsize dCache installation there are one million entries - representing 300 MByte - per day. Gathering accounting information for a longer time interval about transfer rates per group, per file type or per user results in increasing load on the servers holding the billing information. Speeding up these requests renders new approaches to performance optimization worthwhile. Hadoop is a framework for distributed processing of large data using multiple computer nodes. The essential point in our context is the scalability for big data. Data is distributed over many nodes in the Hadoop Distributed File System (HDFS). Queries are processed in parallel on every node to extract the information and combine it in another step. This is called a MapReduce algorithm. As the dCache storage is distributed over many storage nodes combining both on every node is obvious. The transformation of the billing information into the HDFS structure is done by a small script. The MapReduce functions to create the results to the most important queries are implemented for each request. We will present the system's setup and performance comparisons of the created queries using Postgresql, flat files and Hadoop. The overall gain in performance and its dependence on both the amount of analysed data and available machines for paralleling the request will be demonstrated.
Speaker: Kai Leffhalm (Deutsches Elektronen-Synchrotron (DE))
• 98
DCS Data Viewer, a Application that Access ATLAS DCS historical Data
The ATLAS experiment at CERN is one of the four Large Hadron Collider experiments. The DCS Data Viewer (DDV) is an application that provides access to historical data of the ATLAS Detector Control System (DCS) parameters and their corresponding alarm information. It features a server-client architecture: the pythonic server serves as interface to the Oracle-based conditions database and can be operated stand alone using http requests; the client is developed with the Google Web Toolkit (GWT) and offers a user friendly browser independent web interface. The client data visualization is done using various output plugins such as java or javascript applets which are integrated using an open JSON interface allowing for easy plugin development. The default output provides charts, histograms or tables with broad filtering and sorting capabilities. Further, export to ROOT files is supported and smartphone compatibility is taken into consideration. A server-based configuration storage facility allows e.g. for sharing of resulting plots or embedding into other web applications invoking the tool with a single configuration URL. Web security constraints along with database dedicated protection mechanisms permit a successful exposure of the tool to hundreds of collaborators worldwide.
Speaker: Gancho Dimitrov (CERN)
• 99
Deploying an IPv6-enabled grid testbed at GridKa
GridKa, the German WLCG Tier-1 site hosted by Steinbuch Centre for Computing at Karlsruhe Institute of Thechnology, is a collaboration partner in the HEPIX-IPv6 testbed. A special IPv6-enabled gridftp server has been installed previously. In 2013, the IPv6 efforts will be increased. Already the installation of a new Mini-Grid site has been started. This Mini-Grid installation is planned as a dual-stack IPv4/IPv6 environment and will contain the current services of GridKa. Mainly the following EMI services BDII, Cream-CE, WorkerNode, dCache, xrootd as well as a Grid Engine job scheduler will be deployed. The Mini-Grid setup has initially been started with IPv4 only. Hence the IPv6 readiness of each service is to be evaluated. There are several other initiatives analyzing the IPv6 readiness of WLCG software. There is EMI evaluating the middleware, the HEPIX-IPv6 working group evaluating the readiness of transport protocols, EGI IPv6 working group evaluating non-WLCG Grid middleware (e.g. unicore). Our testbed is meant to offer an installation basis for the initiatives in the Grid framework and enable them to use it with as little effort as possible. The paper shows the setup of the IPv6 testbed in detail. It illustrates the IPv6 readiness of the different application and the services offered as well. It also highlights the problems that occurred during the deployment of the testbed and how these obstacles were overcome by thorough investigation and evaluation of all included programs.
Speaker: Andreas Petzold (KIT)
• 100
Design and Performance of the Virtualization Platform for Offline computing on the ATLAS TDAQ Farm
With the LHC collider at CERN currently going through the period of Long Shutdown 1 (LS1) there is a remarkable opportunity to use the computing resources of the large trigger farms of the experiments for other data processing activities. In the case of ATLAS experiment the TDAQ farm, consisting of more than 1500 compute nodes, is particularly suitable for running Monte Carlo production jobs that are mostly CPU and not I/O bound. This contribution gives a thorough review of all the stages of “Sim@P1” project dedicated to the design and deployment of a virtualized platform running on the ATLAS TDAQ computing resources and using it to run the large groups of CernVM based virtual machines operating like a single “CERN-­‐P1” Grid site. This platform has been designed to avoid interference with TDAQ usage of the farm and to guarantee the security and the usability of the ATLAS private network; Openstack has been chosen to provide a cloud management layer. The approaches to organizing support for the sustained operation of the system on both infrastructural (hardware, virtualization platform) and logical (site support and job execution) levels are also discussed. The project is a result of combined effort of the ATLAS TDAQ SysAdmins and NetAdmins teams, CERN IT ES Department and RHIC & ATLAS Computing Facility at BNL.
Speaker: Franco Brasolin (Universita e INFN (IT))
• 101
Detector and Event Visualization with SketchUp at the CMS Experiment
We describe the creation of 3D models of the CMS detector and events using SketchUp, a 3D modelling program. SketchUp provides a Ruby API with which we interface with the CMS Detector Description, the master source of the CMS detector geometry, to create detailed 3D models of the CMS detector. With the Ruby API we also interface with the JSON-based event format used for the iSpy event display to create 3D models of events. These models have many applications related to 3D representation of the CMS detector and events. Figures produced based on the models were used in conference presentations, paper publications, technical design reports for the detector upgrade, and other formal and informal documentation. Furthermore, several art projects, exhibitions, and outreach programs using the models are being planned. We describe technical implementation, show models created, and discuss present and future applications.
Speaker: Tai Sakuma (Texas A & M University (US))
• 102
Development and application of CATIA-GDML geometry builder
Detector geometry exchange between CAD systems and physical Monte Carlo (MC), packages ROOT and Geant4 is a labor-consuming process necessary for fine design optimization. CAD and MC geometries have completely different structure and hierarchy. For this reason automatic conversion is possible only for very simple shapes. CATIA-GDML Geometry Builder is a tool which allows to facilitate significantly creation of MC compatible geometry in GDML format from the CAD system CATIA v.5 which is the main design system in CERN, GSI, LNGS, and other scientific and industrial entities. The tool was introduced at CHEP2010 [1].We employ powerful measurement, design and VBA customization features of various CATIA workbenches for creation of GDML compatible representation of an existing engineering assembly. For many Root/Geant primitives we have developed parameterized CATIA User Defined Features. We have implemented in CATIA concepts of logical, physical and mother volumes. The Constructive Solid Geometry (CSG) Boolean tree can be optimized from point of view of simulation performance. At the end the CSG tree is exported into GDML. For the last three years a number of detector models were transferred from CAD to MC and vice versa. Functionality of the tool was extended and usability was improved according to the practical experience. . The following novelties were the most important for improvement of usability: extension of the set of implemented primitives; introduction of polycone and polyhedra; introduction of a part template for MC compatible geometry; development of a correctness checker of a resulting model before export; automated positioning by clicking; generation of symmetry from an assembly; improved handling of materials. The up-to-date functionality, analysis of nontrivial use cases, an approach to integration with automated recognition of simple shapes, examples and hints on the best practices will be reported at the conference. [1] S. Belogurov, Yu. Berchun, et al. ‘CATIA-GDML GEOMETRY BUILDER’. Journal of Physics: Conference Series, Volume 331, Number 3, 2011 , pp. 32035-32040
Speaker: Sergey Belogurov (ITEP Institute for Theoretical and Experimental Physics (RU))
• 103
di-EOS - "distributed EOS": Initial experience with split-site persistency in a production service
After the strategic decision in 2011 to separate tier-0 activity from analysis, CERN-IT developed EOS as a new petascale disk-only solution to address the fast-growing needs for high-performance low-latency data access. EOS currently holds around 22PB usable space for the four big experiment (ALICE, ATLAS, CMS, LHCb), and we expect to grow to >30PB this year. EOS is one of the first production services to be running in CERN's new facility located in Budapest: we foresee to have about a third of total EOS storage capacity in the new facility in 2013, making it the largest storage service in the new CERN computer centre. We report on the initial experience running EOS as a distributed service (via the new CERN IT Agile Infrastructure tools and a new "remote-hands" contract) in a production environment, as well as on the particular challenges and solutions. Among these solutions we are investigating the stochastic geo-location of data replicas and countermeasures against "split-brain" scenarios for the new high-availability namespace. In addition we are considering optimized clients and draining procedures to avoid link overloads across the two CERN sites as well as maximising data-access and operations efficiency.
Speaker: Xavier Espinal Curull (CERN)
• 104
DIRAC framework evaluation for the Fermi-LAT, CTA and LSST experiments
DIRAC (Distributed Infrastructure with Remote Agent Control) is a general framework for the management of tasks over distributed heterogeneous computing environments. It has been originally developed to support the production activities of the LHCb (Large Hadron Collider Beauty) experiment and today is extensively used by several particle physics and biology communities. Current (Fermi-LAT, Fermi-Large Area Telescope) and planned (CTA, Cherenkov Telescope Array, LSST, Large Synoptic Survey Telescope) with very large processing and storage needs, are currently investigating the usability of DIRAC in this context. Each of these use cases has some peculiarities: Fermi-LAT will interface DIRAC to its own workflow system to allow the access to the grid resources; CTA is using DIRAC as workflow management system for the Monte Carlo production on the grid; LSST is exploring DIRAC to access to heterogeneous resources, like local clusters, grid and cloud. We describe the prototype effort that we lead toward deploying a DIRAC solution for some aspects of Fermi-LAT, CTA, and LSST needs.
Speaker: Luisa Arrabito (LUPM Université Montpellier 2, IN2P3/CNRS)
• 105
Dirac integration with a general purpose bookkeeping DB: a complete general suite
In HEP computing context, R&D studies aiming to the definition of the data and workload models were brought forward by the SuperB community beyond the experiment life itself. This work is considered of great interest for a generic mid- and small size VO to fulfil Grid exploiting requirements involving CPU-intensive tasks. We present the R&D line achievements in the design, developments and test of a distributed resource exploitation suite based on DIRAC. The main components of such a suite are the information system, the job wrapper and the new generation DIRAC framework. The DB schema and the SQL logic have been designed to be able to be adaptive with respect to the VO requirements in terms of physics application, job environment and bookkeeping parameters. A deep and flexible integration with DIRAC features has been obtained using SQLAlchemy technology allowing mapping and interaction with the information system. A new DIRAC extension has been developed to include this functionality along with a new set of DIRAC portal interfaces aimed to the job, distributed resources, and meta-data management. The results of the first functionality and efficiency tests will be reported.
Speakers: Dr Armando Fella (INFN Pisa), Mr Bruno Santeramo (INFN Bari), Cristian De Santis (Universita degli Studi di Roma Tor Vergata (IT)), Dr Giacinto Donvito (INFN-Bari), Marcin Jakub Chrzaszcz (Polish Academy of Sciences (PL)), Mr Milosz Zdybal (Institute of Nuclear Physics, Polish Academy of Science), Rafal Zbigniew Grzymkowski (P)
• 106
Disaster Recovery and Data Center Operational Continuity
The RHIC and ATLAS Computing Facility (RACF) at Brookhaven Lab is a dedicated data center serving the needs of the RHIC and US ATLAS community. Since it began operations in the mid-1990's, it has operated continuously with few unplanned downtimes. In the last 24 months, Brookhaven Lab has been affected by two hurricanes and a record-breaking snowstorm. In this presentation, we discuss lessons learned regarding (natural or man-made) disaster preparedness, operational continuity, remote access and safety protocols, including overall operational procedures developed as a result of these recent events.
Speaker: Dr Tony Wong (Brookhaven National Laboratory)
• 107
Distributed cluster testing using new virtualized framework for XRootD
The Extended ROOT Daemon (XRootD) is a distributed, scalable system for low-latency clustered data access. XRootD is mature and widely used in HEP, both standalone and as core functionality for the EOS system at CERN, and hence requires extensive testing to ensure general stability. However, there are many difficulties posed by distributed testing, such as cluster initialization, synchronization, orchestration, inter-cluster communication and controlled failure handling. A three-layer master/hypervisor/slave model is presented to ameloirate these difficulties by utilizing libvirt and QEMU/KVM virtualization technologies to automate spawning of configurable virtual clusters and orchestrate multi-stage test suites. The framework also incorporates a user-friendly web interface for scheduling and monitoring tests. The framework has been used successfully to build new test suites for XRootD and EOS with existing unit test integration. It is planned for the future to sufficiently generalize the framework to encourage usage by potentially any distributed system.
Speaker: Justin Lewis Salmon (University of the West of England (GB))
• 108
Distributed storage and cloud computing: a test case
Since 2003 the computing farm hosted by the INFN T3 facility in Trieste supports the activities of many scientific communities. Hundreds of jobs from 45 different VOs, including those of the LHC experiments, are processed simultaneously. The currently available shared disk space amounts to about 300 TB, while the computing power is provided by 712 cores for a total of 7400 HEP-SPEC06. Given that normally the requirements of the different computational communities are not synchronized, the probability that at any given time the resources owned by one of the participants are not fully utilized is quite high. A balanced compensation should in principle allocate the free resources to other users, but there are limits to this mechanism. In fact, the Trieste site may not hold the amount of data needed to attract enough analysis jobs, and even in that case there could be a lack of bandwidth for their access. The Trieste ALICE and CMS computing groups, in collaboration with other Italian groups, aim to overcome the limitations of existing solutions using two approaches. Sharing the data among all the participants, avoiding data duplication and taking full advantage of GARR-X wide area networks (10 GB/s) allows to distribute more efficiently the jobs according to the CPU availability, irrespective of the storage system size. Integrating the resources dedicated to batch analysis with the ones reserved for dynamic interactive analysis, through modern solutions as cloud computing, can further improve the use of the available computing power. The first tests of the investigated solutions for both distributed storage on wide area network and cloud computing approaches will be presented.
Speaker: Stefano Piano (INFN (IT))
• 109
Distributing CMS Data between the Florida T2 and T3 Centers using Lustre and Xrootd-fs
We have developed remote data access for large volumes of data over the Wide Area Network based on the Lustre filesystem and Kerberos authentication for security. It this paper we explore a prototype for two-step data access from worker nodes at Florida T3 centers, located behind a firewall and using a private network, to data hosted on the Lustre filesystem at the University of Florida CMS T2 center. The T2-T3 links are 10 Gigabit per second, and the typical round trip times are 10-15 msec. For each T3 center we use a client which mounts securely the Lustre filesystem and hosts a Xrootd server. The worker nodes access the data from the T3 client using POSIX compliant tools via the Xrootd-fs filesystem. We perform scalability tests with up to 200 jobs running in parallel on the T3 worker nodes.
Speaker: Dr Jorge Luis Rodriguez (UNIVERSITY OF FLORIDA)
• 110
Dynamic web cache publishing for IaaS clouds using Shoal
It has been shown possible to run HEP workloads on remote IaaS cloud resources. Typically each running Virtual Machine (VM) makes use of the CERN VM Filesystem (CVMFS), a caching HTTP file system, to minimize the size of the VM images, and to simplify software installation. Each VM must be configured with a HTTP web cache, usually a Squid Cache, in proximity in order to function efficiently.  Regular grid sites which use CVMFS to host worker node software areas use one or more Squid servers for their site. However, each IaaS cloud site has none of the static infrastructure associated with a typical HEP site. Each cloud VM must be configured to use a particular Squid server. We have developed a method and software application called Shoal for publishing squid caches which are dynamically created on IaaS clouds. The Shoal server provides a simple REST interface which allows clients to determine their closest Squid cache. Squid servers advertise their existence by running Shoal agent which uses the Advanced Message Queuing Protocol (AMQP)to publish information about the Squid server to the Shoal server. Having a method for exchanging the information rapidly allows for Squid servers to be instantiated in the cloud in response to load and for clients to quickly learn of their existence. In this work we describe the design of Shoal and evaluate its performance at scale in an operational system.
Speaker: Ian Gable (University of Victoria (CA))
• 111
Efficient computation of hash functions
The performance of hash function computations can impose a significant workload on SSL/TLS authentication servers. In the WLCG this workload shows also in the computation of data transfers checksums. It has been shown in the EGI grid infrastructure that the checksum computation can double the IO load for large file transfers leading to an increase in re-transfers and timeout errors. Storage managers like SToRM try to reduce that impact by computing the checksum during the transfer. That may not be feasible, however, when multiple transfer streams are combined with the use of hashes like MD-5 or SHA-2. We present two alternatives to reduce the hash computation load. First we introduce implementations for the Fast SHA-256 and SHA-512 that can reduce the number of cycles per second of a hash computation from 15 to under 11. Secondly we introduce and evaluate parallel implementations for two novel hash tree functions: NIST SHA-3 Keccak and Skein. These functions were conceived to take advantage of parallel data transfers and their deployment can significantly reduce the timeout and re-transfer errors mentioned above.
Speaker: Dr raul lopes (School of Design and Engineering - Brunel University, UK)
• 112
Enabling IPv6 at FZU - WLCG Tier2 in Prague
The production usage of the new IPv6 protocol is becoming reality in the HEP community and the Computing Centre of the Institute of Physics in Prague participates in many IPv6 related activities. Our contribution will present experience with monitoring in HEPiX distributed IPv6 testbed which includes 11 remote sites. We use Nagios to check availability of services and Smokeping for monitoring the network latency. It is not always trivial to setup DNS in a dual stack environment properly therefore we developed a Nagios plugin for checking whether a domain name is resolvable when using only IP protocol version 6 and only version 4. We will also present local area network monitoring and tuning related to IPv6 performance. One of the most important software for a grid site is a batch system for job execution. We will present our experience with configuring and running Torque and Slurm batch systems in dual stack environment. And we will discuss the steps needed to run VO specific jobs in our IPv6 testbed.
Speaker: Tomas Kouba (Acad. of Sciences of the Czech Rep. (CZ))
• 113
Estimating job runtime for CMS analysis jobs
The basic premise of pilot systems is to create an overlay scheduling system on top of leased resources. And by definition, leases have a limited lifetime, so any job that is scheduled on such resources must finish before the lease is over, or it will be killed and all the computation wasted. In order to effectively schedule jobs to resources, the pilot system thus requires the expected lifetime of the jobs. Past studies have shown that relying on user provided estimates is not a valid strategy, so the system should try to make an estimate by itself. This paper provides a description of a system that makes estimates using machine learning based on past behavior. The work was performed in the context of physics analysis jobs of the CMS experiment at the Large Hadron Collider, using the domain knowledge to improve the accuracy. The attained results are presented in the paper.
Speaker: Mr Igor Sfiligoi (University of California San Diego)
• 114
Evaluating Tier-1 Sized Online Storage Solutions
The Fermilab CMS Tier-1 facility provides processing, networking, and storage as one of seven Tier-1 facilities for the CMS experiment. The storage consists of approximately 15 PB of online/nearline disk managed by the dCache file system, and 22 PB of tape managed by the Enstore mass storage system. Data is transferred to and from computing centers worldwide using the CMS-developed PhEDEx transfer management system. Prior to 2013, control over which files were staged on all Tier-1 nearline storage was provided on a site-by-site basis, despite the fact that the decisions were made centrally by the CMS Computing Operations team. We were required to change this model by mapping two separate PhEDEx transfer endpoints to our storage, allowing CMS to use this tool to manage Tier-1 nearline storage worldwide. In this paper, we evaluate various storage management solutions, most of which involve migrating the bulk of the disk (~13 PB) to another system. Hadoop, Lustre, EOS, dCache 2.x were the systems under consideration. We will present the results from evaluating the performance of metadata services, storage services, and support and operations models.
Speaker: Ian Fisk (Fermi National Accelerator Lab. (US))
• 115
Evaluation of Apache Hadoop for Parallel Data Analysis with ROOT
The Apache Hadoop software is a Java based framework for distributed processing of large data sets across clusters of computers using the Hadoop file system (HDFS) for data storage and backup and MapReduce as a processing platform. Hadoop is primarily designed for processing large textual data sets which can be processed in arbitrary chunks, and must be adapted to the use case of processing binary data files which can not be split automatically. However, Hadoop offers attractive features in terms of fault tolerance, task supervision and controlling, multi-user functionality and job management. For this reason, we have evaluated Apache Hadoop as an alternative approach to PROOF for root data analysis. Two alternatives in distributing analysis data are discussed: Either the data is stored in HDFS and processed with MapReduce, or the data is accessed via a standard Grid storage system (dCache Tier-2) and MapReduce was used only as execution back-end. The focus in the measurements are on the one hand to safely store analysis data on HDFS with reasonable data rates and on the other hand to process data fast and reliably with MapReduce. For evaluation of the HDFS, data rates for writing to and reading from local Hadoop cluster have been measured and are compared to normal data rates on the local NFS. For evaluation of MapReduce, realistic ROOT analyses have been used and event rates were compared to PROOF.
Speakers: Guenter Duckeck (Experimentalphysik-Fakultaet fuer Physik-Ludwig-Maximilians-Uni), Dr Johannes Ebke (Ludwig-Maximilians-Univ. Muenchen (DE)), Sebastian Lehrack (LMU Munich)
• 116
Event processing time prediction at the CMS Experiment of the Large Hadron Collider
The physics event reconstruction in LHC/CMS is one of the biggest challenges for computing. Among the different tasks that computing systems perform, the reconstruction takes most of the CPU resources that are available. The reconstruction time of a single event varies according to the event complexity. Measurements were done in order to find precisely this correlation, creating means to predict it based on the physics conditions of the input data. Currently the data processing system do not account that when splitting a task in chunks(jobs), this can cause a considerable variation in the job length, thus a considerable increase into the workflow Estimated Time of Arrival. The goal is to use this estimate on processing time to more efficiently split the work in chunks, considering the CPU time needed for each chunk and due to this, lowering the standard deviation of the job length distribution in a workflow.
Speaker: Ian Fisk (Fermi National Accelerator Lab. (US))
• 117
Evolution of the pilot infrastructure of CMS: towards a single glideinWMS pool
CMS production and analysis job submission is based largely on glideinWMS and pilot submissions. The transition from multiple different submission solutions like gLite WMS and HTCondor-based implementations was carried out over years and is coming now to a conclusion. The historically explained separate glideinWMS pools for different types of production jobs and analysis jobs are being unified into a single global pool. This enables CMS to benefit from global prioritization and scheduling possibilities. It also presents the sites with only one kind of pilots and eliminates the need of having to make scheduling decisions on the CE level. This paper provides an analysis of the benefits of a unified resource pool, as well as a description of the resulting global policy. It will explain the technical challenges moving forward and present solutions to some of them.
Speaker: Ian Fisk (Fermi National Accelerator Lab. (US))
• 118
Experience with a frozen computational framework from LEP age
The strategy at the end of the LEP era for the long term preservation of physics results and data processing framework was not obvious. One of the possibilities analyzed at the time, previously to the generalization of virtualization techniques, was the setup of a dedicated farm, to be conserved in its original state for medium-long term, at least until the new data from LHC could indicate the need to reanalyze LEP data, the most significant example being the Higgs boson search. Such an infrastructure was setup at IFCA in 2001, including 80 equal servers where the software of the DELPHI experiment was installed and tested, and analysis ntuples and code were stored. This set of servers have been periodically restarted and tested, to examine the feasibility of this approach for complete preservation, and allow a detailed comparison with the approach based on the use of virtual machines. In parallel, all DELPHI data at DST (Data Summary Tapes) level were copied to IFCA and stored in tape in LTO-1 format. The latest results at LHC indicate that there will be likely no need to reanalyze LEP data. This contribution describes this experience, the results obtained after more than 10 years of "freezing", and concludes with the lessons learnt in this cycle across two generation of experiments.
Speaker: Prof. Jesus Marco (IFCA (CSIC-UC) Santander Spain)
• 119
Experience with Intel's Many Integrated Core Architecture in ATLAS Software
Intel recently released the first commercial boards of its Many Integrated Core (MIC) Architecture. MIC is Intel's solution for the domain of throughput computing, currently dominated by general purpose programming on graphics processors (GPGPU). MIC allows the use of the more familiar x86 programming model and supports standard technologies such as OpenMP, MPI, and Intel's Threading Building Blocks. This should make it possible to develop for both throughput and latency devices using a single code base. In ATLAS Software, track reconstruction has been shown to be a good candidate for throughput computing on GPGPU devices. In addition, the newly proposed offline parallel event-processing framework, GaudiHive, uses TBB for task scheduling. The MIC is thus, in principle, a good fit for this domain. In this presentation, we report our experiences of porting to and optimizing ATLAS tracking algorithms for the MIC, comparing the programmability and relative cost/performance of the MIC against those of current GPGPUs and latency-optimized CPUs.
Speaker: Wim Lavrijsen (Lawrence Berkeley National Lab. (US))
• 120
Experiment Dashboard Task Monitor for managing ATLAS user analysis on the Grid
The organization of the distributed user analysis on the Worldwide LHC Computing Grid (WLCG) infrastructure is one of the most challenging tasks among the computing activities at the Large Hadron Collider. The Experiment Dashboard offers a solution that not only monitors but also manages (kill, resubmit) user tasks and jobs via a web interface. The ATLAS Dashboard Task Monitor provides analysis users with a tool that is operating system and Grid environment independent. This contribution describes the functionality of the application and its implementation details, in particular authentication, authorization and audit of the management operations.
Speaker: Laura Sargsyan (ANSL (Yerevan Physics Institute) (AM))
• 121
Exploring virtualization tools with a new virtualization provisioning method to test dynamic grid environments for ALICE grid jobs over ARC grid middleware
The Nordic Tier-1 for the LHC is distributed over several, sometimes smaller, computing centers. In order to minimize administration effort, we are interested in running different grid jobs over one common grid middleware. ARC is selected as the internal middleware in the Nordic Tier-1. The AliEn grid middleware, used by ALICE has a different design philosophy than ARC. In order to use most of the AliEn infrastructure and available software deployment methods for running ALICE grid jobs on ARC, we are investigating different possible virtualization technologies. Software deployment is a critical a part in grid computing. Ideally a contributing computing center doesn't need to change their infrastructure or install additional software, except at the grid entry point. One solution to achieve this is a cloud infrastructure that provides (dynamically) virtual machines with the required software environments. It seems that no easy solution exists to run grid jobs on top of those cloud providers. They are missing tools that submit a job into a virtual machine, that may be created dynamically and collect the results afterwards. Therefore a specialized, virtualization provisioning method was developed. This method could be developed further into a system that is used by smaller sites, which don't want to run a more general and more complex private cloud solution. The new provisioning system is used to test different combinations of virtualization backends and virtual machines. The CernVM project is developing a virtual machine that can provide a common analysis environment for all LHC experiments. One of our interests is to investigate the use of CernVM as a base setup for a dynamical grid environment capable of running grid jobs. For this, performance comparisons between different virtualization technologies have been conducted. CernVM needs an existing virtualization infrastructure, which is not always existing or wanted at some computing sites. To increase the possible application of dynamical grid environments to those sites, we describe several possibilities that are less invasive and have less specific Linux distribution requirements, at the cost of lower performance. Different tools like user-mode Linux (UML), micro Linux distributions, a new software packaging project by Stanford university (CDE) and CernVM are under investigation for their invasiveness, distribution requirements and performance. Comparisons between the different methods with solutions that are closer to the hardware will be presented.
Speaker: Boris Wagner (University of Bergen (NO) for the ALICE Collaboration)
• 122
Speakers: Jakub Cerkala (Technical University of Košice), Slávka Jadlovská (Department of Cybernetics and Artificial Intelligence, Faculty of Electrical Engineering and Informatics, Technical University of Košice)
• 124
FIFE-Jobsub: A Grid Submission System for Intensity Frontier Experiments at Fermilab
The Fermilab Intensity Frontier Experiments use an integrated submission system known as FIFE-jobsub, part of the FIFE (Fabric for Frontier Experiments) initiative, to submit batch jobs to the Open Science Grid. FIFE-jobsub eases the burden on experimenters by integrating data transfer and site selection details in an easy to use and well documented format. FIFE-jobsub automates tedious details of maintaining grid proxies for the lifetime of the grid job. Data transfer is handled using the ifdh (Intensity Frontier Data Handling) tool suite, which facilitates selecting the appropriate data transfer method from many possibilities while protecting shared resources from overload. Chaining of job dependencies into Directed Acyclic Graphs (Condor DAGS) is well supported and made easier through the use of input flags and parameters. We will present a talk describing these features, their implementation, and the benefits to users of this system.
Speaker: Dennis Box (F)
• 125
FPGA-based 10-Gbit Ethernet Data Acquisition Interface for the Upgraded Electronics of the ATLAS Liquid Argon Calorimeters
Speaker: Johannes Philipp Grohs (Technische Universitaet Dresden (DE))
• 126
FwWebViewPlus: integration of web technologies into WinCC-OA based Human-Machine Interfaces at CERN.
Rapid growth of popularity of web applications gives rise to a plethora of reusable graphical components, such as Google Chart Tools or jQuery Sparklines, implemented in JavaScript and running inside a web browser. In the paper we describe the tool that allows for seamless integration of web-based widgets into WinCC Open Architecture, the SCADA system used commonly at CERN to build complex Human-Machine Interfaces. Reuse of widely available widget libraries and pushing the development efforts to a higher abstraction layer based on scripting language allow for significant reduction in maintenance of the code in multi-platform environment, when compared to currently used C++ visualization plugins. Adequately designed interfaces allow for rapid integration of new web widgets into WinCC-OA. At the same time the mechanisms familiar to WinCC-OA HMI developers are preserved, making the use of new widgets "native". Perspectives for further integration between the realms of WinCC-OA/CTRL and Web/JavaScript development are also discussed.
Speaker: Dr Piotr Golonka (CERN)
• 127
Geant4 application in a web browser
Geant4 application in a web browser   Geant4 is a toolkit for the simulation of the passage of particles through matter. The Geant4 visualization system supports many drivers including OpenGL, OpenInventor, HepRep, DAWN, VRML, RayTracer, gMocren and ASCIITree, with diverse and complementary functionalities.   Web applications have an increasing role in our work, and thanks to emerging frameworks such as Wt [1], we are now able to build a web application on top of a C++ application without rewriting all the code. Because the Geant4 toolkit's visualization and user interface modules are well decoupled from the rest of Geant4, it is straightforward to adapt these modules to render in a web application instead of a computer's native window manager. The API of the Wt framework closely matches that of Qt[3] so we can benefit from our experience in developing the Geant4 Qt driver.  Rendering is through the WebGL[2] framework. In this presentation, we will show how we ported the Geant4 interface to a Web application and how, with minimal effort, other Geant4 users can replicated this process to share their own Geant4 applications in a web browser. [1] http://www.webtoolkit.eu [2] http://www.khronos.org/webgl/ [3] http://geant4.web.cern.ch/geant4/UserDocumentation/UsersGuides/ForApplicationDeveloper/html/ch08s03.html#sect.VisDrv.Qt
Speaker: Laurent Garnier (LAL-IN2P3-CNRS)
• 128
Geant4 Based Simulations for Novel Neutron Detector Development
The construction of the European Spallation Source ESS AB, which will become the worlds most powerful source of cold and thermal neutrons (meV scale), is about to begin in Lund, Sweden, breaking ground in 2014 and coming online towards the end of the decade. Currently 22 neutron-scattering instruments are planned as the baseline suite at the facility, and a crucial part of each such beam-line will be the detector at which neutrons are detected after undergoing scattering in a given sample under study. Historically, the technological choices for neutron detection at thermal energies have been Helium-3 based, a gas which in recent years has become unavailable for all but the smallest of detectors, due to the effect of a rapidly dwindling worldwide supply at the same time the demand is increasing heavily. This makes novel neutron detectors a critical technology for ESS to develop, and also neutron detection itself presently is a hot topic to a range of disciplines and industrial applications. Thus, an extensive international R&D programme is currently underway at ESS and in European partner institutes, and worldwide (under the auspices of the International Collaboration for the Development of Neutron Detectors, icnd.org) in order to develop efficient and cost-effective detectors based on alternative isotopes such as Boron-10 based thin-film detectors or Lithium-6 doped scintillators. In this contribution we present the Geant4-based python/C++ simulation and coding framework, which has been developed and used within the ESS Detector Group and in collaboration with FRM-II, in order to aid in these R&D efforts. We show specific examples of results from investigations of specific proposed detector designs, and discuss the extensions to Geant4 which have been implemented in order to include the (at this energy scale) very significant effects of low-energy phenomena such as coherent scattering (Bragg diffraction) in the polycrystalline support materials of the detector. We also present a custom object oriented output file format with meta-data, GRIFF ("Geant4 Results In Friendly Format"), developed in order to facilitate a faster turn-around time when analysing simulation results by enabling high-level whole-event analysis in addition to the usual benefits of a persistified output format (such as multi-processing and fast re-analysis of the same data). Whilst these simulations have been implemented specifically for neutron detectors, it has potential for wider applications in neutron scattering, and in other disciplines.
Speaker: Dr Thomas Kittelmann (European Spallation Source ESS AB)
• 129
Geant4 Electromagnetic Physics for LHC Upgrade
Electromagnetic physics sub-package of the Geant4 Monte Carlo toolkit is an important component of LHC experiment simulation and other Geant4 applications. In this work we present recent progress in Geant4 electromagnetic physics modeling, with an emphasis on the new refinements for the processes of multiple and single scattering, ionisation, high energy muon interactions, and gamma induced processes. These developments affect the results of ongoing analysis of LHC data, in particular, electromagnetic shower shape parameters used for analysis of H->gg and Z->ee decays. The LHC upgrade to future 14 TeV run will bring new requirements regarding the quality of electromagnetic physics simulation: energy, particle multiplicity, and statistics will be increased. To address new requirements high energy electromagnetic models and cross sections are improved. Geant4 testing suite for electromagnetic physics is extended and new validation results will be presented. An evolution of CPU performance and developments for Geant4 multi-threading connected with Geant4 electromagnetic physics sub-packages will also be discussed.
• 130
Geo-localization in CERN’s underground facilities
CERN has recently renewed its obsolete VHF firemen’s radio network and replaced it by a digital one based on TETRA technology. TETRA already integrates an outdoor GPS localization system, but it appeared essential to look for a solution to also locate TETRA users in CERN’s underground facilities. The system which answers this problematic and which has demonstrated a good resistance to radiation effects, is based on autonomous beacons placed in strategic locations and broadcasting specific identification numbers. The radios are able to decode these identification numbers and transmit this information through the TETRA network to the fire brigade Control Center. An application dedicated to the indoor localization is then able to locate the TETRA terminal on a map.
Speaker: Aurelie Pascal (CERN)
• 131
GLUE 2 deployment: Ensuring quality in the EGI/WLCG information system
The GLUE 2 information schema is now fully supported in the production EGI/WLCG information system. However, to make the schema usable and allow clients to rely on the information it is important that the meaning of the published information is clearly defined, and that information providers and site configurations are validated to ensure as far as possible that what they publish is correct. In this paper we describe the definition of a detailed schema usage profile, the implementation of a software tool to validate published information according to the profile and the use of the tool in the production Grid, and also summarise the overall state of GLUE 2 deployment.
Speaker: Oliver Keeble (CERN)
• 132
gluster file system optimization and deployment at IHEP
Gluster file system adopts no metadata architecture, which theoretically eliminates both a central point of failure and a performance bottleneck of metadata server. Firstly, this talk will introduce gluster compared to lustre or hadoop. However, its some mechanisms are not so good in current version. For example, it has to read the extend attributes of all bricks to locate one file. And it is slow to list files in one directory when there are too many bricks or brick servers are busy. Some other functions, such as expand or shrink volume, file distribution, replication policy and so on, performs not so well in large scale storage system. This talk will analyze the advantages and disadvantages of gluster file system in high performance computing system. To solve these problems, we proposed some new methods, optimized or developed some modules, including modifying elastic hash algorithm implementation, introducing a new index module, designing a new replication layer, and on so. This talk will introduce these methods or functions. We already deployed a gluster file system in production data analysis environment. The talk finally describes the deployment scenario and some lessons we can learn from it.
Speaker: Dr Yaodong CHENG (Institute of High Energy Physics,Chinese Academy of Sciences)
• 133
go-hist: a multi-threaded Go package for histogramming
Current HENP libraries and frameworks were written before multicore systems became widely deployed and used. From this environment, a 'single-thread' processing model naturally emerged but the implicit assumptions it encouraged are greatly impairing our abilities to scale in a multicore/manycore world. Thanks to C++11, C++ is finally slowly catching up with regard to concurrency constructs, at the price of further complicating the language and its standard library. Instead, the approach of the Go language is to provide simple concurrency enabling building blocks, readily integrated into the language, in the hope that these goroutines and channels will help to design, write and maintain concurrent applications. To further investigate whether Go is a suitable C++ and python replacement language for HENP, we developed go-hist, a multi-threaded Go package for histogramming. We will first present the overall design of the go-hist package and the various building blocks (goroutines, channels and wait-groups) provided by the Go language which are then leveraged by this library. A special emphasis will be put on the current SIMD support available in Go and thus how vectorization can be leveraged in histogramming code: a cornerstone for automatic performance scalability on the ever-wider-registers architectures the future has in store. Then, I/O considerations, such as read/write performances, ease of serialization and disk sizes will be discussed, for each of the currently implemented backends (protobuf, gob and json.) Finally, comparisons with ROOT, Java ROOT and inlib/exlib (an AIDA implementation in C++) performances will be presented.
Speaker: Dr Sebastien Binet (IN2P3/LAL)
• 134
GPU Enhancement of the High Level Trigger to extend the Physics Reach at the LHC
Significant new challenges are continuously confronting the High Energy Physics (HEP) experiments in particular the Large Hadron Collider (LHC) at CERN who does not only drive forward theoretical, experimental and detector physics but also pushes to limits computing. LHC delivers proton-proton collisions to the detectors at a rate of 40 MHz. This rate must be significantly reduced to comply with the performance limitations of the mass storage hardware, and the capabilities of the computing resources to process the collected data in a timely fashion for physics analysis. At the same time, the physics signals of interest must be retained with high efficiency. The quest for rare new physics phenomena at the LHC and the flexibility of the HLT allows us to propose a GPU enhancement of the conventional computer farm that provides faster and more efficient events selection at the trigger level.The proposed enhancement is made possible due to rising hybrid systems consisting of processors with multiple cores integrated with Graphics processing units (GPU) as a modified form of stream processor A new tracking algorithm will be executed on the hybrid computer farm that will permit optimal use of the tracker information to reconstruct not only all the prompt tracks but also tracks not originating from the interaction point. One of its benefit, It will allow for the first time to reconstruct tracks originating from very long lived particles in the tracker system in the trigger system, hence extending the search for new physics at the LHC. Preliminary results will be presented comparing the algorithm performance between Nvidia K20 and Intel xeon-phi chip.
Speaker: Halyo Valerie
• 135
GPU for Real Time processing in HEP trigger systems
We describe a pilot project for the use of GPUs (Graphics processing units) in online triggering applications for high energy physics experiments. Two major trends can be identified in the development of trigger and DAQ systems for particle physics experiments: the massive use of general-purpose commodity systems such as commercial multicore PC farms for data acquisition, and the reduction of trigger levels implemented in hardware, towards a pure software selection system (trigger-less). The very innovative approach presented here aims at exploiting the parallel computing power of commercial GPUs to perform fast computations in software both in early trigger stages and in high level triggers. General-purpose computing on GPUs is emerging as a new paradigm in several fields of science, although so far applications have been tailored to the specific strengths of such devices as accelerator in offline computation. With the steady reduction of GPU latencies, and the increase in link and memory throughputs, the use of such devices for real-time applications in high-energy physics data acquisition and trigger systems is becoming ripe. We will discuss in details the use of online parallel computing on GPU for synchronous low level trigger with fixed latency. In particular we will show the preliminary results on a first field test in the CERN NA62 experiment. The use of GPUs in high level triggers will be also considered, the CERN ATLAS experiment (and in particular the muon trigger) will be taken as a study case of possible applications.
Speaker: Roberto Ammendola (INFN)
• 136
GPU Implementation of Bayesian Neural Networks in SUSY Studies
The search for new physics has typically been guided by theoretical models with relatively few parameters. However, recently, more general models, such as the 19-parameter phenomenological minimal supersymmetric standard model (pMSSM), have been used to interpret data at the Large Hadron Collider. Unfortunately, due to the complexity of the calculations, the predictions of these models are available at a discrete set of parameter points, which makes the use of analysis techniques that require smooth maps between the parameters and a given prediction problematic. It would be useful, therefore, to have a computationally routine way to construct such mappings. We propose to construct the mappings using Bayesian neural networks (BNN). Bayesian neural networks have been used in a few high-profile analyses in high energy physics for both classification and functional approximation. The main limitation to their widespread use is the time required to construct these functions. In this talk, we describe an efficient Graphical Processing Unit (GPU) implementation of the construction of BNNs using the Hybrid Markov-Chain Monte Carlo (MCMC) method. We describe our implementation of the MCMC algorithm on the GPU, including the speedups we have achieved so far and illustrate the effectiveness of our implementation by mapping the pMSSM parameter space to some of its key predictions.
Speaker: Michelle Perry (Florida State University)
• 137
Grid Accounting Service: State and Future Development
During the last decade, large-scale federated distributed infrastructures have continually developed and expanded. One of the crucial components of a cyber-infrastructure is an accounting service that collects data related to resource utilization and identity of users using resources. The accounting service is important for verifying pledged resource allocation per particular groups and users, providing reports for funding agencies and resource providers, and understanding hardware provisioning requirements. It can also be used for end-to-end troubleshooting as well as billing purposes. In this work we describe Gratia, a federated accounting service jointly developed at Fermilab and University of Nebraska Holland Computing Center (HCC). It has been used in production by the Open Science Grid, Fermilab, HCC, and several other institutions for several years. The current development activities include Virtual Machines provisioning information, XSEDE allocation usage accounting, and Campus Grids resource utilization. We also identify the directions of future work: improvement and expansion of Cloud accounting, persistent and elastic storage space allocation, and the incorporation of WAN and LAN network metric.
Speaker: Derek John Weitzel (University of Nebraska (US))
• 138
Grid Site Testing for ATLAS with HammerCloud
With the exponential growth of LHC (Large Hadron Collider) data in 2012, distributed computing has become the established way to analyze collider data. The ATLAS grid infrastructure includes more than 130 sites worldwide, ranging from large national computing centers to smaller university clusters. HammerCloud was previously introduced with the goals of enabling VO- and site-administrators to run validation tests of the site and software infrastructure in an automated or on-demand manner. The HammerCloud infrastructure has been constantly improved to support the addition of new test workflows. These new workflows comprise e.g. tests of the ATLAS nightly build system, ATLAS MC production system, XRootD federation FAX and new site stress test workflows. We report on the development, optimization and results of the various components in the HammerCloud framework.
Speaker: Johannes Elmsheuser (Ludwig-Maximilians-Univ. Muenchen (DE))
• 139
Grids, Virtualization and Clouds at Fermilab
Fermilab supports a scientific program that includes experiments and scientists located across the globe. To better serve this community, in 2004, the (then) Computing Division undertook the strategy of placing all of the High Throughput Computing (HTC) resources in a Campus Grid known as FermiGrid, supported by common shared services. In 2007, the FermiGrid Services group deployed a service infrastructure that utilized Xen virtualization, LVS network routing and MySQL circular replication to deliver highly available services that offered significant performance, reliability and serviceability improvements. This deployment was further enhanced through the deployment of an distributed redundant network core architecture and the physical distribution of the systems that host the virtual machines across multiple buildings on the Fermilab Campus. In 2010, building on the experience pioneered by FermiGrid in delivering production services in a virtual infrastructure, the Computing Sector commissioned the FermiCloud, GPCF and Virtual Services projects to serve as platforms for support of scientific computing (FermiCloud & GPCF) and core computing (Virtual Services). This work will present the evolution of the Fermilab Campus Grid, Virtualization and Cloud Computing infrastructure together with plans for the future.
Speaker: Dr Gabriele Garzoglio (FERMI NATIONAL ACCELERATOR LABORATORY)
• 140
Handling Worldwide LHC Computing Grid Critical Service Incidents : The infrastructure and experience behind nearly 5 years of GGUS ALARMs
In the Wordwide LHC Computing Grid (WLCG) project the Tier centres are of paramount importance for storing and accessing experiment data and for running the batch jobs necessary for experiment production activities. Although Tier2 sites provide a significant fraction of the resources a non-availability of resources at the Tier0 or the Tier1s can seriously harm not only WLCG Operations but also the experiments' workflow and the storage of LHC data which are very expensive to reproduce. This is why availability requirements for these sites are high and committed in the WLCG Memorandum of Understanding (MoU). In this talk we describe the workflow of GGUS ALARMs, the only 24/7 mechanism available to LHC experiment experts for reporting to the Tier0 or the Tier1s problems with their Critical Services. Conclusions and experience gained from the detailed drills performed in each such ALARM for the last 4 years will be explained and the shift with time of Type of Problems met. The physical infrastructure put in place to achieve GGUS 24/7 availability will be summarised.
Speaker: Maria Dimou (CERN)
• 141
Hangout With CERN - Reaching the Public with the Collaborative Tools of Social Media
On July 4, 2012, particle physics became a celebrity. Around 1,000,000,000 people (yes, 1 billion) saw rebroadcasts of two technical presentations announcing discovery of a new boson. The occasion was a joint seminar of the CMS and ATLAS collaborations, and the target audience were members of those collaborations plus interested experts in the field of particle physics. Yet, the world ate it up like a sporting event. Roughly two days later, in a parallel session of ICHEP in Melbourne, Australia, a group of physicists decided to explain the significance of this discovery to the public. They started up a tool called “Hangout”, part of the new Google+ social media platform, to converse directly with the public via videoconference and webcast. The demand to join this Hangout overloaded the server several times. In the end, a compromise involving Q&A via comments was set up, and the conversation was underway. I present a new project born from this experience, called Hangout With CERN, and discuss its success in creating an effective conversational channel between the public and high-energy physicists. I review earlier efforts by both CMS and ATLAS contributing to this development, and then describe the current programme, involving nearly all aspects of CERN, and some topics that go well beyond that. I conclude by discussing the potential of the program both to improve our accountability to the public and to train our community for public communication.
Speaker: Steven Goldfarb (University of Michigan (US))
• 142
Helix Nebula and CERN: A Symbiotic Approach to Exploiting Commercial Clouds
The recent paradigm shift toward cloud computing in IT, and general interest in "Big Data" in particular, have demonstrated that the computing requirements of HEP are no longer globally unique. Indeed, the CERN IT department and LHC experiments have already made significant R&D investments in delivering and exploiting cloud computing resources. While a number of technical evaluations of interesting commercial offerings from global IT enterprises have been performed by various physics labs, further technical, security, sociological, and legal issues need to be address before their large-scale adoption by the research community can be envisaged. Helix Nebula - the Science Cloud is an initiative that explores these questions by joining the forces of three European research institutes (CERN, ESA and EMBL) with leading European commercial IT enterprises. The goals of Helix Nebula are to establish a cloud platform federating multiple commercial cloud providers, along with new business models, which can sustain the cloud marketplace for years to come. This contribution will summarize the participation of CERN in Helix Nebula. We will explain CERN's flagship use-case and the model used to integrate several cloud providers with an LHC experiment's workload management system. During the first proof of concept, this project contributed over 40.000 CPU-days of Monte Carlo production throughput to the ATLAS experiment with marginal manpower required. CERN's experience, together with that of ESA and EMBL, is providing a great insight into the cloud computing industry and highlighted several challenges that are being tackled in order to ease the export of the scientific workloads to the cloud environments.
Speaker: Ramon Medrano Llamas (CERN)
• 143
Hepdoop
“Big Data” is no longer merely a buzzword, but is business-as-usual in the private sector. High Energy Particle Physics is often cited as the archetypal Big Data use case, however it currently shares very little of the toolkit used in the private sector or other scientific communities. We present the initial phase of a programme of work designed to bridge this technology divide by both performing real HEP analysis workflows using predominately industry “Big Data” tools, formats and techniques, and the reverse: to perform real industry tasks with HEP tools. In doing so it will improve interoperation of those tools, reveal strengths and weakness and enable efficiencies within both communities. The first phase of this work performs key elements of an LHC Higgs Analysis using very common Big Data tools. These elements include data serialization, filtering and data mining. They are performed with a range of tools chosen not just for performance but also for ease-of-use, maturity and size of user community. This includes technologies such as Protocol Buffers, Hadoop and Python Scikit and for each element we make comparisons with the same analysis performed using current HEP tools such as ROOT, Proof and TMVA.
Speaker: Wahid Bhimji (University of Edinburgh (GB))
• 144
High-Level Trigger Performance for Calorimeter based algorithms during LHC Run 1 data taking period
The ATLAS detector operated during the three years of the run 1 of the Large Hadron Collider collecting information on a large number of proton-proton events. One the most important results obtained so far is the discovery of one Higgs boson. More precise measurements of this particle must be performed as well as there are other very important physics topics still to be explored. One of the key components of the ATLAS detector is its trigger system. It is composed of three levels : one (called Level 1 - L1) built on custom hardware and the two others based on software algorithms - called Level 2 (L2) and Event Filter (EF) – altogether referred to as the ATLAS High Level Trigger. The ATLAS trigger is responsible for reducing almost 20 million of collisions per second produced by the accelerator to less than 1000. The L2 operates only in the regions tagged by the first hardware level as containing possible interesting physics while the EF operates in the full detector, normally using offline-like algorithms to reach a final decision about recording the event. Amongst the ATLAS subdetectors, there is a complex set of calorimeter specialized to detect and measure the energy of electrons, photons, taus, jets and even measure global event missing transverse energy. The present work describes the performance of the ATLAS High-Level Calorimeter Trigger. Algorithms for detecting electrons and taus were able to reconstruct clusters of calorimeter cells, measure their energy and shower shape variables for particle classification with quite high efficiency. Since the beginning of the operation only minor modification on cut values were necessary given the wide range of instantaneous luminosity explored (over 5 order of magnitude). Another class of algorithms studies jets and can detect and estimate the energy of these using different cone sizes and jet finding techniques. Finally, an algorithm only for the Event Filter was design to measure with high precision the total transverse energy using only cells that are above their own noise level. As the luminosity increased, it was fundamental to create an option for reducing the rate of events from the L1 into for EF missing E_{T} chains. So, the ATLAS L2 trigger operating paradigm of only using regions around the L1 based had to be broken for the special case of the missing E_{T}. The paper describes the parallel chain built in the ATLAS data acquisition system to accomplish this task. Also, it seemed increasingly important to work on a similar topic for the L2 jets. The usage of full detector information so that jets could be found in full scans of the detector - not limited by the L2 regions - is quite relevant for the efficiency of multiple jet algorithms. The option taken will also be described in the paper. Finally, we will describe the future plans of operation for the LHC run 2 data taking period.
Speakers: Alex Mann (Ludwig-Maximilians-Univ. Muenchen (DE)), Alexander Mann (Ludwig-Maximilians-Universität)
• 145
HS06 benchmark values for an ARM based server
We benchmarked an ARM Cortex A9 based server system with a four-core CPU running at 1.1 GHz. The system used Ubuntu 12.04 as operating system and the hepspec 2006 (HS06) benchmarking suite was compiled natively with gcc-4.4 on the system. The benchmark was run for various settings of the relevant gcc compiler options. We did not find significant influence from the compiler options on the benchmark result. The final HS06 benchmark result is 10.4.
Speaker: Stefan Kluth (Max-Planck-Institut fuer Physik (Werner-Heisenberg-Institut) (D)
• 146
ILCDIRAC, a DIRAC extension for the Linear Collider community
ILCDIRAC was initially developed in the context of the CLIC Conceptual Design Report (CDR), published in 2012-2013. It provides a convenient interface for the mass production of the simulation events needed for the physics performance studies of the two detectors concepts considered, ILD and SID. It was since used in the ILC Detailed Baseline Detector (DBD) studies of the SID detector concept, and is currently moving towards a complete unification of the production systems for the whole Linear Collider community. The CALICE collaboration uses part of the ILC software tools and can transparently use ILCDIRAC for its activities. ILCDIRAC extends the core functionality of DIRAC for multiple aspects: There are currently 14 applications supported, which have very different inherent interfaces. ILCDIRAC simplifies this situation for user convenience and maximum flexibility. For that purpose the relationship between application and job definition was reviewed and they are completely separated. The base application and job classes are designed in a generic way, allowing for simple extension. This design is independent of the Linear Collider use case and can be applied in other contexts. Another specificity of ILCDIRAC is the management of the so called Overlay. The Linear Collider experiments will be subject to intense machine induced backgrounds and physics backgrounds like gamma gamma -> hadrons. For realistic studies these backgrounds need to be included in the simulation. Instead of repeating the simulation of these background events, they are overlaid to the signal events during digitisation. The overlay files are randomly selected from a pool of available files and supplied to the job. The constraint in that system is the fact that many events are needed as input (up to 200 background events per signal event), so the overlay files represent a very large sample per job to be obtained from the Storage Elements. To avoid data access issues, the system prevents too many concurrent jobs to query the Storage Elements at the same time, avoiding problems due to Storage Element overload. The design of the software management of ILCDIRAC ensures maximum availability by using the shared area when/where possible. In case the shared area has no "Role" protection, typical for OSG sites, a locking procedure is implemented to ensure that no job will overwrite an existing software being installed. The procedure also validates the input software tar balls and their content via md5 check sum verifications. These ILCDIRAC specific aspects rely heavily on the DIRAC features, in particular the File Catalog, which was mostly developed for ILCDIRAC. In addition, the Transformation System is used to produce all the events, in particular for the CLIC CDR and the SID DBD. It successfully generated, simulated, reconstructed more than 100 million events in 2.5 years, not counting for the user activities, and the File Catalog contains nearly 7 million files corresponding to approximately 1 PB.
Speakers: Andre Sailer (CERN), Christian Grefe (CERN), Stephane Guillaume Poss (Centre National de la Recherche Scientifique (FR))
• 147
Implementation of the twisted mass fermion operator on accelerators
Lattice Quantum Chromodynamics (LQCD) simulations are critical for understanding the validity of the Standard Model and the results of the High-Energy and Nuclear Physics experiments. Major improvements in the calculation and prediction of physical observables, such as nucleon form factors or flavor singlet meson mass, require large amounts of computer resources, of the order of hundreds of Tflop/s of sustained performance. For the first part of our study, we extended the QUDA library, an open source library for performing calculations in LQCD on NVIDIA GPUs, to include kernels for the non-degenerate twisted mass fermion operator. Next, we implemented the operator on the Intel MIC architecture. A detailed performance analysis for both cases is provided.
Speaker: Dr Alexei Strelchenko (FNAL)
• 148
Implementing long-term data preservation and open access in CMS
Implementation of the CMS policy on long-term data preservation, re-use and open access has started. Current practices in providing data additional to published papers and distributing simplified data-samples for outreach are promoted and consolidated. The first measures have been taken for the analysis and data preservation for the internal use of the collaboration and for the open access to part of the data. Two complementary approaches are followed. First, a virtual machine environment, which will pack all ingredients needed to compile and run a software release with which the legacy data was reconstructed. Second, a validation framework, maintaining the capability not only to read the old raw data, but also to reconstruct them with more recent releases to guarantee long-term reusability of the legacy data.
Speaker: Kati Lassila-Perini (Helsinki Institute of Physics (FI))
• 149
INFN Pisa scientific computation environment (GRID HPC and Interactive analysis)
The INFN-Pisa Tier2 infrastructure is described, optimized not only for GRID CPU and Storage access, but also for a more interactive use of the resources in order to provide good solutions for the final data analysis step. The Data Center, equipped with about 5000 production cores, permits the use of modern analysis techniques realized via advanced statistical tools (like RooFit and RooStat) implemented in multi core systems. In particular a POSIX file storage access integrated with standard srm access is provided. Therefore the unified storage infrastructure is described, based on GPFS and Xrootd, used both for SRM data repository and interactive POSIX access. Such a common infrastructure allows a transparent access to the Tier2 data to the users for their interactive analysis. The organization of a specialized many cores CPU facility devoted to interactive analysis is also described along with the login mechanism integrated with the INFN-AAI (National INFN Infrastructure) to extend the site access and use to a geographical distributed community. Such infrastructure is used also for a national computing facility in use to the INFN theoretical community, it enables a synergic use of computing and storage resources. Our Center initially developed for the HEP community is now growing and includes also HPC resources fully integrated. In recent years has been installed and managed a cluster facility (1000 cores, parallel use via InfiniBand connections) and we are now updating this facility that will provide resources for all the intermediate level HPC computing needs of the theoretical national INFN community.
Speaker: Enrico Mazzoni (INFN-Pisa)
• 150
Installation and configuration of an SDN test-bed made of physical switches and virtual switches managed by an Open Flow controller.
The computing models of HEP experiments, starting from the LHC ones, are facing an evolution with the relaxation of the data locality paradigm: the possibility of a job accessing data files over the WAN is becoming more and more common. One of the key factors for the success of this change is the ability to use the network in the most efficient way: in the best scenario, the network should be capable to change its behavior on the base of a per flow analysis. The SDN (Software Defined Networks) are a promising technology to address this challenging requirement and OpenFlow is a candidate protocol to implement it. At CNAF, we have installed a Software Defined Networks test-bed functional to build a concrete layout where testing OpenFlow protocol and the interoperability between devices of different vendors. The objective of this activity is to build a network with a control plane driven by a software layer in charge of defining the "route" for a specific flow, based on conditions verified inside a datacenter network, in order to solve routing problems not addressable with standard networking protocols.
Speakers: Donato De Girolamo (INFN CNAF), Mr Lorenzo Chiarelli (INFN CNAF), Mr Stefano Zani (INFN CNAF)
• 151
Integrating the Network into LHC Experiments: Update on the ANSE (Advanced Network Services for Experiments) Project
The LHC experiments have always depended upon a ubiquitous, highly-performing network infrastructure to enable their global scientific efforts. While the experiments were developing their software and physical infrastructures, parallel development work was occurring in the networking communities responsible for interconnecting LHC sites. During the LHC's Long Shutdown \#1 (LS1) we have an opportunity to incorporate some of these network developments into the LHC experiment's software. The ANSE (Advance Network Services for Experiments) Project, an NSF CC-NIE funded effort, is targeting the creation, testing and deployment of a set of software tools (network services and an associated API) suitable for use by the LHC experiments. This library will explicitly enable obtaining both knowledge of the networks and provide mechanisms for interacting with those networks. The project will leverage existing infrastructure from DYNES\cite{dynes}, AutoBahn\cite{autobahn}, perfSONAR-PS\cite{psps}, OpenFlow\cite{OpenFlow}, MonALISA\cite{MonaLISA} and Software Defined Networking\cite{SDN} as well as experiment-specific applications to enable these capabilities. We will report on the progress we have made to date, present our initial architecture and show some examples of the kinds of functionality we are developing in the context of the ATLAS and CMS experiments. @misc{dynes, title = "{{MRI}-{R}2 {C}onsortium: {D}evelopment of {D}ynamic {N}etwork {S}ystem ({DYNES})}", key = "dynes", howpublished = "\url{http://www.internet2.edu/ion/dynes.html}" } @misc{autobahn, title = "{{GEANT}2 Auto{BAHN}}", key = "autobahn", howpublished = "\url{http://www.geant2.net/server/show/ConWebDoc.2544}" } @misc{fdt, title = "{Fast Data Transfer (FDT)}", key = "fdt", howpublished = "\url{http://monalisa.cern.ch/FDT}" } @misc{psps, title = "{perf{SONAR}-PS}", key = "psps", howpublished = "\url{http://psps.perfsonar.net}" } @misc{SDN, title = "{{S}oftware {D}efined {N}etworking}", key = "SDN", howpublished = "\url{http://en.wikipedia.org/wiki/Software-defined_networking}" } @misc{OpenFlow, title = "{OpenFlow Layer 2 Communications Protocol}", key = "OpenFlow", howpublished = "\url{http://en.wikipedia.org/wiki/OpenFlow}" } @misc{MonALISA, title = "{MonALISA: An agent based, dynamic service system to monitor, control and optimize distributed system}", key = "MonALISA", howpublished = "\url{http://dx.doi.org/10.1016/j.bbr.2011.03.031}" }
Speaker: Andrew Malone Melo (Vanderbilt University (US))
• 152
Integration and validation testing for PhEDEx, DBS and DAS with the PhEDEx LifeCycle agent
The ever-increasing amount of data handled by the CMS dataflow and workflow management tools poses new challenges for cross-validation among different systems within CMS experiment at LHC. To approach this problem we developed an integration test suite based on the LifeCycle agent, a tool originally conceived for stress-testing new releases of PhEDEx, the CMS data-placement tool. The LifeCycle agent provides a framework for customising the test workflow in arbitrary ways, and can scale to levels of activity well beyond those seen in normal running. This means we can run realistic performance tests at scales not likely to be seen by the experiment for some years, or with custom topologies to examine particular situations that may cause concern some time in the future. The LifeCycle agent has recently been enhanced to become a general purpose integration and validation testing tool for major CMS services (PhEDEx, DBS, DAS). It allows cross-system integration tests of all three components to be performed in controlled environments, without interfering with production services. In this paper we discuss the design and implementation of the LifeCycle agent. We describe how it is used for small-scale debugging and validation tests, and how we extend that to large-scale tests of whole groups of sub-systems. We show how the LifeCycle agent can emulate the action of operators, physicists, or software agents external to the system under test, and how it can be scaled to large and complex systems.
Speaker: Dr Tony Wildish (Princeton University (US))
• 153
Integration of g4tools in Geant4
g4tools, that is originally part of the inlib and exlib packages [1], provides a very light and easy to install set of C++ classes that can be used to perform analysis in a Geant4 batch program. It allows to create and manipulate histograms and ntuples, and write them in supported file formats (ROOT, AIDA XML, CSV and HBOOK). It is integrated in Geant4 through analysis manager classes, thus providing a uniform interface to the g4tools objects and also hiding the differences between the classes for different supported output formats. Moreover, additional features, such as for example histogram activation or support for Geant4 units, are implemented in the analysis classes following users requests. A set of Geant4 user interface commands allows the user to create histograms and set their properties interactively or in Geant4 macros. g4tools was first introduced in the Geant4 9.5 release where its use was demonstrated in one basic example, and it is already used in a majority of the Geant4 examples within the Geant4 9.6 release. In this presentation, we will give an overview and the present status of the integration of g4tools in Geant4 and report on upcoming new features. [1] http://inexlib.lal.in2p3.fr/
Speaker: Ivana Hrivnacova (Universite de Paris-Sud 11 (FR))
• 154
Intergrating configuration workflows with project management system
The complexity of the heterogeneous computing resources, services and recurring infrastructure changes at the GridKa WLCG Tier-1 computing center require a structured approach to configuration management and optimization of interplay between functional components of the whole system. A set of tools deployed at GridKa, including Puppet, Redmine, Foreman, SVN and Icinga, provides the administrative environment giving the possibility to define and develop configuration workflows, reduce the administrative effort and improve sustainable operation of the whole computing center. In this presentation we discuss the developed configuration scenarios implemented at GridKa, which we use for host installation, service deployment, change management procedures, service retirement etc. The integration of Puppet with a project management tool like Redmine provides us with the opportunity to track problem issues, organize tasks and automate these workflows. The interaction between Puppet and Redmine results in automatic updates of the issues related to the executed workflow performed by different system components. The extensive configuration workflows require collaboration and interaction between different departments like network, security, production etc. at GridKa. Redmine plugins developed at GridKa and integrated in its administrative environment provide an effective way of collaboration within the GridKa team. We present the structural overview of the software components, their connections, communication protocols and show a few working examples of the workflows and their automation.
Speakers: Dmitry Nilsen (Karlsruhe Institute of Technology), Dr Pavel Weber (KIT - Karlsruhe Institute of Technology (DE))
• 155
Job Scheduling in Grid Farms
The vast majority of jobs in the Grid are embarrassingly parallel. In particular HEP tasks are divided into atomic jobs without need for communication between them. Jobs are still neither multi-threaded nor multi-core capable. On the other hand, resource requirements reach from CPU-dominated Monte Carlo jobs to network intense analysis jobs. The main objective of any Grid site is to stably operate its Grid farm while achieving a high job slot occupancy, an optimal usage of the computing resources (network, CPU, memory, disk space) and guaranteed shares for the VOs and groups. In order to optimize the utilization of resources, jobs must be distributed intelligently over the slots, CPUs, and hosts. Although the jobs resource requirements cannot be deduced directly, jobs are mapped to POSIX user/group ID based on their VOMS-proxy. The user/group ID allows to distinguish jobs, assuming VOs make use of the VOMS group and role mechanism. The multi-VO Tier-2 site at DESY (DESY-HH) supports ~20 VOs on federated computing resources, using an opportunistic resource usage model. As at many EGI/WLCG sites, the Grid farm is based on the queuing system PBS/TORQUE, which was deployed from the EMI middleware repositories. Initially, the scheduler MAUI was used. It showed severe scalability problems with 4000 job slots as soon as the number of running plus queued jobs approached 10000. Job scheduling became slow or even blocked. In addition, MAUI's many configuration options appeared to be hard to control. To be able to further increase the number of worker nodes as requested by the VOs (to currently 8000 job slots), DESY-HH needed a scalable and performing scheduler, which runs in conjunction with PBS/TORQUE. In the course of studying alternative scheduling models, a home-made scheduler was developed (working title: MySched), which is tailored to then needs of the DESY-HH Grid farm and uses the C-API of PBS/TORQUE. It is based on a simple scheduling model without support for multi-core jobs and job parallelism and is optimized for high job slot occupancy and intelligent distribution of jobs to the worker nodes. Furthermore, it allows for a fine-grained adjustment of limits and parameters on VO and group level. In the contribution to CHEP 2013 we will discuss the impact of a classification of jobs according to their potential resources requirements on scheduling strategies. Subsequently, we will describe our home-made implementation and present operational results.
Speaker: Dr Andreas Gellrich (DESY)
• 156
Keyword Search over Data Service Integration for Accurate Results
Background: The goal of the virtual data service integration is to provide a coherent interface for querying a number of heterogenous data sources (e.g., web services, web forms, proprietary systems, etc.) in cases where accurate results are necessary. This work explores various aspects of its usability. Problem: Querying is usually carried out through a structured query language, such as SQL, which forces the users to learn the language and to get acquainted with data organization (i.e. the schema) thus negatively impacting the system’s usability. Limited access to data instances as well as users’ concern with accurate results of arbitrary queries present additional challenges to traditional approaches (such as query forms, information retrieval, keyword search over relational databases) making them not applicable. Solution: This paper presents a keyword search system which deals with the above discussed problem by operating on available information: the metadata, such as the constraints on allowed values, analysis of user queries, and certain portions of data. Given a keyword query, it proposes a ranked list of structured queries along with the explanations of their meanings. Unlike previous implementations, the system is freely available and makes no assumptions about the input query, while maintaining its ability to leverage the query’s structural patterns - in case they exist. The system is discussed in the context of CMS data discovery service where the simplicity and capabilities of the search interface play a crucial role in the ability of its users to satisfy their information needs.
Speaker: Vidmantas Zemleris (Vilnius University (LT))
• 157
Lessons learned from the ATLAS performance studies of the Iberian Cloud for the first LHC running period
In this contribution we expose the performance of the Iberian (Spain and Portugal) ATLAS cloud during the first LHC running period (March 2010-January 2013) in the framework of the GRID Computing and Data Model. The evolution of the resources for CPU, disk and tape in the Iberian Tier1 and Tier2s is summarized. The data distribution over all ATLAS destinations is shown, focusing in the number of files transferred and the size of the data. The status and distribution of simulation and analysis jobs within the cloud are discussed. The distributed analysis tools used to perform physics analysis are explained as well. Cloud performance in terms of the availability and reliability of its sites is discussed. The effect of the changes in the ATLAS Computing Model on the cloud is analyzed. Finally, the readiness of the Iberian cloud towards the 1st Long Shut Down (LS1) is evaluated and an outline of the foreseen actions to take in the coming years is given. The shutdown will be a good opportunity to improve and evolve the ATLAS Distributed Computing system to prepare for the future challenges of the LHC operation.
Speaker: Victoria Sanchez Martinez (Instituto de Fisica Corpuscular (IFIC) UV-CSIC (ES))
• 158
Leveraging HPC resources for High Energy Physics
High Performance Computing (HPC) provides unprecedented computing power for a diverse range of scientific applications. As of November 2012, over 20 supercomputers deliver petaflop peak performance with the expectation of "exascale" technologies available in the next 5 years. Despite the sizeable computing resources on offer there are a number of technical barriers that limit the use of HPC resources for High Energy Physics applications. HPC facilities have traditionally opted for specialised hardware architectures and favoured tightly coupled parallel MPI-based workloads rather than the high throughput commodity computing model typically used in HEP. However, more recent HPC facilities use x86-based architectures managed by Linux-based operating systems which could potentially allow unmodified HEP software to be run on supercomputers. There is now renewed interest from both the LHC experiments and the HPC community to accommodate data analysis and event simulation production on HPC facilities either from a dedicated resource share or from opportunistic use during low utilisation periods. If existing job scheduling and execution frameworks used by the LHC experiments could be successfully adapted for HPC use it would significantly increase the total amount of computing resources available. A feasibility study into the use of LHC software in an HPC environment will be presented. The HECToR supercomputer in the UK and the SuperMUC supercomputer in Germany will be used as demonstrators. The challenges faced from the perspective of software execution and the interaction with existing LHC distributed computing environments will be highlighted using software typical in data analysis workflows. In particular, we will discuss how tighter restrictions on HPC worker node access could limit the functionality of existing middleware and how this can be potentially adapted for external job submission and more efficient job scheduling.
Speaker: Andrew John Washbrook (University of Edinburgh (GB))
• 159
LHC Grid Computing in Russia- present and future
The review of the distributed grid computing infrastructure for LHC experiments in Russia is given. The emphasis is placed on the Tier-1 site construction at the National Research Centre "Kurchatov Institute" (Moscow) and the Joint Institute for Nuclear Research (Dubna). In accordance with the protocol between CERN, Russia and the Joint Institute for Nuclear Research (JINR) on participation in LCG Project approved in 2003 and Memorandum of Understanding (MoU) on Worldwide LHC Computing Grid (WLCG) signed in October of 2007. Russia and the Joint Institute for Nuclear Research bear responsibility for nine Tier-2 centers. Here and now Russia and JINR computing Tier-2 infrastructure fully satisfies the WLCG Computing Requirements and provides proper support of the LHC experiments' Data Processing and Analysis Tasks. In March of 2011 the proposal to create the LCG Tier1 center as an integral part of the central data handling service of the LHC Experiments in Russia was expressed in the official letter by Minister of Science and Education of Russia to CERN Director General. In 2011 The Federal Target Programme Project: «Creation of the automated system of data processing for experiments at the Large Hadron Collider of Tier-1 level and maintenance of Grid services for distributed analysis of this data» was approved for the period 2011-2013. The Project is aimed at the creation of a Tier-1 computer-based system in Russia and JINR for the processing of experimental data received from LHC and provisioning of Grid services for a subsequent analysis of this data at the distributed centers of the LHC computing Grid. It is shared that the National Research Centre "Kurchatov Institute" (Moscow) is responsible primarily for support of ALICE, ATLAS, and LHC-B experiments while the JINR (Dubna) provides Tier-1 services for the CMS experiment. The master construction plan consists of two phases. The first phase is the construction of prototype in the middle of 2013 and the second one is building of full-scale fully functional Tier-1 which has to be completed in 2014.
Speaker: Eygene Ryabinkin (National Research Centre Kurchatov Institute (RU))
• 160
Long Term Data Preservation for CDF at INFN-CNAF
Long-term preservation of experimental data (intended as both raw and derived formats) is one of the emerging requirements coming from scientific collaborations. Within the High Energy Physics community the Data Preservation in High Energy Physics (DPHEP) group coordinates this effort. CNAF is not only one of the Tier-1s for the LHC experiments, it is also a computing center providing computing and storage resources to many other HEP and non-HEP scientific collaborations, including the CDF experiment. After the end of data taking in 2011, CDF is now facing the challenge to both preserve the large amount of data produced during several years of data taking and to retain the ability to access and reuse it in the future. CNAF is heavily involved in the CDF Data Preservation activities, in collaboration with the FNAL computing sector. At the moment about 5 PB of data (raw data and analysis-level “ntuples”) are being copied from FNAL to the CNAF tape library and the framework to subsequently access the data is being set up. In parallel to the data access system, a data analysis framework is being developed which allows to run the complete CDF analysis chain in the long term future, from raw data reprocessing to analysis-level “ntuple” production. In this contribution we illustrate the technical solutions we put in place to address the issues encountered as we proceeded in this activity.
Speaker: Luca dell'Agnello (INFN-CNAF)
• 161
Managing and throttling federated xroot across WLCG Tier 1s
WLCG is moving towards greater use of xrootd. While this will in general optimise resource usage on the grid, it can create load problems at sites when storage elements are unavailable. We present some possible methods of mitigating these problems and the results from experiments at STFC
Speaker: Shaun De Witt (STFC - Science & Technology Facilities Council (GB))
• 162
Many-core on the Grid: From Exploration to Production
A number of High Energy Physics experiments have successfully run feasibility studies to demonstrate that many-core devices such as GPGPUs can be used to accelerate algorithms for trigger systems and data analysis. After this exploration phase experiments on the Large Hadron Collider are now investigating how these devices can be incorporated into key areas of their software framework in advance of the significant increase in data volume expected in the next phase of LHC operations. A recent survey performed by the EGI GPGPU Resource Centre indicates that there is increasing community interest for GPUs to be made available on the existing grid infrastructure. However, despite this anticipated usage there is no standard method available to run GPU-based applications in distributed computing environments. Before GPU resources are available on the grid operational issues such as job scheduling and resource discovery will have to be addressed. For example, software developed for many-core devices is often optimised for a particular device and therefore specific architecture details - such as GPU shared memory capacity and thread block size - will be required for job resource matching. The key technical challenges for grid-enabling many-core devices will be discussed. Consideration will be given to how jobs can be scheduled to maximise device occupancy, how they can be advertised using methods familiar to the grid user-community and how their usage can be accurately evaluated for accounting purposes. A selection of GPU devices will be made available at two Tier-2 Grid sites in the UK to demonstrate resource usage. Functionality will be evaluated using software from existing many-core feasibility studies in addition to examples provided by the EPIC project who will use grid-enabled GPUs as a critical part of their workflow.
Speaker: Andrew John Washbrook (University of Edinburgh (GB))
• 163
Matrix Element Method with Graphics Processing Units (GPUs)
The Matrix Element Method has been used with great success in the past several years, notably for the high precision top quark mass determination, and subsequently the single top quark discovery, at the Tevatron. Unfortunately, the Matrix Element method is notoriously CPU intensive due to the complex integration performed over the full phase space of the final state particles arising from high energy interactions. At the Large Hadron Collider (LHC), high luminosities mean much larger numbers of events to analyse than we had at the Tevatron. This makes the Matrix Element method difficult to use, particularly now that computing resources are being utilised already at capacity. We have studied the feasibility of using Graphics Processing Units (GPUs) to reduce the computing time required for the measurement of Higgs decaying to two muons at the LHC. We present the technical approach followed, the speed-up gained, and the prospects for future improvements in the analysis through the use of GPUs.
Speaker: Mr Stephen Lloyd (University of Edinburgh)
• 164
The WLCG uses HEP-SPEC as its benchmark for measuring CPU performance. This provides a consistent and repeatable CPU benchmark to describe experiment requirements, lab commitments and existing resources. However while HEP-SPEC has been customized to represents WLCG applications it is not a perfect measure. The Rutherford Appleton Laboratory (RAL), is the UK Tier 1 site and provides CPU and disk resources for the four largest LHC experiments as well as to numerous other experiments. Recent generations of hardware procurement at RAL have included CPUs with Hyper-Threading. Previous studies have shown that as the number of logical cores being used increases, the measured HEP-SPEC will also increase but by increasingly smaller amounts. The more jobs that are running, the higher the chance that their will be contention on other resources which will cause jobs to slow down. It is therefore not obvious what is the optimal number of jobs to run on a Hyper-Threaded machine. This paper details the work done to maximize job throughput at RAL. Over the course of several months different machine configurations were tested at RAL to see the impact on real job throughput. The results have allowed RAL to maximize job throughput while also accurately reporting the available HEP-SPEC and provided useful information for future procurements.
• 165
MCM : The Evolution of PREP. The CMS tool for Monte-Carlo Request Management.
The analysis of the LHC data at the CMS experiment requires the production of a large number of simulated events. In 2012, CMS has produced over 4 Billion simulated events in about 100 thousands of datasets. Over the past years a tool (PREP) has been developed for managing such a production of thousands of samples. A lot of experience working with this tool has been gained, and conclusions on its limitations have been drawn. For better interfacing with the CMS production infrastructure and data book-keeping system, new database technology (couchDB) has been adopted. More recent server infrastructure technology (cherrypy + java) has been set as the new platform for an evolution of PREP. The operational limitations encountered over the years of usage have been solved in the new system. The aggregation of the production information of samples has been much improved for a better traceability and prioritization of work. This contribution will cover the description of the functionalities of this major evolution of the software for managing samples of simulated events for CMS.
Speaker: Dr Jean-Roch Vlimant (CERN)
• 166
MICE Data Handling on the Grid
Speaker: Dr Janusz Martyniak (Imperial College London)
• 167
MICE Experiment Data Acquisition system
The Muon Ionization Cooling Experiment (MICE) is under development at the Rutherford Appleton Laboratory (UK). The goal of the experiment is to build a section of a cooling channel that can demonstrate the principle of ionization cooling and to verify its performance in a muon beam. The final setup of the experiment will be able to measure a 10% reduction in emittance (transverse phase space size) of the beam with a relative precision of 1%. The Data Acquisition (DAQ) system of the MICE experiment must be able to acquire data for ~600 muons, crossing the experiment during 1 ms long beam spill. To fulfill this requirement, the Front-End Electronics (FEE) must digitize the analog signals in less than 500 ns and store the digitized data in buffer memory. The time before the arrival of the next spill (~1 s) is used to read out the buffers and record the data. The acquisition of the data coming from the detectors is based on VME FEE interfaced to Linux PC processors. The event building, and the DAQ user interface software has been developed from the DATE package, originally developed for the ALICE experiment. The DAQ system communicates with the Control and Monitoring System of the experiment, using the EPICS (Experimental Physics and Industrial Control System) platform. This communication is used to exchange large number of parameters, describing the run conditions and the status of the data taking process. All these parameters are recorded in a Configuration Data Base. The Detector DAQ is strongly dependent on the Trigger System, which is divided into two parts. The first part is responsible for the generation of the so called “Particle Trigger”, which triggers the digitization of the analog signals received from the detectors. The second part of the Trigger System generates the so called "DAQ Trigger". This signal is generated after the end of the spill and causes the readout and storage of the digital data, corresponding to all the particle triggers received during a spill. A new Trigger System for the MICE experiment, based on programmable FPGA logic is now under development and tests.
Speaker: Yordan Ivanov Karadzhov (Universite de Geneve (CH))
• 168
Migration of the CERN IT Data Center Support System to ServiceNow
The large potential and flexibility of the ServiceNow infrastructure based on "best practices" methods is allowing the migration of some of the ticketing systems traditionally used for the tracing of the servers and services available at the CERN IT Computer Center. This migration enables a standardization and globalization of the ticketing and control systems implementing a generic system extensible to other departments and users. One of the activities of the Service Management project together with the IT-CF group has been the migration of the ITCM structure based in Remedy to ServiceNow within the context of one of the ITIL processes called Event Management. From the technical point of view, the adaptation of ServiceNow to the well known ITCM Remedy based structure has been instrumental to ensure a controlled and smooth migration. In addition, the versatility of ServiceNow has allowed the implementation of generic facilities based in best practices processes to extend the potential provided by ITCM including Service Level Management facilities for the tracking of external contractors , specific forms, etc. From the operational point of view a complete reporting infrastructure in ServiceNow has been created with a particular emphasis on the end users training. The migration has meant a first step towards the interface with other systems (such as SDB and Agile) and a generalization of the Event Management process to all services at CERN. In this talk we will present the full architecture, the workflow and the facilities implemented in ServiceNow for this migration. The experience gained during the first months of operation will be discussed together with the interface of the current Event Management structure to other monitoring systems including Spektrum and the future Agile structure and other service databases such as SDB and CDB. The usage of this structure is extended to the service tracking at the Wigner Center in Budapest.
Speaker: Dr Patricia Mendez Lorenzo (CERN)
• 169
Monitoring in a grid cluster
The monitoring of a grid cluster (or of any piece of reasonably scaled IT infrastructure) is a key element in the robust and consistent running of that site. There are several factors which are important to the selection of a useful monitoring framework, which include ease of use, reliability, data input and output. It is critical that data can be drawn from different instrumentation packages and collected in the framework to allow for a uniform view of the running of a site. It is also very useful to allow different views and transformations of this data to allow its manipulation for different purposes, perhaps unknown at the initial time of installation. In this context, we firstly present the findings of an investigation of the Graphite monitoring framework and its use at the Scotgrid Glasgow site. In particular, we examine the messaging system used by the framework and means to extract data from different tools, including the existing framework Ganglia which is in use at many sites, in addition to adapting and parsing data streams from external monitoring frameworks and websites. We also look at different views in which the data can be presented to allow for different use cases. We report on the installation and maintenance of the framework from a system manager perspective. This work is relevant to site managers and anyone interested in high level, adaptable site monitoring.
Speaker: Mark Mitchell (University of Glasgow)
• 170
Monitoring of large-scale federated data storage: XRootD and beyond.
The computing models of the LHC experiments are gradually moving from hierarchical data models with centrally managed data pre-placement towards federated storage which provides seamless access to data files independently of their location and dramatically improved recovery due to fail-over mechanisms. Enabling loosely coupled data clusters to act as a single storage resource should increase opportunities for data analysis and should enable more effective use of computational resources at sites with limited storage capacities. Construction of the data federations and understanding the impact of the new approach to data management on user analysis requires complete and detailed monitoring. Monitoring functionality should cover the status of all components of the federated storage, measuring data traffic and data access performance, as well as being able to detect any kind of inefficiencies and to provide hints for resource optimization and effective data distribution policy. Data mining of the collected monitoring data provides a deep insight into new patterns of usage of the storage resources, beyond that provided by other monitoring strategies. In the WLCG context, there are several federations currently based on the XRootD technology. The talk will focus on monitoring for the ATLAS and CMS XRootD federations (Federated Atlas XRootD (FAX) and Any Data, Any Time, Anywhere (AAA)) implemented in the Experiment Dashboard monitoring framework. Both federations consist of many dozens of sites accessed by many hundreds of clients and they continue to grow in size. Handling of the monitoring flow generated by these systems has to be well optimized in order to achieve the required performance. The talk will demonstrate that though FAX and AAA Dashboards are being developed for XRootD federations, the implementation is generic and can be easily adapted for other technologies, such as HTTP/WebDAV federations.
Speaker: Alexandre Beche (CERN)
• 171
Monitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY
Small Virtual Organizations (VO) employ all components of the EMI or gLite Middleware. In this framework, a monitoring system is designed for the H1 Experiment to identify and recognize within the GRID the best suitable resources for execution of CPU-time consuming Monte Carlo (MC) simulation tasks (jobs). Monitored resources are Computer Elements (CEs), Storage Elements (SEs), WMS-servers (WMSs), CernVM File System (CVMFS) available to the VO "hone" and local GRID User Interfaces (UIs). The general principle of monitoring of the GRID elements is based on the execution of short test jobs on different CE queues using submission through various WMSs and directly to the CREAM-CEs as well. Real H1 MC Production jobs with a small number of events are used to perform the tests. Test jobs are periodically submitted into GRID queues, the status of these jobs is checked, output files of completed jobs are retrieved, the result of each job is analyzed and the waiting time and run time are derived. Using this information, the status of the GRID elements is estimated and the most suitable ones are included in the automatically generated configuration files for use in the H1 MC production. Monitored information is stored in a MySQL database and is presented in detail on web pages and the MonAlisa visualisation system. The monitoring system allows for identification of problems in the GRID sites and promptly reacts on it (for example by sending GGUS trouble tickets). The system can easily be adapted to identify the optimal resources for tasks other than MC production, simply by changing to the relevant test jobs. The monitoring system is written mostly in Python and Perl with insertion of a few shell scripts. In addition to the test monitoring system we additionally use information from real production jobs to monitor the availability and quality of the GRID resources. The monitoring tools register the number of job resubmissions, the percentage of failed and finished jobs relative to all jobs on the CEs and determine the average values of waiting and running time for the involved GRID queues. CEs which do not meet the set criteria can be removed from the production chain by including them in an exception table. All of these monitoring actions lead to a more reliable and faster execution of MC requests.
Speaker: Bogdan Lobodzinski (DESY, Hamburg, Germany)
• 172
Negative improvements
An extensively documented, quantitative study of software evolution resulting in deterioration of physical accuracy over the years is presented. The analysis concerns the energy deposited by electrons in various materials produced by Geant4 versions released between 2007 and 2013. The evolution of the functional quality of the software is objectively quantified by means of a rigorous statistical analysis, which combines goodness-of-fit tests and methods of categorical data testing to validate the simulation against high precision experimental measurements. Significantly lower compatibility with experiment is observed with the later Geant4 versions subject to evaluation; the significance level of the test is 0.01. Various issues related to the complexity of appraising the evolution of software functional quality are considered, such as the dependence on the experimental environment where the software operates and its sensitivity to detector characteristics. Methods and techniques to mitigate the risk of “negative improvements” are critically discussed: they concern various disciplines of the software development process, including not only testing and quality assurance, but also domain decomposition, software design and change management. Concrete prototype solutions are presented. This study is intended to provide a constructive contribution to identify possible causes of the deterioration of software functionality, and means to address them effectively. It is especially relevant to the HEP software environment, where widely used tools and experiments’ software are expected to stand long life-cycles and are necessarily subject to evolution.
Speaker: Georg Weidenspointner (MPE Garching)
• 173
New physics and old errors: validating the building blocks of major Monte Carlo codes
A large-scale project is in progress, which validates the basic constituents of the electromagnetic physics models implemented in major Monte Carlo codes (EGS, FLUKA, Geant4, ITS, MCNP, Penelope) against extensive collections of experimental data documented in the literature. These models are responsible for the physics observables and the signal generated in particle detectors, including those originating from the products of hadronic interactions. Total and differential cross sections, angular distributions and other basic physics features are examined. In addition to currently implemented models, theoretical calculations, semi-empirical models and physics data libraries not yet exploited in Monte Carlo codes are evaluated and compared to those currently in use. The results of this analysis identify and quantify the state of the art achievable in Monte Carlo simulation based on the available body of knowledge. This validation analysis highlights similarities and differences in the modeling choices adopted by major Monte Carlo codes, and quantitatively documents their accuracy based on rigorous statistical methods. Categorical data analysis techniques are applied to quantify objectively the significance of the differences in compatibility with experiment among the examined Monte Carlo codes. Duplicated models are identified, not only across different Monte Carlo systems, but also between packages of the same system, and even within the same package. Guidelines for pruning duplicated functionality in Geant4 are discussed. In parallel to the evaluation of physics accuracy, the computational performance of alternative physics models and calculation approaches is estimated. Quantitative results show that there is no univocal winner between analytical calculations and data library interpolation in terms of computational speed. The interplay of this validation methodology with epistemic uncertainties embedded in Monte Carlo codes and its applicability also to hadronic physics models are discussed.
Speaker: Dr Maria Grazia Pia (Universita e INFN (IT))
• 174
Next Generation HEP Networks at Supercomputing 2012
We review the demonstration of next generation high performance 100 Gbps networks for HEP that took place at the Supercomputing 2012 (SC12) conference in Salt Lake City. Three 100 Gbps circuits were established from the California Institute of Technology, the University of Victoria and the University of Michigan to the conference show floor. We were able to to efficiently utilize these circuits using limited set of hardware surpassing previous records established at SC11. Highlights include a record overall disk to disk rate using the three links of 187 Gbps, a unidirectional transfer between storage systems in Victoria and Salt Lake on one link of 96 Gbps, an 80 Gbps transfer from Caltech to a single server with two 40GE interfaces at Salt Lake with nearly 100% use of the servers' interfaces at both ends, and a transfer using Remote Data Memory Access (RDMA) over Ethernet between Pasadena and Salt Lake that sustained 75 Gbps with a CPU load on the servers of only 5%. A total of 3.8 Petabytes was transferred over the three days of the conference exhibit, including 2 Petabytes on the last day. Three different storage setup were used during the demonstration: an conventional disk Lustre system, 2U rack servers containing solid state disks and systems containing PCI Express 3.0 Solid State storage cards.
Speaker: Ian Gable (University of Victoria (CA))
• 175
Next Generation PanDA Pilot for ATLAS and Other Experiments
The Production and Distributed Analysis system (PanDA) has been in use in the ATLAS Experiment since 2005. It uses a sophisticated pilot system to execute submitted jobs on the worker nodes. While originally designed for ATLAS, the PanDA Pilot has recently been refactored to facilitate use outside of ATLAS. Experiments are now handled as plug-ins, and a new PanDA Pilot user only has to implement a set of prototyped methods in the plug-in classes, and provide a script that configures and runs the experiment specific payload. We will give an overview of the Next Generation PanDA Pilot system and will present major features and recent improvements including live user payload debugging, data access via the federated xrootd system, stage-out to alternative storage elements, support for the new ATLAS DDM system (Rucio), and an improved integration with glExec, as well as a description of the experiment specific plug-in classes. The performance of the pilot system in processing LHC data on the OSG, LCG and NorduGrid infrastructures used by ATLAS will also be presented. We will describe plans for future development on the time scale of the next few years.
Speaker: Paul Nilsson (University of Texas at Arlington (US))
• 176
Next-Generation Navigational Infrastructure and the ATLAS Event Store
The ATLAS event store employs a persistence framework with extensive navigational capabilities. These include real-time back navigation to upstream processing stages, externalizable data object references, navigation from any data object to any other both within a single file and across files, and more. The 2013-2014 shutdown of the Large Hadron Collider provides an opportunity to enhance this infrastructure in several ways that both extend these capabilities and allow the collaboration to better exploit emerging computing platforms. Enhancements include redesign with efficient file merging in mind, content-based indices in optimized reference types, and support for forward references. The latter provide the potential to construct valid references to data before those data are written, a capability that is useful in a variety of multithreading, multiprocessing, distributed processing, and deferred processing scenarios. This paper describes the architecture and design of the next generation of ATLAS navigational infrastructure.
Speaker: Dr Peter Van Gemmeren (Argonne National Laboratory (US))
• 177
Offline software for the PANDA Luminosity Detector
Precise luminosity determination is crucial for absolute cross-section measurements and scanning experiments with the fixed target PANDA experiment at the planned antiproton accelerator HESR (FAIR, Germany). For the determination of the luminosity we will exploit the elastic antiproton-proton scattering. Unfortunately there are no or only a few data with large uncertainties available in the momentum range we need for PANDA. Therefore in order to minimize the large systematic uncertainty from theory and former experiments we will perform measurements at very small momentum transfer (and thus very small scattering angle), where the Coulomb part of the elastic cross section can be calculated very precisely. To achieve the precision needed for the luminosity determination the detector should have full angular acceptance and good spatial resolution. The current design are four planes of sensors with distances of 10 or 20 cm in between. The sensors themselves are HV-MAPS (High Voltage Monolithic Active Pixel Sensor). The whole set-up is placed in vacuum to minimize multiple scattering. In parallel to the prototype construction this design is under study with Monte Carlo based simulation. The luminosity detector is located outside of any magnetic field and will register hits of the straight tracks of the scattered antiprotons. Our reconstruction software includes hit reconstruction, track search, track fitting and software alignment. Moreover there are specific procedures like luminosity extraction or background treatment. Nowadays tracking systems usually measure up to hundred hits per track, which demands quite complicated pattern recognition algorithms, but makes track reconstruction more robust against loosing of hits in a track. In our case we have only a little amount of hits per track and have to be sure of using most of them. Therefore we developed two different algorithms for track search and did a performance study for the comparison of the algorithms. In order to achieve the best resolution of the scattering angle measurement even a misalignment of 50 um has to be avoided. The Millipede algorithm will adjust the alignment corrections for all sensors simultaneously. The limitation of the Millipede algorithm (originally developed for a large number of layers) is studied in detail. In this presentation the basic concept will be introduced before focusing on Monte Carlo based performance studies of each reconstruction step.
Speaker: Anastasia Karavdina (University Mainz)
• 178
Optimising network transfers to and from QMUL, a large WLCG Tier-2 Grid site
The WLCG, and high energy physics in general, relies on remote Tier-2 sites to analyse the large quantities of data produced. Transferring this data in a timely manner requires significant tuning to make optimum usage of expensive WAN links. In this paper we describe the techniques we have used at QMUL to optimise network transfers. Use of the FTS with settings and appropriate TCP window sizes allowed us to saturate a 1 Gbit link for 24 hours - whilst still achieving acceptable download speeds. Source based routing and multiple gridftp servers allowed us to use an otherwise unused "resilient" link. After the WAN link was upgraded to 10Gbit/s, a significant reduction in transfer rates was observed from some sites - due to suboptimal routing resulting in packet loss on congested links. Solving this dramatically improved performance. The use of jumbo frames (MTU=9000) offers potential improvements in performance, particularly for latency limited links. Whilst much of the Internet backbone is capable of supporting this, most sites are not, and path MTU discovery fails at some sites. We describe our experience with this.
Speaker: Christopher John Walker (University of London (GB))
• 179
Optimising query execution time in LHCb Bookkeeping System using partition pruning and partition wise joins
The LHCb experiment produces a huge amount of data which has associated metadata such as run number, data taking condition (detector status when the data was taken), simulation condition, etc. The data are stored in files, replicated on the Computing Grid around the world. The LHCb Bookkeeping System provides methods for retrieving datasets based on their metadata. The metadata is stored in a hybrid database model, which is a mixture of Relational and Hierarchical database models and is based on the Oracle Relational Database Management System (RDBMS). The database access has to be reliable and fast. In order to achieve a high timing performance, the tables are partitioned and the queries are executed in parallel. When we store large amounts of data the partition pruning is essential for database performance, because it reduces the amount of data retrieved from the disk and optimises the resource utilisation. This research presented here is focusing on the extended composite partitioning strategy such as range-hash partition, partition pruning and usage of the partition wise joins. The system has to serve thousands of queries per minute, the performance and capability of the system is measured when the above performance optimisation techniques are used.
Speaker: Zoltan Mathe (CERN)
• 180
Optimization of Italian CMS Computing Centers via MIUR funded Research Projects
The Italian Ministry of Research (MIUR) funded in the past years research projects aimed to an optimization of the analysis activities in the Italian CMS computing Centers. A new grant started in 2013, and activities are already ongoing in 9 INFN sites, all hosting local CMS groups. Main focus will be on the creation of an italian storage federation (via Xrootd initially, and later HTTP) which allows all the italian CMS physicists to a privileged access to CMS data and simulations. Another task will focus on the optimization of the last step of a CMS analysis, via interactive access to resources; this will result in a number of small- to medium-sized analysis centers, where access will be granted at national level to multicore machines, PROOF facilities, high throughput local queues. An important part of this last activity will imply experimenting with on demand analysis machine instantiation via Clouds, using the experience and the resources INFN is building on the subject.
Speakers: Dr Giacinto Donvito (INFN-Bari), Tommaso Boccali (Sezione di Pisa (IT))
• 181
Parallelization of particle transport with INTEL TBB
Speaker: Egor Ovcharenko (ITEP Institute for Theoretical and Experimental Physics (RU))
• 182
Performance and development plans for the Inner Detector trigger algorithms at ATLAS
We present a description of the algorithms and the performance of the ATLAS Inner Detector trigger for LHC run I, as well as prospects for a redesign of the tracking algorithms in run 2. The Inner Detector trigger algorithms are vital for many trigger signatures at ATLAS. The performance of the algorithms for muons, electrons, taus and b-jets is presented. The ATLAS trigger software after will be restructured from 2 software levels into a single stage which poses a big challenge on the trigger algorithms in terms of execution time and maintaining the physics performance. Expected future improvements in the timing and efficiencies of the Inner Detector triggers are discussed, utilising the planned merging of the current two-stage software of the ATLAS trigger.
Speaker: Stewart Martin-Haugh (University of Sussex (GB))
• 183
Performance evaluation and capacity planning for a scalable and highly available virtulization infrastructure for the LHCb experiment
The virtual computing is often run to satisfy different needs: reduce costs, reduce resources, simplify maintenance and the last but not the least add flexibility. The use of Virtualization in a complex system such as a farm of PCs that control the hardware of an experiment (PLC, power supplies ,gas, magnets..) put as in a condition where not only an High Performance requirements need to be carefully considered but also a deep analysis of strategies to achieve a certain level of High Availability. We conducted a performance evaluation on different and comparable storage/network/virtulization platforms. The performance is measured using a series of independent benchmarks , testing the speed an the stability of multiple VMs runnng heavy-load operations on the I/O of virtualized storage and the virtualized network. The result from the benchmark tests allowed us to study and evaluate how the different workloads of Vm workloads interact with the Hardware/Software resource layers.
Speakers: Enrico Bonaccorsi (CERN), Francesco Sborzacchi (Istituto Nazionale Fisica Nucleare (IT)), Niko Neufeld (CERN)
• 184
Performance evaluation of a dCache system with pools distributed over a wide area
• 185
Performance of most popular open source databases for HEP related computing problems
Databases are used in many software components of the HEP computing, from monitoring and task scheduling to data storage and processing. While the database design choices have a major impact on the system performance, some solutions give better results out of the box than the others. This paper presents detailed comparison benchmarks of the most popular Open Source systems for a typical class of problems relevant to HEP computing.
Speaker: Dr Dmytro Kovalskyi (Univ. of California Santa Barbara (US))
• 186
Phronesis, a diagnosis and recovery tool for system administrators
The backbone of the LHCb experiment is the Online system, which is a very large and heterogeneous computing center. Making sure of the proper behavior of the many different tasks running on the more than 2000 servers represents a huge workload for the small expert-operator team and is a 24/7 task. At the occasion of CHEP 2012, we presented a prototype of a framework that we designed in order to support the experts. The main objective is to provide them with always improving diagnosis and recovery solutions in case of misbehavior of a service, without having to modify the original applications. Our framework is based on adapted principles of the Autonomic Computing model, on reinforcement learning algorithms, as well as innovative concepts such as Shared Experience. While the presentation made at CHEP 2012 showed the validity of our prototype on simulations, we here present a version with improved algorithms, manipulation tools, and report on experience with running it in the LHCb Online system.
Speaker: Christophe Haen (Univ. Blaise Pascal Clermont-Fe. II (FR))
• 187
PLUME – FEATHER
PLUME - FEATHER is a non-profit project created to Promote economicaL, Useful and Maintained softwarE For the Higher Education And THE Research communities. The site references software, mainly Free/Libre Open Source Software (FLOSS) from French universities and national research organisations, (CNRS, INRA...), laboratories or departments. Plume means feather in French. The main goals of PLUME – FEATHER are: • promote the community's own developments, • contribute to the developement and sharing FLOSS (Free/Libre Open Source Software) information, experiences and expertise in the community, • bring together FLOSS experts and knowledgeable people to create a community, • foster and facilitate FLOSS use, deployment and contribution in the higher education and the research communities. PLUME - FEATHER was initiated by the CNRS unit UREC, which has been integrated into the CNRS computing division DSI in 2011. Various resources are provided by the main partners involved in the project, coming from many french research institutes. The French PLUME server contains more than 1000 software reference cards, edited and peer-reviewed by more than 909 contributors, out of 2000 overall members from the research and education community. The site http://projet-plume.org/ is online since November 2007, and the first English pages have been published in April 2009. Currently there are 91 software products referenced in the PLUME-FEATHER area, 14 of them having been published since January 2012. The PLUME project is presented regularly in national conferences and events of the FLOSS community, and at CHEP 2012. Therefore we renew our proposal to expose a poster about the services available from the PLUME project on the international level to find not only users, but also contributors: editors and reviewers of frequently used software in our domain.
Speaker: Dr Dirk Hoffmann (Centre de Physique des Particules de Marseille, CNRS/IN2P3)
• 188
Popularity Prediction Tool for ATLAS Distributed Data Management
This paper describes a popularity prediction tool for data-intensive data management systems, such as the ATLAS distributed data management (DDM) system. The tool is fed by the DDM popularity system, which produces historical reports about ATLAS data usage and provides information about the files, datasets, users and sites where data was accessed. The tool described in this contribution uses this historic information to make a prediction about the future popularity of data. It finds trends in the usage of data using a set of neural networks and a set of input parameters and predicts the number of accesses in the near term future. This information can then be used in a second step to improve the distribution of replicas at sites, taking into account the cost of creating new replicas (bandwidth and load on the storage system) compared to the gain of having new ones (faster access of data for analysis). The tool ensures that the total amount of space available on the grid is not exceeded. This information can then help to make a decision about adding and also removing data from the grid to make a better use of the available resources. The design and architecture of the popularity prediction tool is described, examples of its use are shown and an evaluation of its performance is presented.
Speaker: Graeme Andrew Stewart (CERN)
• 189
Preparing the Gaudi-Framework and the DIRAC-WMS for Multicore Job Submission
Due to the continuously increasing number of cores on modern CPUs, it is important to adapt HEP applications. This must be done at different levels: the software which must support parallelization and the scheduling has to differ between multicore and singlecore jobs. The LHCb software framework (GAUDI) provides a parallel prototype (GaudiMP), based on the multiprocessing approach. It allows a reduction of the overall memory footprint and a coordinated access to data via separated reader and writer processes. A comparison between the parallel prototype and multiple independent Gaudi jobs in respect to CPU-time and memory consumption will be shown. In the context of parallelization speedup is the most important metric, as it shows how software scales with the number of cores. It is influenced by many factors, due to software limitations like synchronization, but also due to hardware configurations, like frequency scaling. Those limitations and their dependencies will be discussed and the influence of hardware features will be evaluated, in order to forecast the spread in CPU-time. Furthermore, speedup must be predicted in order to find the limit beyond which the parallel prototype (GaudiMP) does not support further scaling. This number must be known as it indicates the point, where new technologies must be introduced into the software framework. In order to reach further improvements in the overall throughput, scheduling strategies for mixing parallel jobs can be applied. It allows overcoming limitations in the speedup of the parallel prototype. Those changes require modifications at the level of the workload management system (DIRAC). Results will be presented for the reconstruction, simulation and analysis software of the LHCb experiment.
Speaker: Nathalie Rauschmayr (CERN)
• 190
The ALEPH Collaboration took data at the LEP (CERN) electron-positron collider in the period 1989-2000, producing more than 300 scientific papers. While most of the Collaboration activities stopped in the last years, the data collected still has physics potential, with new theoretical models emerging, and needing a check with data at the Z and WW production energies. An attempt to revive and preserve the ALEPH Computing Environment is presented; the aim is not only the preservation of the data files (usually called “bit preservation”), but of the full environment a physicist would need to perform brand new analyses. Technically, a Virtual Machine approach has been chosen, using the VirtualBox platform. Concerning simulated events, the full chain from event generators to physics plots is possible, and reprocessing of data events is also functioning. Interactive tools like the DALI event display can be used on both data and simulated events. The Virtual Machine approach seems suited for both interactive usage, and for massive computing using Cloud like approaches. Studies are now moving from technical functionality tests (which are positively concluded), to tests and development on how to guarantee an easy and transparent access to ALEPH data in the virtualized platform.
Speaker: Simone Coscetti (Sezione di Pisa (IT))
• 191
Prototyping a Multi‐10‐Gigabit Ethernet Event‐Builder for a Cherenkov Telescope Array
We are developing the prototype of a high speed data acquisition (DAQ) system for the Cherenkov Telescope Array. This experiment will be the next generation ground-based gamma-ray instrument. It will be made up of approximately 100 telescopes of at least three different sizes, from 6 to 24 meters in diameter. Each camera equipping the telescopes is composed of hundreds of light detecting modules pushing out data through gigabit Ethernet links. They will generate a total data flow of up to 24 Gb/s. Merging and handling such data rates with a single off the shelf computer and switches without any data loss require well designed and tested software and hardware. In a first stage, the software receives and reconstructs the incoming partial events. In a second stage, it performs on-line calculations with the data in order to improve event selection and sends the remaining data to the central DAQ. For the purpose of testing and stimulating our DAQ system, we designed and started to build a full scale dummy camera cluster with single board computers. This cluster provides 300 physical gigabit Ethernet ports which will send pre-calculated simulation data to the DAQ system, reproducing the timing and the instantaneous flow of a real camera. With this equipment, we are able to validate our hardware and software architectures. We will present our approach for the development of such a high data rate system, first measurements and solutions that we have applied to solve the problems we encountered to sustain the maximum dataflow reliably.
Speaker: Dr Dirk Hoffmann (Centre de Physique des Particules de Marseille, CNRS/IN2P3)
• 192
Quality Assurance for simulation and reconstruction software in CBMROOT
The software framework of the CBM experiment at FAIR - CBMROOT - has been continuously growing over the years. The increasing complexity of the framework and number of users require improvements in maintenance, reliability and in overall software development process. In this report we address the problem of the software quality assurance (QA) and testing. Two main problems are considered in our test suit. First, test of the build process (configuration and compilation) on different systems. Second, test of correctness of the simulation and reconstruction results. The build system and QA infrastructure are based on CMake, CTest and CDash. The build process is tested using the standard above-mentioned set of tools. For the simulation and reconstruction tests a set of tools was developed, which includes base classes for reports, histogram management, a simulation and reconstruction QA classes and scripts. Test results in form of the user-friendly reports are published on the CDash and on the dedicated web-server where developer can browse, for example, the tracking performance two weeks ago in order to fix the bug. Described QA system considerably improves the development process and leads to a faster development cycles of CBMROOT.
Speaker: Semen Lebedev (Justus-Liebig-Universitaet Giessen (DE))
• 193
R&D work for a data model definition: data access and storage system studies
In HEP computing context, R&D studies aiming to the definition of the data and workload models were brought forward by the SuperB community beyond the experiment life itself. This work is considered of great interest for a generic mid- and small size VO during its Computing Model definition phase. Data-model R&D work we are presenting, starts with the general design description of the crucial components in terms of typical HEP use cases; a discussion on strategies and motivations for the taken choices in the fields of data access, mass data transfer and meta-data catalog system is provided firstly. In such a context we focused the evaluation, test and development work on storage systems enabled for geographically-distributed data management: data access, data replication, data recovery and backup in WAN scenarios. HadoopFS and GlusterFS distributed file-system have been mainly considered in this analysis. Data availability in a distributed environment is a key point in the definition of the computing model for an HEP experiment. Among all the possible interesting data models, we identify the WAN direct access via reliable and robust protocols such as HTTP/WebDAV and xrootd as a viable option. The development of a dedicated library has been carried on allowing an optimized file access procedure on remote storage resources. The implemented features include read-ahead and data prefetching techniques, caching mechanism and optimized target file localization. The results of performance and efficiency tests will be presented for the treated subjects trying to describe in conclusion the general strategy lines and technologies for the drafting of a concrete data model design report.
Speakers: Dr Armando Fella (INFN Pisa), Mr Domenico Diacono (INFN Bari), Dr Giacinto Donvito (INFN-Bari), Mr Giovanni Marzulli (GARR), Paolo Franchini (Universita e INFN (IT)), Dr Silvio Pardi (INFN)
• 194
Re-designing the PhEDEx security model
PhEDEx. the data-placement tool used by the CMS experiment at the LHC, was conceived in a more trusting time. The security model was designed to provide a safe working environment for site agents and operators, but provided little more protection than that. CMS data was not sufficiently protected against accidental loss caused by operator error or software bugs or from loss of data caused by deliberate manipulation of the database. Operations staff were given high levels of access to the database, beyond what should have been needed to accomplish their tasks. This exposed them to the risk of suspicion should an incident occur. Multiple implementations of the security model led to difficulties maintaining code, which can lead to degredation of security over time. In order to meet the simultaneous goals of protecting CMS data, protecting the operators from undue exposure to risk, increasing monitoring capabilities and improving maintainability of the security model, the PhEDEx security model was redesigned and re-implemented. Security was moved from the application layer into the database itself, fine-grained access roles were established, and tools and procedures created to control the evolution of the security model over time. In this paper we describe this work, we describe the deployment of the new security model, and we show how the resulting enhancements have improved security on several fronts simultaneously.
Speaker: Dr Tony Wildish (Princeton University (US))
• 195
Real-time flavor tagging selection in ATLAS
In high-­‐energy physics experiments, online selection is crucial to reject most uninteresting collisions and to focus on interesting physical signals. The b-­‐jet selection is part of the trigger strategy of the ATLAS experiment and is meant to select hadronic final states with heavy-­‐flavor content. This is important for the selection of physics channels with more than one b-­‐jet in the final state, enabling to reject QCD light jets and maintain affordable trigger rates without raising jet energy thresholds. ATLAS introduced b-­‐jet triggers in 2011 and deployed more complex and better performing tagging algorithms in 2012. An overview of the b-­‐jet trigger menu and its performance on real data is presented in this contribution. Data-­‐driven techniques to extract the online b-­‐tagging performance, a key ingredient for all analyses relying on such triggers, are also discussed and results presented.
Speaker: Adrian Buzatu (University of Glasgow (GB))
• 196
Recent Developments in the Geant4 Hadronic Framework
In this paper we present the recent developments in the Geant4 hadronic framework, as well as in some of the existing physics models. Geant4 is the main simulation toolkit used by the LHC experiments and therefore a lot of effort is put into improving the physics mod els in order for them to have more predictive power. As a consequence, the code complexity increases, which requires constant improvement and optimisation on the programming side. At the same time, we would like to review and eventually reduce the complexity of the hadronic software framework. As an example, a factory design pattern has been applied in Geant4 to avoid duplications of objects, like cross sections, which can be used by several processes or physics models. This approach has been applied also for physics lists, to provide a flexible configuration mechanism at run-time, based on macro files. Moreover, these developments open the future possibility to build Geant4 with only a specified sub-set of physics models. Another technical development focused on the reproducibility of the simulation, i.e. the possibility to repeat an event once the random generator status at the beginning of the event is known. This is crucial for debugging rare situations which may occur after long simulations. Moreover, reproducibility in normal, sequential Geant4 simulation is an important prerequisite to verify the equivalence with multi-threaded Geant4 simulations.
Speaker: Witold Pokorski (CERN)
• 197
Reconstruction of the Higgs mass in H -> tautau Events by Dynamical Likelihood techniques
An algorithm for reconstruction of the Higgs mass in $H \rightarrow \tau\tau$ decays is presented. The algorithm computes for each event a likelihood function $P(M_{\tau\tau})$ which quantifies the level of compatibility of a Higgs mass hypothesis $M_{\tau\tau}$, given the measured momenta of visible tau decay products plus missing transverse energy reconstructed in the event. The algorithm is used in the CMS $H \rightarrow \tau\tau$ analysis. It is found to improve the sensitivity for the Standard Model Higgs boson in this decay channel by about $30\%$.
Speaker: Christian Veelken (Ecole Polytechnique (FR))
• 198
Redundant Web Services Infrastructure for High Performance Data Access
RESTful web services are popular solution for distributed data access and information management. Performance, scalability and reliability of such services is critical for the success of data production and analysis in High Energy Physics as well as other areas of science. At FNAL, we have been successfully using REST HTTP-based data access architecture to provide access to various types of data. We have built a simple yet versatile infrastructure which allows us to use redundant copies of web application servers to increase service performance and availability. It is designed to handle both state-less and state-full data access methods using distributed web server. In most cases, the infrastructure allows us to add or remove individual application servers at any time without a visible interruption of the service. This infrastructure has been successfully used for several years now with data web services as well as with interactive web applications. We will present components of our infrastructure and several examples of how it can be used.
Speaker: Mr Igor Mandrichenko (Fermilab)
• 199
Request for All - Generalized Request Framework for PhEDEx
PhEDEx has been serving CMS community since 2004 as the data broker. Every PhEDEx operation is initiated by a request, such as request to move data, request to delete data, and so on. A request has it own life cycle, including creation, approval, notification, and book keeping and the details depend on its type. Currently, only two kinds of requests, transfer and deletion, are fully integrated in PhEDEx. They are tailored specifically to the operations' workflows. To be able to serve a new type of request it generally means a fair amount of development work. After several years of operation, we have gathered enough experience to rethink the request handling in PhEDEx. "Generalized Request Project" is set to abstract such experience and come up with a request system which is not tied into current workflow yet it is general enough to accommodate current and future requests. The challenges are dealing with different stages in a request's life cycle, complexity of approval process and complexity of the ability and authority associated with each role in the context of the request. We start with a high level abstraction driven by a deterministic finite automata, followed by a formal description and handling of approval process, followed by a set of tools that make such system friendly to the users. As long as we have a formal way to describe the life of a request and a mechanism to systematically handle it, to server a new kind of request is merely a configuration issue, adding the description of the new request rather than development effort. In this paper, we share the design and implementation of a generalized request framework and the experience of taking an existing serving system through a re-design and re-deployment.
Speaker: Dr Tony Wildish (Princeton University (US))
• 200
ROOT I/O in JavaScript - Reading ROOT files in a browser
In order to be able to browse (inspect) ROOT files in a platform independent way, a JavaScript version of the ROOT I/O subsystem has been developed. This allows the content of ROOT files to be displayed in most available web browsers, without having to install ROOT or any other software on the server or on the client. This gives a direct access to ROOT files from any new device in a light way. It is possible to display simple graphical objects such as histograms and graphs (TH1, TH2, TH3, TProfile, TGraph, ...). The rendering of 1D/2D histograms and graphs is done with an external JavaScript library (d3.js), and 2D & 3D histograms with another library (three.js). This poster will describe the techniques used to stream and display the content of a ROOT file, with a rendering being now very close to the one provided by ROOT.
Speaker: Bertrand Bellenot (CERN)
• 201
ROOT: native graphics on Mac OS X
In my poster I'll present a new graphical back-end for ROOT that has been developed for the Mac OS X operating system as an alternative to the more than 15 year-old X11-based version. It represents a complete implementation of ROOT's GUI, 2D and 3D graphics based on Apple's native APIs/frameworks, written in Objective-C++.
Speaker: Bertrand Bellenot (CERN)
• 202
Round-tripping DIRAC: Automated Model-Checking of Concurrent Software Design Artifacts
A big challenge in concurrent software development is early discovery of design errors which can lead to deadlocks or race-conditions. Traditional testing does not always expose such problems in complex distributed applications. Performing more rigorous formal analysis, like model-checking, typically requires a model which is an abstraction of the system. For object-oriented software, UML is the industry-adopted modeling language, offering behavioral views that capture the dynamics of the system with features for modeling code-like structures, such as loops, conditions, and referring to existing interactions. We present an automatic procedure for translating UML into mCRL2 process algebra models, amenable to automatic model-checking. Our prototype is able to produce a formal model, and feed model-checking traces back into any UML modeling tool, without the user having to leave the UML domain. We apply our methodology to the newly developed Executors framework, part of DIRAC’s WMS system responsible for orchestrating the workflow steps of jobs submitted to the grid. Executors process any task sent to them by a Dispatcher, each one responsible for a different step (such as resolving the input data for a job). The Dispatcher takes care of persisting the jobs states and distributing them among the Executors. Occasionally, tasks submitted in the system would not get dispatched out of the queue, despite the fact that their responsible Executors were idle at the moment. The root cause of this problem could not be identified by certification testing with different workload scenarios, nor by analysis of the generated logs. We used our toolset to generate an mCRL2 model of this system, based on the reverse-engineered sequence diagrams. Model-checking the generated mCRL2 model discovered a trace violating the desired progress requirement, the bug being localized in the Dispatcher component implementation. The trace was automatically translated back to the UML domain, showing an intuitive view of the communication between components which leads to the faulty behavior.
Speaker: Daniela Remenska (NIKHEF (NL))
• 203
Scientific Collaborative Tools Suite at FNAL
Over several years, we have developed a number of collaborative tools used by groups and collaborations at FNAL, which is becoming a Suite of Scientific Collaborative Tools. Currently, the suite includes: - Electronic Logbook (ECL), - Shift Scheduler, - Speakers Bureau and - Members Database. These product organize and help run the collaboration at every stage of its life cycle from detector building to simulation, data acquisition, processing, analysis and publication of the results of the research. These projects share common technologies and architecture solutions. Mobile computing opens whole new area of exciting opportunities for collaborative tools and new vision of the human/computer interface. The presentation will cover history, current status and future development plans for the Scientific Collaborative Tools Suite at FNAL.
Speaker: Mr Igor Mandrichenko (Fermilab)
• 204
Security in the CernVM File System and the Frontier Distributed Database Caching System
Both the CernVM File System (CVMFS) and the Frontier Distributed Database Caching System (Frontier) distribute centrally updated data worldwide for LHC experiments using http proxy caches. Neither system provides privacy or access control on reading the data, but both control access to updates of the data and can guarantee the integrity of the data transferred to clients over the internet. CVMFS has since its early days required digital signatures and secure hashes on all distributed data, and recently both CVMFS and Frontier have added X509-based integrity checking. In this paper we detail and compare the security models of CVMFS and Frontier.
Speaker: Jakob Blomer (CERN)
• 205
Self managing experiment resources
Within this paper we present an autonomic Computing resources management system used by LHCb for assessing the status of their Grid resources. Virtual Organizations Grids include heterogeneous resources. For example, LHC experiments very often use resources not provided by WLCG and Cloud Computing resources will soon provide a non-negligible fraction of their computing power. The lack of standards and procedures across experiments and sites generated the appearance of multiple information systems, monitoring tools, ticket portals, etc... which nowadays coexist and represent a very precious source of information for running HEP experiments Computing systems as well as sites. These two facts lead to many particular solutions for a general problem: managing the experiment resources. In this paper we present how LHCb, via the DIRAC interware addressed such issues. With a renewed Central Information Schema hosting all resources metadata and a Status System ( Resource Status System ) delivering real time information, the system controls the resources topology, independently of the resource types. The Resource Status System applies data mining techniques against all possible information sources available and assesses the status changes, that are then propagated to the topology description. Obviously, giving full control to such an automated system is not risk-free. Therefore, in order to minimise the probability of misbehavior, a battery of tests has been provided in order to certify the correctness of its assessments. We will demonstrate the performance and efficiency of such a system in terms of cost reduction and reliability.
Speakers: Federico Stagni (CERN), Mario Ubeda Garcia (CERN)
• 206
Self-Organizing Map in ATLAS Higgs Searches
Artificial Neural Networks (ANN) are widely used in High Energy Physics, in particular as software for data analysis. In the ATLAS experiment that collects proton-proton and heavy ion collision data at the Large Hadron Collider, ANN are mostly applied to make a quantitative judgment on the class membership of an event, using a number of variables that are supposed to discriminate between different classes. The discrimination between quark-initiated and gluon-initiated jets is an example of the possible applications in the ATLAS experiment and it potentially has a great chance to improve many physics analyses. In this work the application of the unsupervised Self-Organizing Map (SOM) is proposed as quark-gluon tagging for events with two jets in the ATLAS experiment. The performance of the SOM application for quark-gluon discrimination are shown in different ranges of jet pT , confirming the feasibility of the quark-gluon tagging down to very low pT values. The application of the SOM technique to the Higgs searches is described and the results are shown and discussed.
Speaker: Giovanni Zurzolo (Universita e INFN (IT))
• 207
Simulation of the PANDA Lambda disks
Ajay Kumar and Ankhi Roy For the PANDA collaboration Indian Institute of Technology Indore, Indore-4520017, India Email- ajayk@iiti.ac.in The PANDA experiment is one of the main experiments at the future accelerator facility FAIR which is currently under construction in Darmstadt, Germany. Experiments will be performed with intense, phase space cooled antiproton beams incident on a hydrogen or heavy nuclear target. The main physics motivation of PANDA is to explore the non-perturbative regime of QCD and to study hadronic states. In this context, here is a possibility to include hyperon studies in the PANDA physics program. Hyperons travel a large distance before they decay into other particles. In order to increase the acceptance to measure these particles, there is a concept to include an additional so-called “Lambda Disk” detector. The Micro Vertex Detector (MVD) is the innermost tracking detector of PANDA. It consists of four barrel layers and six forward disk layers. It is made up of two types of silicon sensors – silicon hybrid pixels and double sided silicon strips. The last layer of the MVD forward disk is situated at 23 cm downstream of the interaction point and the first layer of GEM tracking station is located 110 cm downstream from the interaction point. Hence, there is a large region without tracking information. Therefore, it is proposed to place two additional disks known as the Lambda disks in this region. One layer is at 40 cm and the other is at 60 cm downstream from the interaction point. The detector will enhance the reconstruction probability for hyperons. As a starting geometry, it has been proposed for the Lambda disks to be made up of only double-sided silicon strip sensor. In this conceptual design, the outer ring has been kept similar to the outermost layers of the MVD forward disks and inner layer of Lambda disks has been designed using silicon strip sensor but of different size. At present, we are involved in simulation studies of the Lambda disks detector with proton anti-proton to lambda anti-lambda to calculate reconstruction efficiency and resolution of this channel. This channel provides essential inputs in understanding the vertex reconstruction of hyperon pairs. We have also started to study different parameters related to the development of the Lambda disks detector. In this presentation we will report about the reconstruction efficiency and identification probability of Lambda and Anti-Lambda particles with and without the Lambda disks detector. In addition, simulation study of detector coverage, material budget, radiation damage and rate estimation with the Lambda disks detector which are essential for the development of the detector will be presented.
Speaker: Mr Ajay Kumar (Indian Institute of Technology Indore)
• 208
softinex, inlib, exlib, ioda, g4view, g4exa, wall
Softinex names a software environment targeted to data analysis and visualization. It covers the C++ inlib and exlib "header only" libraries that permit, through GL-ES and a maximum of common code, to build applications deliverable on the AppleStore (iOS), GooglePlay (Android), traditional laptops/desktops under MacOSX, Linux and Windows, but also deliverable as a web service able to display in various web browsers compatible with WebGL (FireFox, Chrome, Safari). The ioda app permits (with fingertips on a tablet) to read files at various formats (xml-aida, cern-root, fits) and visualize some of their data such as images, histograms, ntuples and geometries. The g4view app permits to visualize Geant4 gdml files and do, for example on a tablet, some simple outreach particle physics. g4exa is a simple Geant4 template open source code for people wanting to create their own app done in the same spirit. The wall programs permit to visualize HEP data (plots, geometries, events) on a large display surface done with an assembly of screens driven by a set of computers. We want to present this software suite but also the grounding ideas, such as the "Software Least Action Principle", that led to their developments.
Speaker: Dr Guy Barrand (Universite de Paris-Sud 11 (FR))
• 209
Solving Small Files Problem in Enstore
Enstore is a tape based Mass Storage System originally designed for Run II Tevatron experiments at FNAL (CDF, D0). Over the years it has proven to be reliable and scalable data archival and delivery solution, which meets diverse requirements of variety of applications including US CMS Tier 1, High Performance Computing, Intensity Frontier experiments as well as data backups. Data intensive experiments like CDF, D0 and US CMS Tier 1 generally produce huge amount of data stored in files with the average size of few Gigabytes, which is optimal for writing and reading data to/from tape. In contrast, much of the data produced by Intensity Frontier experiments, Lattice QCD and Cosmology is sparse, resulting in accumulation of large amounts of small files. Reliably storing small files on tape is inefficient due to file marks writing which takes significant amount of the overall file writing time (few seconds). There are several ways of improving data write rates, but some of them are unreliable, some are specific to the type of tape drive and still do not provide transfer rates adequate to rates offered by tape drives (20% of the drives potential rate). In order to provide good rates for small files in a transparent and consistent manner, the Small File Aggregation (SFA) feature has been developed to provide aggregation of files into containers which are subsequently written to tape. The file aggregation uses reliable internal Enstore disk buffer. File grouping is based on policies using file metadata and other user defined steering parameters. If a small file, which is a part of a container, is requested for read, the whole container is staged into internal Enstore read cache thus providing a read ahead mechanism in anticipation of future read requests for files from the same container. SFA is provided as service implementing file aggregation and staging transparently to user. The SFA is has been successfully used since April 2012 by several experiments. Currently we are preparing to scale up write/read SFA cache. This paper describes Enstore Small Files Aggregation feature and discusses how it can be scaled in size and transfer rates.
Speaker: Dr Alexander Moibenko (Fermi NAtiona Accelerator Laboratoy)
• 210
SPADE : A peer-to-peer data movement and warehousing orchestration
The SPADE application was first used by the IceCube experiment to move its data files from the South Pole to Wisconsin. Since then is has been adapted by the DayaBay experiment to move its data files from its experiment, just outside Hong Kong, to both Beijing an LBNL. The aim of this software is to automate much of the data movement and warehousing that is often done by hand or home-grown script on smaller experiments. The latest generation of this software has been developed to be experiment independent and fully peer-to-peer. This means that it can be used anywhere, not only to move raw experiment data to computing centers but also as a means of exchanging data between centers. For example, gathering Monte Carlo production data from member institutions to an experiment's main warehouse and distributing copies back out to the users who need them. Special attention has been paid to keeping administration and infrastructure time and effort to a minimum so that this software can be easily used by experiments that do not have large computer support resources. Additional features of the software include facilities to run prompt analysis on files as they arrive in the warehouse, the archiving of files to HPSS and the option to create local copies of files before they are moved to enable immediate assessment of the data by computers based at the experiment. This paper will examine the requirement of the new version of the software. It will then discuss the main architecture of the application and show how this satisfies those requirements. This will be followed by a quick tour of what is needed to install and run the basic version, after which an explanation of now experiment specific customization can be made. Finally, the DayaBay use case will be summarized to show a practical deployment of the application.
Speaker: Dr Simon Patton (LAWRENCE BERKELEY NATIONAL LABORATORY)
• 211
Squid monitoring tools - a common solution for LHC experiments.
During the early running of the LHC , multiple collaborations began to include Squid caches in their distributed computing models. The two main use cases are: for remotely accessing conditions data via Frontier, which is used by ATLAS and CMS; and serving collaboration software via CVMFS, which is used by ATLAS, CMS, and LHCb, and is gaining traction with some non-LHC collaborations. As a result, Squid cache deployment across the grid has rapidly increased, with hundreds of Squids now in production. While some effort has been made between collaborations to share monitoring tools, this had been done on an ad-hoc basis, and it was found that some sites had duplicated monitoring, while others had no monitoring of their cache service at all. The WLCG Squid Monitoring Task Force was established to decide how to improve and better deploy Squid monitoring with WLCG common operations, and produce an architecture for a common Squid monitoring system configuration for use by all collaborations. In this paper, we present the recommendations produced by the Task Force, and the subsequent work necessary to implement them.
Speaker: Alastair Dewhurst (STFC - Science & Technology Facilities Council (GB))
• 212
Status and new developments of the Generator Services project
The LCG Generator Services project provides validated, LCG compliant Monte Carlo generators code for both the theoretical and experimental communities at the LHC. It collaborates with the generators authors, as well as the experiments software developers and the experimental physicists. In this paper we present the recent developments and the future plans of the project. We start with reporting on the current status of the generators repository. Due to the increased number of dependencies on external packages the generators repository was reorganized to follow the releases of the repository "LCG external". In order to perform the GENSER installation in a way consistent with LCG external, it was decided to migrate to the new cmake - based system, LCGCMAKE. This migration is planned to be finished in the first half of 2013. cmake - based build system is now provided for several generators. Some LHC experiments directly use generator libraries installed in the GENSER repository, others use source tarballs prepared by GENSER and also put in the repository. We discuss different testing and physics validation procedures. The regression tests system automatically checks that several physical observables does not change from one version to another and for dirrerent computer platforms supported by LCG (there are 5 such platforms now). It checks also the source tarballs provided by GENSER. This ensures that all LHC experiments use the same generator code. More detailed physical tests are performed with HepMC Analysis Tool (mainly comparison of different versions) and Rivet (mainly comparison of generation results with data). We present a new activity, within Generator Services, MCPLOTS. This subproject is intended as a simple browsable repository of MC (Monte Carlo) plots comparing High Energy Physics event generators to a wide variety of available experimental data, for tuning and reference purposes. Apart from individual plots contained in papers and presentations, there has so far not been any central database where people can quickly see how tune X of version Y of generator Z looks on distribution D. The idea with mcplots is to provide such a repository.
Speaker: Witold Pokorski (CERN)
• 213
Strategies for preserving the software development history in LCG Savannah
For more than ten years, the LCG Savannah portal has successfully served the LHC community to track issues in their software development cycles. In total, more than 8000 users and 400 projects use this portal. Despite its success, the underlying infrastructure that is based on the open-source project "Savane" did not keep up with the general evolution of web technologies and the increasing need for information security. Thus, LCG Savannah is about to be replaced by a new service based on the Jira issue and project tracking software. During the many years of preparation for LHC running, a huge amount of project specific data were collected in LCG Savannah. We discuss the importance of these historical data and the valuable knowledge they represent for the collaborations. We present our strategy for preserving the LCG Savannah data, the tools we have developed that support the migration process, and we describe the current status of the project.
Speaker: Benedikt Hegner (CERN)
• 214
Streamlining CASTOR to manage the LHC data torrent
This contribution describes the evolution of the main CERN storage system, CASTOR, as it manages the bulk data stream of the LHC and other CERN experiments, achieving nearly 100 PB of stored data by the end of LHC Run 1. Over the course of 2012 the CASTOR service has addressed the Tier-0 data management requirements, focusing on a tape-backed archive solution, ensuring smooth operations of the required experiments' workflow (data taking, reconstruction, export to other sites), and guaranteeing data safety by enforcing strong authentication. This evolution was marked by the introduction of policies to optimize the tape sub-system throughput, going towards a cold storage system where data placement is managed by the experiments' production managers. More efficient tape migrations and recalls have been implemented and deployed where bulk metadata operations greatly reduce the overhead due to small files. A repack facility is now integrated in the system and it has been enhanced in order to automate the repacking of several tens of petabytes, required in 2014 in order to prepare for the next LHC run. Finally the scheduling system has been evolved to integrate the internal monitoring for a more efficient dynamic usage of the underlying disk resources. To efficiently manage the service a solid monitoring infrastructure is required, able to analyze the logs produced by the different components (~1 kHz of log messages). A new system has been developed and deployed, which uses a transport messaging layer provided by the CERN-IT Agile Infrastructure and exploits technologies including Hadoop and HBase. This enables efficient data mining by making use of MapReduce techniques, and real-time data aggregation and visualization. We will also present the outlook for the future. A further simplification of the CASTOR system is foreseen: xrootd will take the role of the main native protocol, and only few disk pools for each experiment would serve as staging areas for the Tier-0 workflow. The adoption of xrootd as main protocol opens the possibility to exploit its upcoming features, including for instance the support of xroot federations or the HTTP protocol. Directions and possible evolutions will be discussed in view of the restart of data taking activities.
Speaker: Dr Xavier Espinal Curull (CERN)
• 215
Sustainable software and the Xenon 1 T high-level trigger
In the coming years, Xenon 1 T, a ten-fold expansion of Xenon 100, will further explore the dark matter WIMP parameter space and must be able to cope with correspondingly higher data rates. With a focus on sustainable software architecture, and a unique experimental scale compared to collider experiments, a high-level trigger system is being designed for the next many years of Xenon 1 T running. As a high level trigger, processing time must be minimized while maintaining efficient event selection. By using Python as a 'glue' language around quicker C routines, the complexity of the system can be minimized and testing facilitated. After discussing the data flow and parallelization, the talk will focus on testing techniques and software sustainability. For example, in addition to unit tests, in situ (i.e., control room) and ex situ (i.e., cloud based) system tests use old data to demonstrate the reliability of and benchmark the system. Strategies appropriate to this experimental scale that ensure sustainability will also be discussed, including the documentation and training policies, testing requirements, style guides, and general project management. In this way, the high level trigger system will be maintainable, robust, and reliable for the foreseen lifetime of the experiment.
Speaker: Christopher Tunnell
• 216
Sustainable Software LifecycManagement for Grid Middleware: Moving from central control to the open source paradigms
Speaker: Oliver Keeble (CERN)
• 217
Synergy between the CIMENT tier-2 HPC centre in Grenoble (France) and the HEP community at LPSC ("Laboratoire de Physique Subatomique et de Cosmologie")
We describe the synergy between CIMENT (a regional multidisciplinary HPC centre) and the infrastructures used for the analysis of data recorded by the ATLAS experiment at the LHC collider and the D0 experiment at the Tevatron. CIMENT is the High Performance Computing (HPC) centre developed by Grenoble University. It is a federation of several scientific departments and it is based on the gridification of a dozen HPC machines with iRODS storage. CIMENT is a medium scale centre, or a tier-2 in the HPC pyramidal scheme, of about 35 TFlops, placing it in the top-5 of the French tier-2 sites in 2012. The acquisition of an additional machine in spring 2013 is more than doubling its computing capacity and hence consolidating CIMENT's leading role in the French HCP landscape. CIMENT aims at providing significant computing power for tests and algorithm developments before execution on national [tier-1] or european [tier-0] platforms. This profile of resource allocation necessarily implies that not all resources are used at all times. The main goals of the LHC collider at CERN are the search for a subatomic particle called the Higgs boson and the search for new phenomena beyond the Standard Model of particle physics. The observation of a Higgs-like boson has been reported last year by ATLAS and CMS, the general-purpose experiments operating at the LHC. Current research is focused on the characterisation of the newly discovered boson, and on the search for new phenomena. Researchers at LPSC in Grenoble are leading the search for one type of new phenomena in ATLAS, namely the search for additional spatial dimensions which are likely to manifest themselves in LHC collisions. Given the rich multitude of physics studies proceeding in parallel in the ATLAS collaboration, one of the limiting factors in the timely analysis of ATLAS data is the availability of computing resources for physics analysis. Another LPSC team suffers from a similar limitation. This team is leading the ultimate precision measurement based on the data from the D0 experiment: the precise measurement of the W boson mass, which yields an indirect constraint on the mass of the Higgs boson and completes the direct search at the LHC. A relative precision of 10^-4 in the measurement of the W boson mass is needed to help elucidate the nature of the newly discovered Higgs-like particle. Such a measurement requires simulations of unprecedented precision, and therefore considerable computing power. The limitation in available computing power becomes problematic in the months that precede the international conferences where major results are released. The sharing of resources between different scientific fields, like the ones discussed in this article, constitutes a valuable synergy, because the spikes in need for computing resources are uncorrelated in time between different fields (HPC and HEP). The results of our collaboration between fields manifest themselves in the timely delivery of HEP results that had been eagerly awaited both by the particle physics community and by the general public.
Speaker: Dr Catherine Biscarat (LPSC/IN2P3/CNRS France)
• 218
System level traffic shaping in diskservers with heterogeneous protocols
Disk access and tape migrations compete for network bandwidth in CASTOR’s disk servers, over various protocols: RFIO, Xroot, root and GridFTP. As there are a limited number of tape drives, it is important be keep them busy all the time, at their nominal speed. With potentially 100s of user read streams per server, the bandwidth for the tape migrations has to be guaranteed to a controlled level, and not the default fair share the system gives by default. Xroot provides a prioritization mechanism , but using it implies moving exclusively to the Xroot protocol, which is not possible in short to mid-term time frame, as users are equally using all. The greatest commonality of all those protocols is not more that usage of TCP/IP. We investigated the Linux kernel traffic shaper to control TCP/IP bandwidth. The performance and limitations of the traffic shaper have been understood in test environment, and satisfactory working point has been found for production. Notably, TCP offload engines' negative impact on traffic shaping, and the limitations of the length of the traffic shaping rules were discovered and measured. A suitable working point has been found and the traffic shaping is now successfully deployed in the CASTOR production systems at CERN. This system level approach could be transposed easily to other environments.
Speaker: Daniele Francesco Kruse (CERN)
• 219
T2K-ND280 Computing Model
ND280 is the off-axis near detector for the T2K neutrino experiment. ND280 is a sophisticated, multiple sub-system detector designed to characterize the T2K neutrino beam and measure neutrino cross-sections. We have developed a complicated system for processing and simulating the ND280 data, using computing resources from North America, Europe and Japan. The first key challenge has been dealing with very different computing infrastructure in different continents; a second challenge has been dealing with the relatively large ~1PB ND280 MC dataset. We will describe the software, data storage and data distribution solutions developed to meet these challenges. We will also briefly discuss the database infrastructure developed to support ND280.
Speaker: Thomas Lindner (T)
• 220
Testing and Open Source installation and server provisioning tool for the INFN-CNAF Tier1 Storage system
In large computing centers, such as the INFN-CNAF Tier1, is essential to be able to set all the machines, depending on use, in an automated way. For several years at the Tier1 has been used Quattor, a server provisioning tool, which is currently used in production. Nevertheless we have recently started a comparison study involving other tools able to provide specific server installation and configuration features and also to offer a proper full customizable solution as an alternative to Quattor. Our choice at the moment fell on integration between two well-known tools: Cobbler for the installation phase and Puppet for the server provisioning and management operation. The tool should provide the following properties in order to replicate and gradually improve the actual system features: 1) Implement a system check for storage specific constrain such as kernel modules black list at boot time to avoid undesired SAN access during disk partitioning. 2) A simple and effective mechanism for kernel upgrade and downgrade. 3) The ability of setting package provider using yum, rpm or apt. 4) Easy to use Virtual Machine installation support including bonding and specific Ethernet configuration. 5) Scalability for managing thousands of nodes and parallel installation. This paper describes the results of the comparison and the experiments carried to verify the above requirements and if the new system is suitable for INFN-T1 storage system will be also described in details.
Speaker: michele pezzi (Infn-cnaf)
• 221
Testnodes – a Lightweight Node-Testing Infrastructure
A key aspect of ensuring optimum cluster reliability and productivity lies in keeping worker nodes in a healthy state. Testnodes is a lightweight node testing solution developed at Liverpool. While Nagios has been used locally for general monitoring of hosts and services, Testnodes is optimised to answer one question: is there any reason this node should not be accepting jobs? This tight focus enables Testnodes to inspect nodes frequently with minimal impact and provide a comprehensive and easily extended check with each inspection. On the server side, Testnodes, implemented in python, interoperates with the Torque batch server to control the nodes production status. Testnodes remotely and in parallel executes client-side test scripts and processes the return codes and output, adjusting the node's online/offline status accordingly to preserve the integrity of the overall batch system. Testnodes reports via log, email and Nagios, allowing a quick overview of node status to be reviewed and specific node issues to be identified and resolved quickly. This presentation will cover testnodes design and implementation, together with the results of its use in production at Liverpool, and future development plans.
Speaker: Robert Fay (University of Liverpool)
• 222
The Abstract geometry Modeling Language (AgML): Experience and Road map toward eRHIC
The STAR experiment has adopted an Abstract Geometry Modeling Language (AgML) as the primary description of our geometry model. AgML establishes a level of abstraction, decoupling the definition of the detector from the software libraries used to create the concrete geometry model. Thus, AgML allows us to support both our legacy GEANT3 simulation application and our ROOT/TGeo based reconstruction software from a single source, which is demonstrably self-consistent. While AgML was developed primarily as a tool to migrate away from our legacy FORtran-era geometry codes, it also provides a rich syntax geared towards the rapid development of detector models. AgML has been successfully employed by users to quickly develop and integrate the descriptions of several new detectors in the RHIC/STAR experiment including the Forward GEM Tracker (FGT) and Heavy Flavor Tracker (HFT) upgrades installed in STAR for the 2012 and 2013 runs. AgML has furthermore been heavily utilized to study future upgrades to the STAR detector as it prepares for the eRHIC era. With its track record of practical use in a live experiment in mind, we present the status, lessons learned and future of the AgML language as well as our experience in bringing the code into our production and development environments. We will discuss the path toward eRHIC and pushing the current model to accommodate for detector miss-alignment and high precision physics.
Speaker: Jason Webb (Brookhaven National Lab)
• 223
The ALICE DAQ infoLogger
ALICE (A Large Ion Collider Experiment) is a heavy-ion detector studying the physics of strongly interacting matter and the quark-gluon plasma at the CERN LHC (Large Hadron Collider). The ALICE DAQ (Data Acquisition System) is based on a large farm of commodity hardware consisting of more than 600 devices (Linux PCs, storage, network switches). The DAQ reads the data transferred from the detectors through 500 dedicated optical links at an aggregated and sustained rate of up to 10 Gigabytes per second and stores at up to 2.5 Gigabytes per second. The infoLogger is the log system which collects centrally the messages issued by the thousands of processes running on the DAQ machines. It allows to report errors on the fly, and to keep a trace of runtime execution for later investigation. More than 500000 messages are stored every day in a MySQL database, in a structured table keeping track for each message of 16 indexing fields (e.g. time, host, user, ...). The total amount of logs for 2012 exceeds 75GB of data and 150 million rows. We present in this paper the architecture and implementation of this distributed logging system, consisting of a client programming API, local data collector processes, a central server, and interactive human interfaces. We review the operational experience during the 2012 run, in particular the actions taken to ensure shifters receive manageable and relevant content from the main log stream. Finally, we present the data mining and reporting tools developed to analyze and make best use of this vast amount of information.
• 224
The ALICE Data Quality Monitoring: qualitative and quantitative review of 3 years of operations
ALICE (A Large Ion Collider Experiment) is a detector designed to study the physics of strongly interacting matter and the quark-gluon plasma produced in heavy-ion collisions at the CERN Large Hadron Collider (LHC). Due to the complexity of ALICE in terms of number of detectors and performance requirements, Data Quality Monitoring (DQM) plays an essential role in providing an online feedback on the data being recorded. It intends to provide shifters with precise and complete information to quickly identify and overcome problems, and as a consequence to ensure acquisition of high quality data. This paper presents a review of the DQM operations during the first three years of data taking from a quantitative and qualitative point of view. We will start by presenting the DQM software and tools before moving on to the various analyses carried out and the underlying physics. An overview of the produced monitoring quantities will be given, presenting the diversity of usage and the flexibility of the DQM system. Well-prepared shifters and experts, in addition to a precise organisation, were required to ensure smooth and successful operations. The description of the measures taken to ensure both aspects and an account of the DQM shifter’s job are followed by a summary of the evolution of the system. We will then give a quantitative review of the final setup of the system used during the whole year 2012. We conclude the paper with real world cases when the DQM proved to be very valuable, scalable and efficient and with the plans for the coming years.
Speaker: Mr Barthelemy Von Haller (CERN)
• 225
The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
Modern scientific experiments collect vast amounts of data that must be cataloged to meet multiple use cases and search criteria. In particular, high-energy physics experiments currently in operation produce several billion events per year. A database with the references to the files including each event in every stage of processing is necessary in order to retrieve the selected events from data storage systems. The ATLAS EventIndex project is studying the best way to store the necessary information using modern data storage technologies (Hadoop, HBase etc.) that allow saving in memory key-value pairs and select the best tools to support this application from the point of view of performance, robustness and ease of use. At the end of this development, a new technology that is inherently independent of the type of data that are stored in the database -- and therefore directly applicable to all scientific experiments with large amounts of data -- will be available and demonstrated by the example of the ATLAS experiment. This paper describes the initial design and performance tests and the project evolution towards deployment and operation in 2014.
Speaker: Dr Dario Barberis (Università e INFN Genova (IT))
• 226
The ATLAS Muon Trigger
CERN’s Large Hadron Collider (LHC) is the highest energy proton-proton collider, providing also the highest instantaneous luminosity as a hadron collider. Bunch crossings occurred every 50 ns in 2012 runs. Amongst of which the online event selection system should reduce the event recording rate down to a few 100 Hz, while events are in a harsh condition with many overlapping proton-proton collisions occurring in a same bunch crossing. Muons often provide an important and clear signature of physics processes that are searched for, for instance as in the discovery of Higgs particle in year 2012. The ATLAS experiment deploys a three-levels processing scheme at online. The level-1 muon trigger system gets its input from fast muon trigger detectors. Fast sector logic boards select muon candidates, which are passed via an interface board to the central trigger processor and then to the High Level Trigger (HLT). The muon HLT is purely software based and encompasses a level-2 (L2) trigger followed by an event filter (EF) for a staged trigger approach. It has access to the data of the precision muon detectors and other detector elements to refine the muon hypothesis. Trigger-specific algorithms were developed and are used for the L2 to increase processing speed for instance by making use of look-up tables and simpler algorithms, while the EF muon triggers mostly benefit from offline reconstruction software to obtain most precise determination of the track parameters. There are two algorithms with different approaches, namely inside-out and outside-in tracking, which was used in trigger with conditional-OR to obtain maximum efficiency with least processing time. This presentation gives a full overview of the ATLAS muon trigger system, summarizes the 3 years running experiences and reports about online performances for instance processing time and trigger rates as well as trigger efficiency, resolution, and other general performance.
Speaker: Martin Woudstra (University of Manchester (GB))
• 227
The CC1 system – a solution for private cloud computing.
In the multidisciplinary institutes the traditional way of computations is highly ineffective. A computer cluster dedicated to a single research group is typically exploited at a rather low level. The private cloud model enables various groups to share computing resources. It can boost the efficiency of the infrastructure usage by a large factor and at the same time reduce maintenance costs. The complete cloud computing system has been developed in the Institute of Nuclear Physics PAN, Cracow. It is based on the Python programming language and a low level virtualization toolkit – libvirt. The CC1 system provides resources within the Infrastructure as a Service (IaaS) model. The main features of the system are the following: • custom web-based user interface, • automatic creation of virtual clusters equipped with a preconfigured batch system, • groups of users with the ability to share resources, • permanent virtual storage volumes that can be mounted to a VM, • distributed structure – a federation of clusters running as a uniform cloud, • quota for user resources, • monitoring and accounting. Emphasis was put on the simplicity of user access, administrative tasks and installation procedure. The self-service access to the system is provided via the intuitive Web interface. The administration module contains a rich set of tools for user management and system configuration. Particular attention was paid to the preparation of the automatic installation procedure based on a standard package management system of Linux distributions. This way the system can be setup quickly within one hour and operated without the need of a deep understanding of the underlying cloud computing technology. One of the crucial features is easy creation of clusters of virtual machines equipped with a preconfigured batch system. This way the intensive calculations can be performed on demand without the need of time-consuming configuration of clusters. When finished, the resources can be released and made accessible to other users. The stable system was put into operation at the beginning of 2012 and was extensively used for calculations by various research teams, in particular by HEP groups. The CC1 software is distributed under the Apache License 2.0. The web site of the project is located at http://cc1.ifj.edu.pl.
Speaker: Rafal Zbigniew Grzymkowski (P)
• 228
The CMS Data Quality Monitoring software: experience and future improvements
The Data Quality Monitoring (DQM) Software proved to be a central tool in the CMS experiment. Its flexibility allowed its integration in several environments: Online, for real-time detector monitoring; Offline, for the final, fine-grained Data Certification; Release-Validation, to constantly validate our reconstruction software; in Monte Carlo productions. The central tool to deliver Data Quality information is a web site for browsing data quality histograms (DQMGUI). In this presentation the usage of the DQM Software in the different environments and its integration in the CMS Reconstruction Software Framework (CMSSW) and in all production workflows are presented. The main technical challenges and the adopted solutions to them will be also discussed with emphasis on functionality, long-term robustness and performance. Finally the experience in operating the DQM systems over the past years will be reported.
Speaker: Dr Federico De Guio (CERN)
• 229
The Design and Performance of the ATLAS jet trigger
The ATLAS jet trigger is an important element of the event selection process, providing data samples for studies of Standard Model physics and searches for new physics at the LHC. The ATLAS jet trigger system has undergone substantial modifications over the past few years of LHC operations, as experience developed with triggering in a high luminosity and high event pileup environment. In particular, the region-of-interest (ROI) based strategy has been replaced by a full scan of the calorimeter data at the third trigger level, and by a full scan of the level-1 trigger input at level-2 for some specific trigger chains. Hadronic calibration and cleaning techniques are applied in order to provide improved performance and increased stability in high luminosity data taking conditions. In this presentation we discuss the implementation and operational aspects of the ATLAS jet trigger during 2011 and 2012 data taking periods at the LHC.
Speaker: Shima Shimizu (Kobe University (JP))
• 230
The Design and Realization of the Distributed Data Sharing System of the Detector Control System of the Daya Bay Neutrino Experiment
The Daya Bay reactor neutrino experiment is designed to determine precisely the neutrino mixing angle θ13 with the sensitivity better than 0.01 in the parameter sin22θ13 at the 90% confidence level. To achieve this goal, the collaboration has built eight functionally identical antineutrino detectors. The detectors are immersed in water pools that provide active and passive shielding against backgrounds. The experiment has been taking data for almost 1.5 years and making steady progress. Eight antineutrino detectors are taking data now in 3 experimental halls. And the first results have already been released. The detector control and monitoring system(DCS) was developed to support the running experiment. And according to the difference of different hardware systems, such as high voltage crates, front end electronic crates, water system, gas system, low voltage crates, temperature and Humidity of the environment system etc., different data acquisition(DAQ) modules are developed. A global control system is developed to monitor and control the entire running status of the whole experiment. Sharing data from the subsystems are most important for both equipment monitoring and data analysis. This paper will present the design and the realization of a distributed data sharing system used in the Detector Control System of the experiment. The interface of the embedded DAQ of the sensors and the communication logic will be show in details. The integration of the developed remote control framework will be introduced as well. Key words：Daya Bay, Neutrino, Detector Control System, Embedded System, Data Acquisition, Distributed Sharing Data System, Remote Control
Speaker: Mei YE (IHEP)
• 231
The DMLite Rucio Plugin: ATLAS data in a filesystem
Rucio is the next-generation data management system supporting ATLAS physics workflows in the coming decade. Historically, clients interacted with the data management system via specialised tools, but in Rucio additional methods are provided. To support filesystem-like interaction with all ATLAS data a plugin to the DMLite software stack has been developed. It is possible to mount Rucio as a filesystem, and execute regular filesystem operations in a POSIX fashion. This is exposed via various protocols like HTTP/WebDAV or NFS, which then removes any dependency on Rucio for client software. The main challenge for this work is the mapping of the set-like ATLAS namespace into a hierarchical filesystem, whilst preserving the high performance features of the former. This includes listing and searching for data, creation of files, datasets and containers, and the aggregation of existing data - all within POSIX directories with potentially millions of entries. This contribution will detail the design and implementation of the plugin, and demonstrate how physicist can interact with ATLAS data via commonly available and standard tools. Furthermore, an evaluation of the performance characteristics is given, to show that this approach can scale to the requirements of ATLAS physics analysis.
Speaker: Mario Lassnig (CERN)
• 232
The GridKa Tier-1 Computing Center within the ALICE Grid Framework
The GridKa computing center, hosted by Steinbuch Centre for Computing at the Karlsruhe Institute for Technology (KIT) in Germany, is serving as the largest Tier-1 center used by the ALICE collaboration at the LHC. In 2013, GridKa provides 30k HEPSEPC06, 2.7 PB of disk space, and 5.25 PB of tape storage to ALICE. The 10Gbit/s network connections from GridKa to CERN, several Tier-1 centers and the general purpose network are used by ALICE intensively. In 2012 a total amount of ∼1 PB was transferred to and from GridKa. As Grid framework, AliEn (ALICE Environment) is being used to access the resources, and various monitoring tools including the MonALISA (MONitoring Agent using a Large Integrated Services Architecture) are always running to alert in case of any problem. GridKa on-call engineers provide 24/7 support to guarantee minimal loss of availability of computing and storage resources in case of hardware or software problems. We introduce the GridKa Tier-1 center from the viewpoint of ALICE services.
Speaker: Dr WooJin Park (KIT)
• 233
The ILC Detailed Baseline Design Exercise: Simulating the Physics capabilities and detector performance of the Silicon Detector using the Grid
The International Linear Collider (ILC) physics and detector community recently completed an exercise to demonstrate the physics capabilities of detector concepts. The Detailed Baseline Design (DBD) involved the generation, simulation, reconstruction and analysis of large samples of Monte Carlo datasets. The detector simulations utilized extremely detailed Geant4 implementations of engineered detector elements. The datasets incorporated the full set of Standard Model physics backgraounds, as well as machine backgrounds, all overlaid with the correct time structure of overlapping events. We describe how this exercise was undertaken using Grid computing and storage elements, as well as the submission, bookkeeping and cataloging infrastructure developed to support this international endeavor.
Speaker: Norman Anthony Graf (SLAC National Accelerator Laboratory (US))
• 234
The keys to CERN conference rooms - Managing local collaboration facilities in large organisations
For a long time HEP has been ahead of the curve in its usage of remote collaboration tools, like videoconference and webcast, while the local CERN collaboration facilities were somewhat behind the expected quality standards for various reasons. This time is now over with the creation by the CERN IT department in 2012 of an integrated conference room service which provides guidance and installation services for new rooms (either equipped for video-conference or not), as well as maintenance and local support. Managing now nearly half of the 250 meeting rooms available on the CERN sites, this service has been built to cope with the management of all CERN rooms with limited human resources. This has been made possible by the intensive use of professional software to manage and monitor all the room equipment, maintenance and activity. This paper will focus on presenting these packages, either off-the-shelf commercial products (asset and maintenance management tool, remote audiovisual equipment monitoring systems, local automation devices, new generation touch screen interfaces for interacting with the room) when available or locally developed integration and operational layers (generic audiovisual control and monitoring framework) and how they help overcoming the challenges presented by such a service. The aim is to minimise local human interventions while preserving the highest service quality and placing the end user back to the center of this collaboration platform.
Speaker: Thomas Baron (CERN)
• 235
The Legnaro-Padova distributed Tier-2: challenges and results
Speaker: Mr Massimo Sgaravatto (INFN Padova)
• 236
The LHCb Silicon Tracker - Control system specific tools and challenges
The experiment control system of the LHCb experiment is continuously evolving and improving. The guidelines and structure initially defined are kept, and more common tools are made available to all sub-detectors. Although the main system control is mostly integrated and actions are executed in common for the whole LHCb experiment, there is some degree of freedom for each sub-system to implement the control system using these tools or by creating new ones. The implementation of the LHCb Silicon Tracker control system was extremely disorganized and with little documentation. This was due to either lack of time and manpower, and/or to limited experience and specifications. Despite this, the Silicon Tracker control system has behaved well during the first LHC run. It has continuously evolved since the start of operation and been adapted to the needs of operators with very different degrees of expertise. However, improvements and corrections have been made on a best effort basis due to time constraints placed by the need to have a fully operating detector. The system will be transformed by an ambitious rework of the code which will take place in the first months of the LS1. Performance issues with the regard to configuration and monitoring will be addressed, and the maintainability and use of the system will be improved. This work describes the main tools which have been created specifically for the Silicon Tracker operation including the safety tree and the automated safety actions which are implemented to prevent damage to the detector electronics. In addition, the automation of recurrent tasks related mostly to data mining, error detection and recovery will be discussed. It describes also the new features and improvements that will be introduced after the code rework during the LS1, and a summary of the main tasks needed to accomplish this.
Speaker: Sandra Saornil Gamarra (Universitaet Zuerich (CH))
• 237
The LHCb Trigger Architecture beyond LS1
The LHCb experiment is a spectrometer dedicated to the study of heavy flavor at the LHC. The rate of proton-proton collisions at the LHC is 15 MHz, but resource limitations mean that only 5 kHz can be written to storage for offline analytsis. For this reason the LHCb data acquisition system -- trigger -- plays a key role in selecting signal events and rejecting background. In contrast to previous experiments at hadron colliders like for example CDF or D0, the bulk of the LHCb trigger is implemented in software and deployed on a farm of 20k parallel processing nodes. This system, called the High Level Trigger (HLT) is responsible for reducing the rate from the maximum at which the detector can be read out, 1.1 MHz, to the 5 kHz which can be processed offline,and has 20 ms in which to process and accept/reject each event. In order to minimize systematic uncertainties, the HLT was designed from the outset to reuse the offline reconstruction and selection code. During the long shutdown it is proposed to extend this principle and enable the HLT to access offline quality detector alignment and calibration, by buffering events on the HLT nodes for long enough for this alignment and calibration to be performed and fed into the HLT algorithms. This will in turn allow the HLT selections to be tightened and hence will significantly increase the purity of the data being written for offline analysis. This contribution describes the proposed architecture of the HLT beyond LS1 and the technical challenges of implementing a real-time detector alignment and calibration in the LHC environment.
Speaker: Sebastian Neubert (CERN)
• 238
The MICE Run Control System
The Muon Ionization Cooling Experiment (MICE) is a demonstration experiment to prove the feasibility of cooling a beam of muons for use in a Neutrino Factory and/or Muon Collider. The MICE cooling channel is a section of a modified Study II cooling channel which will provide a 10% reduction in beam emittance. In order to ensure a reliable measurement, MICE will measure the beam emittance before and after the cooling channel at the level of 1%, or an absolute measurement of 0.001. This renders MICE a precision experiment which requires strict controls and monitoring of all experimental parameters in order to control systematic errors. The MICE Controls and Monitoring system is based on EPICS and integrates with the DAQ, Data monitoring systems, and a configuration database. The new MICE Run Control has been developed for to ensure proper sequencing of equipment and use of system resources to protect data quality. A description of this system, its implementation, and performance during recent muon beam data collection will be discussed.
Speaker: Pierrick Hanlet (Illinois Institute of Technology)
• 239
The performance of the ATLAS tau trigger in 8 TeV collisions and novel upgrade developments for 14TeV
The LHC is the world's highest energy and luminosity proton-proton (p-p) collider. During 2012 luminosities neared 10^34 cm-2 s-1, with bunch crossings occurring every 50 ns. The online event selection system of the ATLAS detector must reduce the event recording rate to only a few hundred Hz and, at the same time, selecting events considered interesting. This presentation will specifically cover the online selection of events with tau leptons decaying to hadronic final states. The "hadronic tau trigger" has been operated successfully since 2009. Tau leptons provide a unique window for understanding physics at the Tera scale. They are the most crucial signature for measuring the fermion couplings of the Higgs boson, an important next step in identifying the recently discovered Higgs-like boson. Many theories also predict new particles beyond the Standard Model that have large couplings to tau leptons. The first step in their identification is the online event trigger. ATLAS employs a 3-level trigger scheme. The level-1 (L1) hadronic tau trigger system measures energy deposited in electromagnetic (EM) and hadronic (HAD) calorimeter trigger towers to select taus based on energy in core and isolation regions. The remaining two levels, are software-based. At the second level, further identification is done within regions of interest identified by L1, taking into account track and calorimeter information with dedicated fast algorithms. At the third and final level, the event filter algorithms closely match the algorithms used for offline reconstruction. Taus are often mimicked by similarly behaved quark jets, which have large rates. The rate at which events must be selected severely limits the complexity of reconstruction algorithms, compounded by the high probability of overlap (pileup) between bunch crossings. ATLAS has a program dedicated to the continued improvement of the tau trigger to address these problems, including: Multivariate identification algorithms using high granularity calorimeter and tracking information, fast track reconstruction methods for accurate determination of impact parameters reducing pile up, and new topological criteria using multiple event features. The latter involves simultaneous triggering on taus with muons, electrons, and missing energy. These developments have given ATLAS the potential to explore a large program of tau physics analysis. This presentation gives a full overview of the ATLAS tau trigger system, summarising the running experience over the past 3 years. We demonstrate that the ATLAS tau trigger performed remarkably well throughout its operation, and discuss computational innovations in 2012. Results of the performance of the tau trigger from the full 22 fb-1 2012 p-p data taking period will be shown, including measurements of the trigger efficiency using Z->tau tau and W->tau nu events and the application to searches for tau tau resonances, such as the Higgs boson. We also outline the upgrade plan to 2015 for 14(13) TeV LHC proton-proton collisions, which includes the use of a novel associative memory trigger for track finding throughout the full detector.
Speaker: Joern Mahlstedt (NIKHEF (NL))
• 240
The Reconstruction Software for the Muon Ionisation Cooling Experiment Trackers
The international Muon Ionisation Cooling Experiment (MICE) is designed to demonstrate the principle of muon ionisation cooling for the first time, for application to a future Neutrino Factory or Muon Collider. In order to measure the change in beam emittance, MICE is equipped with a pair of high precision scintillating fibre trackers. The trackers are required to measure a 10% change in beam emittance to 1% accuracy (giving an overall emittance measurement of 0.1%). This paper describes the tracker reconstruction software, as a part of the overall MICE software framework, MAUS. The process of producing fibre digits is described for both the GEANT4 based Monte Carlo case, and for real data. Fibre clustering is described, proceeding to the formation of spacepoints, which are then associated with particle tracks using pattern recognition algorithms. Finally a full custom Kalman track fit is performed, to account for energy loss and multiple scattering. Exemplar results are shown for both Monte Carlo and cosmic ray data.
Speaker: Pierrick Hanlet (Illinois Institute of Technology)
• 241
The Repack Challenge
Physics data stored in CERN tapes is quickly reaching the 100 PB milestone. Tape is an ever-changing technology that is still following Moore's law in terms of capacity. This means we can store every year more and more data in the same amount of tapes. However this doesn't come for free: the first obvious cost is the new higher capacity media. The second less known cost is related to moving the data from the old tapes to the new ones. This activity is what we call repack. Repack is vital for any large tape user: without it, one would have to buy more tape libraries and more floor space and, eventually, data on old non supported tapes would become unreadable and be lost forever. The challenge is not an easy one. First, to make sure we won't need any more tape slots in the near future, we will have to repack 120 PB from 2014 to 2015, this in turn means that we will have to be able to cope with peaks of 3.5 GB/s smoothly. Secondly, all the repack activities will have to run concurrently and in harmony with the existing experiment tape activities. Making sure that this works out seamlessly implies careful planning of the resources and the various policies for sharing them fairly and conveniently. Our previous setup allowed for an average repack performance of only 360 MB/s. Our needs demand this figure increase tenfold by 2013. To tackle this problem we needed to fully exploit the speed and throughput of our modern tape drives. This involved careful dimensioning and configuration of the disk arrays (the middle step between an old source tape and a new higher capacity destination tape) and all the links between them and the tape servers (the machines responsible for managing the tape drives). We also planned a precise schedule and provided a visual monitoring tool to check the progress over time. The new repack setup we deployed brought an average 80% increase in the throughput of tape drives, allowing them to perform closer to their design specifications. This improvement in turn meant a 40% decrease in the number of drives needed to achieve the 3.5 GB/s goal. CERN is facing its largest data migration challenge yet. By restructuring the repack infrastructure we allowed the vital repack and LHC experiments activities to coexist without the need for new expensive tape drives.
Speaker: Daniele Francesco Kruse (CERN)
• 242
The role of micro size computing clusters for small physics groups
A small physics group (3-15 persons) might use a number of computing facilities for the analysis/simulation, developing/testing, teaching. It is discussed different types of computing facilities: collaboration computing facilities, group local computing cluster (including colocation), cloud computing. The author discuss the growing variety of different computing options for small groups and does emphasize the role of the group owned computing cluster of micro size.
Speaker: Mr Andrey SHEVEL (Petersburg Nuclear Physics Institute)
• 243
The Role of the Collaboratory in enabling Large-Scale Identity Management for HEP
As HEP collaborations grow in size (10 years ago, BaBar was 600 scientists; now, both CMS and ATLAS are on the order of 3000 scientists), the collaboratory has become a key factor in allowing identity management (IdM), once confined to individual sites, to scale with the number of members, number of organizations, and the complexity of the science collaborations. Over the past two decades (at least) there has been a great deal of applied research and success in implementing collaboratories, but there has also been a great deal of controversy and variety of implementations in the community. A common implementation, or even a model for contrasting different implementations, does not yet exist. This lack of common approach makes collaboration between existing collaboratories and establishment of new collaboratories a challenge. The eXtreme Scale Identity Management (XSIM) project is addressing this short-coming by defining a model for IdM that captures existing and future collaboratory implementations. XSIM is first capturing an understanding of the trust relationships in today’s scientific collaborations and their resource providers and analyzing how the trade-offs between the policies and trust relationships affect current IdM models. This understanding is being developed through a review of existing literature and one-on-one interviews with dozens of members of the communities involved to fully understand the motivations for the decisions and the lessons learned. Building on this research, XSIM is proposing a model for identity management that describes the core trust relationships between HEP collaborations and resource providers, and the different choices for those relationships, both in terms of levels and types of trust, and implementation. The model must be sufficiently comprehensive to encompass the reality of the existing IdM architectures; be understandable and useful to future collaboratory developers who are not IdM experts; relate well to efforts in the HEP community; and be accepted by resource providers. Developing such a model will give the community a language in which to express differences in identity management solutions, and easily communicate and understand the impacts of changes in the trust relationships involved with different choices. This in turn will expedite understanding and establishment of new collaborations. The presentation will provide a summary of the interviews and literature, the resulting analysis, and a model that captures the core trust relationships, especially those relating to IdM.
Speaker: Bob Cowles (BrightLite Information Security)
• 244
The STAR "Plug and Play" Event Generator Framework
The STAR experiment pursues a broad range of physics topics in pp, pA and AA collisions produced by the Relativistic Heavy Ion Collider (RHIC). Such a diverse experimental program demands a simulation framework capable of supporting an equally diverse set of event generators, and a flexible event record capable of storing the (common) particle-wise and (varied) event-wise information provided by the external generators. With planning underway for the next round of upgrades to exploit ep and eA collisions from the electron-ion collider (or eRHIC), these demands on the simulation infrastructure will only increase and requires a versatile framework. STAR has developed a new event-generator framework based on the best practices in the community (a survey of existing approach had been made and the “best of all worlds” kept in mind in our design). It provides a common set of base classes which establish the interface between event generators and the simulation and handles most of the bookkeeping associated with a simulation run. This streamlines the process of integrating and configuring an event generator within our software chain. Developers implement two classes: the interface for their event generator, and their event record. They only need to loop over all particles in their event and push them out into the event record. The framework is responsible for vertex assignment, stacking the particles out for simulation, and event persistency. Events from multiple generators can be merged together seamlessly, with an event record which is capable of tracing each particle back to its parent generator. We present our work and approach in detail and illustrate its usefulness by providing examples of event generators implemented within the STAR framework covering for very diverse physics topics. We will also discuss support for event filtering, allowing users to prune the event record of particles which are outside of our acceptance, and/or abort events prior to the more computationally expensive digitization and reconstruction phases. Event filtering has been supported in the previous framework and showed to save enormous amount of resources – the approach within the new framework is a generalization of filtering.
Speaker: Jason Webb (Brookhaven National Lab)
• 245
The upgrade and re-validation of the Compact Muon Solenoid Electromagnetic Calorimeter Control System
The Electromagnetic Calorimeter (ECAL) is one of the sub-detectors of the Compact Muon Solenoid (CMS) experiment of the Large Hadron Collider (LHC) at CERN. The Detector Control System (DCS) that has been developed and implemented for the CMS ECAL was deployed in accordance with the LHC schedule and has been supporting the detector data-taking since LHC physics runs started in 2009. During these years, the control system has been regularly adapted according to operational experience and new requirements, always respecting the constraints imposed on significant changes to a running system. Several hardware and software upgrades and system extensions were therefore deferred to the first LHC Long Shutdown (LS1). This paper presents the main architectural differences between the system that supported the CMS ECAL during its first years and the new design for the coming physics runs after LS1. Details on the upgrade planning, including the certification methods performed in the CMS ECAL DCS laboratory facilities, reports on the implementation progress and the expectations for the post-LS1 system are highlighted.
Speaker: Oliver Holme (ETH Zurich, Switzerland)
• 246
Tier-1 experience with provisioning virtualized worker nodes on demand
While migration from the grid to the cloud has been gaining increasing momentum in recent times, WLCG sites are currently still expected to accept grid job submission, and this is likely to continue for the foreseeable future. Furthermore, sites which support multiple experiments may need to provide both cloud and grid-based access to resources for some time, as not all experiments may be ready to move to the cloud at the same time. In order to make optimal use of resources, a site with a traditional batch system as well as a cloud resource may want to make opportunistic use of their cloud at times when there are idle jobs and all worker nodes in the batch system are busy, by extending the batch system into the cloud. We present two implementations, one based on HTCondor and the other based on SLURM as a batch system, in which virtualized worker nodes are provisioned on demand using a StratusLab cloud.
Speaker: Andrew David Lahiff (STFC - Science & Technology Facilities Council (GB))
• 247
Tier-1 Site Evolution in Response to Experiment Requirements
LHC experiments are moving away from a traditional HSM solution for Tier 1's in order to separate long term tape archival from disk only access, using the tape as a true archive (write once, read rarely). In this poster we present two methods by which this is being achieved at two distinct sites, ASGC and RAL, which have approached this change in very different ways.
Speaker: Shaun De Witt (STFC - Science & Technology Facilities Council (GB))
• 248
Tier-2 Optimisation for Computational Density/Diversity and Big Data
As the number of cores on chip continues to trend upwards and new CPU architectures emerge, increasing CPU density and diversity presents multiple challenges to site administrators. These include scheduling for massively multi-core systems (potentially including GPU (integrated and dedicated) and many integrated core (MIC)) to ensure a balanced throughput of jobs while preserving overall cluster throughput, in addition to meeting data demands as both dataset sizes increase and as the rate of demand scales with increased computational power, along with the practical management of these resources. In this report, we evaluate the current tools and technologies available to manage these emerging requirements, including cluster software (batch, scheduling), resource management solutions (VMs, clouds, containers) and infrastructure (hardware and network specification and optimisation, software services) in order to assess what options are available at the present time, the limits therein, and to identify issues remaining to be addressed.
Speaker: Robert Fay (University of Liverpool)
• 249
Time structure analysis of the LHCb Online network
The LHCb Online Network is a real time high performance network, in which 350 data sources send data over a Gigabit Ethernet LAN to more than 1500 receiving nodes. The aggregated throughput of the application, called Event Building, is more than 60 GB/s. The protocol employed by LHCb makes the sending nodes transmit simultaneously portions of events to one receiving node at a time, which is selected using a credit-token scheme. The resulting traffic is very bursty and sensitive to irregularities in the temporal distribution of packet-bursts to the same destination or region of the network. In order to study the relevant properties of such a dataflow, a non-disruptive monitoring setup based on a networking capable FPGA (NetFPGA) has been deployed. The NetFPGA allows order of hundred nano-second precise time-stamping of packets. We study in detail the timing structure of the Event Building communication, and we identify potential effects of micro-bursts like buffer packet drops or jitter.
Speaker: Daniel Hugo Campora Perez (CERN)
• 250
Toward a petabyte-scale AFS service at CERN
AFS is a mature and reliable storage service at CERN, having worked for more than 20 years as the provider of Linux home directories and application areas. Recently, our AFS service has been growing at unprecedented rates (300% in the past year), thanks to innovations in both the hardware and software components of our file servers. This work will present how AFS is used at CERN and how the service offering is evolving with the increasing storage needs of its local and remote user communities. In particular, we will demonstrate the usage patterns for home directories, workspaces and project spaces, as well as show the daily work which is required to rebalance data and maintaining stability and performance. Finally, we will highlight some recent changes and optimisations made to the AFS Service, thereby revealing how AFS can possibly operate at all while being subjected to frequent — almost DDOS-like — attacks from its users.
Speakers: Dr Daniel van der Ster (CERN), Dr Jakub Moscicki (CERN)
• 251
Toward the Cloud Storage Interface of the INFN CNAF Tier-1 Mass Storage System
Speakers: Daniele Gregori (Istituto Nazionale di Fisica Nucleare (INFN)), Luca dell'Agnello (INFN-CNAF), Pier Paolo Ricci (INFN CNAF), Tommaso Boccali (Sezione di Pisa (IT)), Dr Vincenzo Vagnoni (INFN Bologna), Dr Vladimir Sapunenko (INFN)
• 252
Towards a centralized Grid Speedometer
Given the distributed nature of the grid and the way CPU resources are pledged and scared around the globe, VO's are facing the challenge to monitor the use of these resources. For CMS and the operation of centralized workflows the monitoring of how many production jobs are running and pending in the Glidein WMS production pools is very important. The Dashboard SSB (Site Status Board) provides a very flexible framework to collect, aggregate and visualize data. The CMS production monitoring team uses the SSB to define the metrics that have to be monitored and the alarms that have to be set. During the integration of the CMS production monitoring into the SSB, several enhancements to the core functionality of the SSB were implemented; all in a generic way, so that other VOs using the SSB can use them as well. Alongside these enhancements, there were a few changes to the core of the SSB framework from which the CMS production team was able to benefit. We will present the details of the implementation and the advantages for current and future usage of the new features in the VO agnostics Dashboard.
Speaker: Ivan Antoniev Dzhunov (University of Sofia)
• 253
Towards more stable operation of the Tokyo Tier2 center
The Tokyo Tier2 center, which is located at International Center for Elementary Particle　Physics (ICEPP) in the University of Tokyo, was established as a regional analysis center in Japan for the ATLAS experiment. The official operation with WLCG was started in 2007 after the several years development since 2002. In December 2012, we have replaced almost all hard wares as the third system upgrade to deal with analysis for further growing data of the ATLAS experiment. The number of CPU cores are increased by factor of two (9984 cores in total), and the performance of individual CPU core is improved by 14% according to the HEPSPEC06 benchmark test at 32bit compile mode. It is estimated as 17.06 per core by using Intel Xeon E5-2680 2.70GHz. Since all worker nodes are made by 16 CPU cores configuration, we deployed 624 blade servers in total. They are connected to 6.7PB of disk storage system with non-blocking 10Gbps internal network backbone by using two center network switches (NetIron MLXe-32). The disk storage is made by 102 of RAID6 disk arrays (Infortrend DS S24F-G2840-4C16DO0) and served by equivalent number of 1U file servers with 8G-FC connection to maximize the file transfer throughput per storage capacity. As of February 2013, 2560 CPU cores and 2.00PB of disk storage have already been deployed for the WLCG. Currently, the remaining non-grid resources for both CPUs and disk storages are used as dedicated resources for the data analysis by the ATLAS Japan collaborators. Since all HWs in the non-grid resources are made by same architecture with Tier2 resource, they will be able to be migrated as the Tier2 extra resource on demand of the ATLAS experiment in the future. In addition to the upgrade of computing resources, we expect the improvement of connectivity on the wide area network. Thanks to the Japanese NREN (NII), another 10Gbps trans-Pacific line from Japan to Washington will be available additionally with existing two 10Gbps lines (Tokyo to NY and Tokyo to LA). The new line will be connected to the LHCONE for the more improvement of the connectivity. In this circumstance, we are working for the further stable operation. For instance, we have newly introduced GPFS (IBM) for the non-grid disk storage, while Disk pool manager (DPM) are continued to be used as Tier2 disk storage from the previous system. Since the number of files stored in a DPM pool will be increased with increasing the total amount of data, the development of stable database configuration is one of the crucial issues as well as scalability. We have started some studies on the performance of asynchronous database replication so that we can take daily full backup. In this presentation, we would like to introduce several improvements in terms of the performances and stabilities of our new system, and also present the status of the wide area network connectivity from Japan to US and/or EU with LHCONE.
Speaker: Dr Tomoaki Nakamura (University of Tokyo (JP))
• 254
Towards Provenance and Traceability in CRISTAL for HEP
Efficient, distributed and complex software is central in the analysis of high energy physics (HEP) data. One area that has been somewhat overlooked in recent years has been the tracking of the development of the HEP software and of its use in data analyses and its evolution over time. This area of tracking analyses to provide records of actions performed, outcomes achieved and (re-)design decisions taken is an active part of computer science research known as provenance data capture and management. In recent years there has been a wealth of research conducted in the computer science community on this topic, however very little work has been done to address the requirements that have emerged from the HEP domain in the LHC era. This paper discusses a system known as CRISTAL which has been in development and active use at CERN for the past decade. CRISTAL is a mature and very stable system which was originally developed to track the construction of the ECAL element of CMS. The current usage is discussed in this paper in the context of its application at CMS. CRISTAL has also been commercialised by two external companies. The first company, M1i (Annecy France), has developed a purely BPM (Business Process Management) solution and has sold the product (Agilium) in the retail, logistics and manufacturing sectors; in the second company, Technoledge (Geneva, Switzerland) it is being applied to fuel cell production lines with a focus on provenance data capture and management, therefore demonstrating its maturity as a provenance system. CRISTAL is currently being moved towards an open source license, and is being used in several EC projects, one example of which is N4U (or neuGRID for Users) where a so-called Analysis Service is being developed to enable neuroimaging researchers to record and track their complex workflows and analyses. This Analysis Service allows for the reuse of clinical research analysis workflows by scientists proving its generic application as a tool for the management of scientific data. We are currently aiming to apply CRISTAL for the indexing of previous HEP data with our team based at CERN. We also feel that CRISTAL can be applied to aid scientists at CERN in creating their experiments through use of the N4U Analysis Service built on top of CRISTAL, allowing them to share, reuse or amend past HEP analyses. In addition to analysis reuse and sharing, CRISTAL's unique approach to provenance capture provides a means for scientists to log errors and to audit which analyses can be used in conjunction with various datasets. Consequently, CRISTAL provides an unique viewpoint for investigators to see where and more importantly why their experiments may have failed and to store their results. Some initial ideas for the use of CRISTAL in HEP are outlined in detail in this paper. Currently we are investigating the feasibility of using the N4U Analysis Service or a derivative along with CRISTAL to address the requirements of data and analysis logging and provenance capture within the HEP environment.
Speaker: Jetendr Shamdasani (University of the West of England (GB))
• 255
Transactional Aware Tape Infrastructure Monitoring System
Administrating a large-scale, multi-protocol, hierarchical tape storage infrastructure like the one at CERN, which stores around 30PB / year, requires an adequate monitoring system for quick spotting of malfunctions, easier debugging and on demand report generation. The main challenges for such system are: to cope with log format diversity and its information scattered among several log files, the need for long term information archival, the strict data consistency requirements and the group based GUI visualization. For this purpose, we have designed, developed and deployed a centralized system consisting of four independent layers: a Log Transfer layer for collecting log lines from all tape servers to a single aggregation server, a Data Mining layer for combining log data into transactional context, a Storage layer for archiving the resulting transactions and finally a Web UI layer for accessing the information. Having flexibility, extensibility and maintainability in mind, each layer is designed to work as a message broker for the next layer, providing a clean and generic interface while ensuring consistency, redundancy and ultimately fault tolerance. This system unifies information previously dispersed over several monitoring tools into a single user interface, using Splunk, which also allows us to provide information visualization based on access control lists (ACL). Since its deployment, it has been successfully used by CERN tape operators for quick overview of transactions, performance evaluation, malfunction detection and by managers for report generation. In this paper we present our design principles, problems with corresponding solutions, disaster cases and how we handle them, comparison with other solutions and future work that can be done.
Speaker: Daniele Francesco Kruse (CERN)
• 256
Upgrades for Offline Data Quality Monitoring at ATLAS
The ATLAS offline data quality monitoring infrastructure functioned successfully during the 2010-2012 run of the LHC. During the 2013-14 long shutdown, a large number of upgrades will be made in response to user needs and to take advantage of new technologies - for example, deploying richer web applications, improving dynamic visualization of data, streamlining configuration, and moving applications to a common messaging bus. Additionally consolidation and integration activities will occur. We will discuss lessons learned so far and the progress of the upgrade project, as well as associated improvements to the data reconstruction and processing chain.
Speaker: Morten Dam Joergensen (Niels Bohr Institute (DK))
• 257
Upgrading HFGFlash for Faster Simulation at Super LHC
HFGFlash is a very fast simulation of electromagnetic showers using parameterizations of the profiles in Hadronic Forward Calorimeter. HF GFlash has good agreement to Collision Data and previous Test Beam results. In addition to good agreement to Data and previous Test Beam results, HFGFlash can simulate about 10000 times faster than Geant4. We will report the latest development of HFGFlash for Faster Simulation at Super LHC.
Speaker: Rahmat Rahmat (University of Mississippi (US))
• 258
Use of VMWare for providing cloud infrastructure for the Grid
The need to maximize computing facilities whilst maintaining versatile and flexible setups leads to the need for on demand virtual machines through the use of cloud computing. GridPP is currently investigating the role that Cloud Computing, in the form of Virtual Machines, can play in supporting Particle Physics analyses. As part of this research we look at the ability of VMWare's ESXi hypervisors to provide such Virtual Machines; the advantages of such systems and their overall performance. We conclude with a contrast between the ability of VMWare and other major OpenSource alternatives such as Openstack and StratusLab in fulfilling this role.
Speaker: Robin Eamonn Long (Lancaster University (GB))
• 259
Using enterprise-class software to monitor the Grid - The CycleServer experience
Monitoring is an important aspect of any job scheduling environment, and Grid computing is no exception. Writing quality monitoring tools is however a hard proposition, so the Open Science Grid decided to leverage existing enterprise-class tools in the context of the glideinWMS pilot infrastructure, which powers a large fraction of its Grid computing. The product chosen is the CycleServer, created and maintained by Cycle Computing LLC, for which OSG negotiated a very advantageous licensing deal. In this poster we present an overview of its features, alongside the experience of its use by several OSG VOs.
Speaker: Igor Sfiligoi (University of California San Diego)
• 260
Using Puppet to contextualize computing resources for ATLAS analysis on Google Compute Engine
With the advent of commercial as well as institutional and national clouds, new opportunities for on-demand computing resources for the HEP community become available. With the new cloud technologies come also new challenges, and one such is the contextualization of cloud resources with regard to requirements of the user and his experiment. In particular on Google's new cloud platform Google Compute Engine (GCE) upload of user's virtual machine images is not possible, which precludes application of ready to use technologies like CernVM and forces users to build and contextualize their own VM images from scratch. We investigate the use of Puppet to facilitate contextualization of cloud resources on GCE, with particular regard to ease of configuration, dynamic resource scaling, and high degree of scalability.
Speaker: Carl Henrik Ohman (Uppsala University (SE))
• 261
Using ssh and sshfs to virtualize Grid job submission with rcondor
The HTCondor based glideinWMS has become the product of choice for exploiting Grid resources for many communities. Unfortunately, its default operational model expects users to log into a machine running a HTCondor schedd before being able to submit their jobs. Many users would instead prefer to use their local workstation for everything. A product that addresses this problem is rcondor, a module delivered with the HTCondor package. RCondor provides command line tools that simulate the behavior of a local HTCondor installation, while using ssh under the hoods to execute commands on the remote node instead. RCondor also interfaces with sshfs, virtualizing access to remote files, thus giving the user the impression of a truly local HTCondor installation. This paper presents a detailed description of rcondor, as well as comparing it to the other methods currently available for accessing remote HTCondor schedds.
Speaker: Igor Sfiligoi (University of California San Diego)
• 262
ValDb: an aggregation platform to collect reports on the validation of CMS software and calibrations
The scrutiny and validation of the software and of the calibrations used to simulate and reconstruct the collision events, have been key elements to the physics performance of the CMS experiment. Such scrutiny is performed in stages by approximately one hundred experts who master specific areas of expertise, ranging from the low-level reconstruction and calibration which specific to a sub-detector, to the reconstruction of higher level quantities such as particle candidates and global event properties. In this paper we present ValDb, a web-based aggregation platform which collects from the all validation experts reports consisting of a concise write-up and links to pictures and plots. The reports are organized in campaigns, each targeting one specific software release or calibration update, and are all marked with a final summary icon (pass, fail, changes expected). ValDb is integrated with the CMS hypernews mailing system, where reports are sent to concerned fora.
Speaker: Antanas Norkus (Vilnius University (LT))
• 263
VomsSnooper - a tool for managing VOMS records
VomsSnooper is a tool that provides an easy way to keep documents and sites up to date with the newest VOMS records from the Operations Portal, and removes the need for manual edits to security configuration files. Yaim is used to configure the middle-ware at grid sites. Specifically, Yaim processes variables that define which VOMS services are used to authenticate users of any VO. The data for those variables is administered centrally at the Operations Portal, and it is made available in XML format. It was necessary at each site to manually convert the XML data to make it suitable for Yaim. At Liverpool, we wrote VomsSnooper to partly automate this process by checking and creating new VOMS records directly from the portal, providing a bridge between the Operations Portal and the site configuration. The tool is also used to automatically obtain and publish online the VOMS data of thirty GridPP Approved VOs in the GridPP wiki. This tool solves several problems for Liverpool and the wider GridPP community. Firstly, the VOMS records for GridPP Approved VOs were available from two sources that were not necessarily consistent, i.e. the Operations Portal and the GridPP wiki. So the first use case for VomsSnooper was to periodically synchronise the data on the GridPP wiki to the canonical source of the data in the Operations Portal, making both data sources compatible. The Approved VOs wiki now receives reliable, accurate, semi-automatic updates on a weekly basis, and long-term staleness has been eliminated. Sites can now update their records from either data source with more confidence. Once a process was developed to automatically extract and format the VOMS records from the XML, it was apparent that the intermediate step of reading the Approved VOs wiki could be eliminated altogether. To this end, use cases were developed to both check the VOMS records at any site, and create new records directly from the Operations Portal. Sites who choose this approach can keep their records update to date in a semi-automatic manner, without reference to the Approved VOs wiki, and without manual edits to the security configuration files.
Speaker: Stephen Jones (Liverpool University)
• 264
WLCG Transfers Dashboard: A unified monitoring tool for heterogeneous data transfers.
The Worldwide LHC Computing Grid provides resources for the four main virtual organizations. Along with data processing, data distribution is the key computing activity on the WLCG infrastructure. The scale of this activity is very large, the ATLAS virtual organization (VO) alone generates and distributes more than 40 PB of data in 100 million files per year. Another challenge is the heterogeneity of data transfer technologies. Currently there are two main alternatives for data transfers on the WLCG: File Transfer Service (FTS) and XRootD protocol for transferring data on the XRootD federated storage. Each LHC VO has its own monitoring system which allows it to understand its own transfer activity but is limited to the scope of that particular VO. There is a need for a global system which would provide a complete cross-VO and cross-technology picture of all WLCG data transfers. We present a unified monitoring tool - WLCG Transfers Dashboard - where all the VOs and technologies coexist and are monitored together. The scale of the activity and the heterogeneity of the system raises a number of technical challenges. Each technology comes with its own monitoring specificities and some of the VOs use several of these technologies. The presentation will describe the implementation of the system with particular focus on the design principles applied to ensure the necessary scalability and performance, and to easily integrate any new technology providing additional functionality which might be specific to that technology.
Speakers: Alexandre Beche (CERN), David Tuckett (CERN)
• 265
XRootd, disk-based, caching-proxy for optimization of data-access, data-placement and data-replication