(LAL / IN2P3), Sandy Philpott
The HEPiX meetings bring together IT system support engineers from the High Energy Physics (HEP) laboratories and institutes, such as BNL, CERN, DESY, FNAL, IN2P3, INFN, JLAB, NIKHEF, RAL, SLAC, TRIUMF and others.
They have been held regularly since 1991, and are an excellent source of information for IT specialists.
That's why they enjoy large participation also from the non-HEP organizations.
New developments at the RHIC/ATLAS Computing Facility (RACF) at
Brookhaven National Laboratory.
(Brookhaven National Laboratory)
CERN site report
Report on news at CERN-IT since the Fall meeting in Berkeley
Storage and Filesystems: I
Progress Report 2010 for HEPiX Storage Working Group
Evaluation of NFS v4.1 (pNFS) with dCache
The talk will be covering experience accumulated with NFS4.1/dCache at DESY.
Building up a high performance data centre with commodity hardware
High data throughput is a key feature in today's LHC analysis centres. This talk will present the approach of the DESY site Zeuthen to build up a high performance dCache out of commodity hardware. Additional optimisations to achieve further throughput enhancements will be mentioned, as well. By presenting current benchmarks we show competitive results compared to more expensive installations.
(Deutsches Elektronen-Synchrotron (DESY))
Petersburg Nuclear Physics Institute (PNPI) status report
In this short presentation I plan to describe the responsibility for Computing Systems Department (CSD): LAN (400 hosts), mail service for the Institute, other centralized servers, computing cluster. The computing needs for small physics laboratory, experience with the cluster at PNPI HEPD and plan for cluster future in connection to LHC data analysis are discussed in more details.
(Petersburg Nuclear Physics Institute (PNPI))
The new Generations of CPU-GPU Clusters for e-Science
The third paradigm of research known today as e-Science requires that the computational sciences community rely heavily on HPC clusters to run their simulations. At the core of each cluster lie commodity devices with increasingly parallel capabilities. The two most successful processing devices today complement each other: the multicore CPUs and the manycore GPUs as accelerator vector devices. The main focus of this communication will be on an overview of some of the most promising approaches in this new breed of processing devices (and on the CUDA environment) to build the new generations of HPC clusters and how the scientific community has reacted to this challenge.
(Dep. Informática, Universidade do Minho)
Site Reports: II
SLAC Site Report
SLAC site update since the Fall 2009 HEPiX meeting.
Fermilab Site Report
Site Report from Fermilab
Lessons Learned from a Site-Wide Power Outage
On January 19th, 2010, a series of powerful thunderstorms knocked out all power to the SLAC site (and surrounding areas). While the SLAC Facilities personnel worked to restore power to the computer center and the rest of the site, Computing Division planned for restoring services. This included setting priorities for such things as business systems needed to prepare the payroll and servers needed by the Fermi Gamma-Ray Space Telescope to process their data in a timely manner. I will describe problems that arose, what solutions were found, and what lessons were learned to minimize the impact of futures and streamline recovery from them.
Storage and Filesystems: II
CERN Lustre evaluation and storage outlook
The CERN evaluation of Lustre as a potential solution for home directories, project space and physics storage has now completed. This experience will be presented along with outlook for bulk data storage and life cycle management in a capacity hungry and performance sensitive environment.
LCLS Data Analysis Facility
I will present the experience and lessons learned from the first few month of real users taking real data at LCLS. From those lessons, a new offline data analysis facility was designed and will be installed before the next set of users arrive on May 6.
Storage and Filesystems: II (cont'd)
GEMSS: Grid Enabled Mass Storage System for LHC experiments
New Grid Enabled MSS solution based on IBM's products (GPFS and TSM) has been developed and commissioned into production for LHC experiments in Tier1 at CNAF. Architectural view as well as installation and configuration details will be presented. We will also present technical details of data migration from CASTOR to GEMSS for CMS, LHCb and ATLAS experiments.
OpenAFS Performance Improvements: Linux Cache Manager and Rx RPC Library
This talk will focus on two areas of performance improvement that will be shipping with the OpenAFS 1.6 series this coming Summer.
Significant work on improving performance has been completed on the OpenAFS 1.5 branch. Linux memory management has been heavily overhauled, removing a number of deadlock conditions, and significantly speeding access to data which is already held in the local cache. We now make correct use of the Linux page cache, and use data cached in memory in preference to pulling it from local disk. Performance in real world applications (a web server with its working set in the local cache) has been more than doubled by these changes.
The Rx RPC library has also been a focus of considerable attention with funding from the U.S. Dept of Energy. A wide variety of implementation errors that resulted in lock contention, lost packets, and extended timeouts were identified and corrected.
(Your File System Inc.), Simon Wilkinson
(Your File System Inc.)
Lavoisier : a way to integrate heteregeneous monitoring systems.
Lavoisier is an Open Source Tool developed at CC-IN2P3 used in the project EGEE/ LCG by the Operations Portal. Lavoisier is an extensible service designed to provide an unified view of data collected from multiple heterogeneous data sources (e.g. database, Web Service,LDAP, text files, XML files...). This unified view is represented as XML documents, which can be queried using standard languages, such as XSLT (and soon: XPath, XQuery).
The access to data views can be done through different standard ways : SOAP, REST/JSON or REST/XML.
Lavoisier also improves performance by caching the generated data views. The caching mechanism can be tuned according to the characteristics of the data views and of the data sources, and according to the constraints of the use-cases. This tuning has no impact, neither on plug-ins, nor on user's code.
The global architecture has four layers: the legacy data sources, the adapters used to generate the data views, the cache mechanism with flexible refresh triggering rules, and the service exposing both SOAP and REST (with XML or JSON)
Moreover the Lavoisier architecture permits a clear separation of roles and changes related to a role have no impact on other roles :
- the plug-in developer adds support for new technologies by developing adapters.
- the service administrator configures the adaters and cache management according to the characteristics of the data views and data sources, and according to the constraints of the use-cases.
- the user queries the data views.
This last year an effort has been put to ease the integration of data for service adminstrators by developing web interfaces which permits to :
- see the list of views and their characteristics
- see the status of these views and the eventual errors linked with these views
- see the configuration file
- evaluate the dependencies between views .
Lavoisier has proven effective in increasing maintainability of the Operations portal, by making its code independent from the technologies used by the data sources and from the data cache management policy. Its design and interfaces made easier writing reusable code, and good performances are easily obtained by tuning the cache mechanisms in an absolutely transparent way for the portal code.
Indeed, the different components work in a standardized way through the output of the Lavoisier Web Service. The translation of resource information into this standardized output is provided by different plug-ins.
Recently, a huge joint effort has been recently put into the configuration of Lavoisier, the structure of its caches and the rules of refreshing to have efficient, scalable and reliable data handling. Indeed, all information has been structured around the base component of the Operational Model: the site. We retrieve the global information about primary sources like GGUS (official ticketing system for EGEE) , GOC DB (official database for sites in EGEE), SAM (Monitoring System) , or Nagios (another Monitoring system) and we organize it by sites.
The main idea is to construct a summary of the different available information for a site: firstly, this organization permits to continue to work with the caches, even if a primary source is unavailable; then you access only information you need on the web page. Your information is structured around a synoptic view of the site and you don't access hundreds of times the primary sources but a subset of them with the site view.
Finally, we refresh the data sources only when we need it and only when an action has been triggered.
Last but not least, it is very easy to add a new data source in this model. In the configuration file of Lavoisier you add the access to the primary source and also the split of this information per site.
This information is then readily available in the synoptic view of the site.
Scientific Computing: first quantitative methodologies for a production environment
In Data Center the necessity for the accounting of resources used by different groups is growing.
In this talk (and the correspondent note) the context and the scientific computing activity will be defined. Taking as an example a medium size Data Center (Pisa INFN) an industrial production approach will be followed.
A first methodology for the quantitative evaluation of the following data will be proposed:
- production level
- resource utilization efficiency
- cost distribution.
The evaluation of the resource utilization and the cost accounting are based on the electric power consumption: it will be described that this value can be considered as an practical example. The same methodology can be extended to different kind of costs and resources.
As practical application the results of a survey regarding the INFN-Pisa are reported.
We will present a report on the current status of the PDSF cluster at NERSC/LBNL.
(Lawrence Berkeley National Lab. (LBNL)-Unknown-Unknown)
Grid and WLCG: I
CESGA Experience with the Grid Engine batch system
Grid Engine (GE) is an open source batch system with a complete documentation and support for advanced features like the possibility to configure a shadow master host for failover purposes, support for up to 10.000 nodes per master server, application level checkpointing, array jobs, DRMAA, fully integrated MPI support, a complete administration GUI, a web-based accounting tool (ARCo), etc.
CESGA has been using GE in its systems during more than 5 years. Currently it is the only batch system used at CESGA both for the supercomputers and for grid clusters.
One of the last challenges was the integration of GE with the FINIS TERRAE supercomputer installed at CESGA, it is formed by 142 Itanium nodes of 16 CPUs and 128GB of memory each one. Several stress tests were done to check SGE behaviour in a large cluster, results will be shown. It also will be explained some special SGE configurations like:
- HP-MPI and Intel MPI integration: most of the jobs that run in Finis Terrae are MPI jobs.
- Checkpointing. It has been configured in the queue configuration.
- Exclusivity: possibility to request a node exclusively.
- SSH-less node configuration: remote connections between nodes are done using qrsh instead of ssh. This is very important in the case of MPI jobs to avoid jobs expanding outside the nodes they have been assigned or trying to use more resources in a node than the ones assigned to the job.
- Interactive jobs: jobs run interactively using a shell interface.
- Application integration with GE: for example Gaussian, Gaussian is used only under the batch system, the batch job requirements are taken accordingly to Gaussian input.
In order to manage special user requirements, CESGA has developed a qsub wrapper to implement some additional functionalities like special resources, i.e. the possibility to request additional resources for the jobs than the established limits (memory, CPU time, number of processors, space on scratch disk, …)
Some kind of jobs (challenges, priority agreements, …) need to be prioritized, it will be explained how these jobs are prioritized using GE functionalities.
Thanks to the efforts of IC, LIP and CESGA currently GE is fully supported by gLite middleware.
In a standard configuration it requires a Computer Element (CE) for each grid infrastructure. CESGA has done several modifications to the standard configuration to support different grid projects (EGEE, EELA, int.eu.grid, Ibergrid, and other regional grid projects) using just one centralized batch system. This way resources can be shared among different grids with minimal system administration overhead.
The batch system is shared using one single GE qmaster server and a shadow qmaster for fault tolerance purposes. Jobs are submitted from different sources but all jobs are collected in a single batch server that distributes them between all the available WN.
MrEsteban Freire Garcia
CERN Grid Data Management Middleware plan for 2010
The 2010 program of work for the CERN Grid Data Management team is presented,
describing the planned development activity for 2010. The plan for FTS,
DPM/LFC and gfal/lcg_util is described along with longer term perspectives.
Grid and WLCG: I (cont'd)
(LIP Laboratorio de Instrumentaco e Fisica Experimental de Particulas)
EGEE Site Deployment: The UMinho-CP case study
EGEE is a dynamic organism with requirements that constantly evolve over time.
The deployment of UMinho-CP - an EGEE site supporting Civil Protection related activities -, revealed new challenges, some of which were overcome till now, while others are planned for future work.
A great effort has been made to automate the distribution, installation and configuration of an EGEE site, for efficient deployment and maintenance, making each site an agile intervenient of the European-wide grid infrastructure.
Update on HEPiX Working Group on Virtualisation
The HEPiX working group on Virtualisation started it activities since the last HEPiX meeting. This talk will present the current status and future plans.
Virtualization at CERN: a status report
During the HEPiX meeting 2009 in Berkeley several reports
related to virtualization efforts at CERN were given. This presentation will
be a status report of these projects, with the focus on service consolidation
and batch virtualization. Recent developments like the evaluation of the
Infrastructure Sharing Facility (ISF) from Platform computing,
issues seen with the tools we use and the technical implementations, as well as the general
status of the different projects will be reviewed, and first experiences in an
operational mode will be presented.
virtual machines over PBS
Since PIC is supporting the data management of several scientific projects it is faced to requirements that might not be possible to enable in a homogeneous infrastructure. When looking for an alternative approach we have chosen to export the specific requirements into a virtualized environment. In order to minimize the impact to our physical computing infrastructure it was necessary to develop tools that could be incorporated into our native batch system. A first solution is already put in production and can provide almost completely arbitrary software environments in a transparent manner to the end-user.
Marc Rodriguez Espadamala
An Adaptive Batch Environment for Clouds
We have developed a method using Condor and a simple software component we call Cloud Scheduler to run High Throughput Computing workloads on multiple clouds situated at different sites. Cloud Scheduler is able to instantiate user-customized VMs to meet the requirements of jobs in Condor queues. Users supply Condor job attributes to indicate the type of VM required to complete their job. Condor then matches jobs with the appropriate VMs. Cloud Scheduler supports Nimbus and Amazon EC2, in addition we have preliminary support for Eucalyptus, and we plan support for OpenNebula 1.4. We present our early experiences with two separate projects using this method.
(University of Victoria)
HEPiX Board Meeting (Board Only)Conference Room - Ground Floor (LIP - Av. Elias Garcia, 14 - Lisboa)
An update on JLab's Scientific Computing facilities, including the new ARRA cpu and gpu clusters for USQCD, Lustre, experimental physics' data analysis farm, data storage, and networking.
Virtualization in the gLite Grid Middleware software process. Use-cases, technologies and future plans.
During the past years, the Grid Deployment and Grid Technology groups at CERN have gained experience in using virtualization to support the gLite Grid Middleware software-engineering process.
Some technologies such as VNode and VMLoader have been developed to leverage virtualization in supporting activities such as development, build, test, integration and certification of software.
This know-how has been collected in use-case analysis, usage statistics and requirements to help the development of a common virtualization solution strategy.
Management Information Systems and Information Systems Management: Two Sides of the Same Coin
The relationship between information systems and management outside large organizations has traditionally been one way only, with information systems supporting management but management not supporting information systems.
However, with information systems becoming ever more critical and complex, the need to manage information systems is becoming pervasive in most, if not all, organizations. The success rate of information systems projects, which *decreased* last year, is just an example that management is desperately needed in the IT world.
In this talk I will remember what is information systems and management, and then discuss the two relationships that exist between them. I will pay special attention to ITIL -- which has been (wrongly) proposed as the solution for all kinds of problems -- and its role in the much more general and interesting IT Governance area. The talk will finish with a reminder that innovation, not technology, is the source of economic growth.
Prof.Miguel Mira da Silva
Virtualization: II (cont'd)
Virtual Network and Web Services (An Update)
This talk describes techniques for providing highly
available network services with help of a hardware based load
balancing cluster handled by a sophisticated control software.
An abstract representation of web, mail and other services
is implemented on base of application specific routing on network
layers 3 to 7. Combining application, network and security issues in
one point you may change server configurations or enhance availability and
performance without service interruption. This is an
update talk with some emphasis on the application and server side.
Virtualisation for Oracle databases and application servers
With the growing number of Oracle database instances and Oracle application servers, and the need to control the necessary resources in terms of manpower, electricity and cooling, virtualisation is a strategic direction that is being seriously considered at CERN for databases and application servers.
Oracle VM is the Oracle certified and supported server virtualisation solution for x86/x86_64 platforms.
In order to be able to use Oracle VM in large scale environments, it must be prepared so that it can be installed and configured in a very short time and without any human interaction as well as maintained in a central way. At CERN, this is done with the CERN ELFms (Extremely Large Fabric management system).
This presentation will explain the steps and work done to install and maintain 2 versions of Oracle VM: 2.1.5 and 2.2. The presentation will also cover the tests of Databases and Application servers running in this virtual environment.
Carlos Garcia Fernandez
Storage and Filesystems: III
Lustre-HSM binding is a collaborative effort between Sun and CEA.
This presentation will introduce the first version of Lustre-HSM:
HSM support model, its features, components, development status, and roadmap.
It will also give examples of importing files from an existing HPSS namespace, archiving and restoring data, releasing disk space...
The RAL Tier1 is continuing to benefit from bringing more of the Tier1 systems under Quattor control. That experience will be presented along with a report on recent and planned developments with Quattor across the wider grid and non-grid environments.
MrIan Peter Collier
Spacewalk and Koji at Fermilab
Spacewalk is an open source Linux systems management solution. It is the upstream community project from which the Red Hat Network Satellite product is derived.
Koji is Fedora's build platform.
Fermilab is currently testing Spacewalk for various roles. We are testing it for desktop and server monitoring, system configuration, and errata information. Although we currently are still in the testing stage, we have been impressed with it's functionality and development.
Fermilab's Scientific Linux development team is looking at Koji for the building of SL6 and possibly SL5. Many of it's built in functionality's make building packages easier, and it's reporting tools are very nice. But there is a steep learning curve when installing and setting up Koji.
Virtualization Working Group F2F
For WG members.
Agenda: see http://indico.cern.ch/conferenceDisplay.py?confId=92038.
WG meeting (http://indico.cern.ch/conferenceDisplay.py?confId=92038)
Preliminary Measurements of Hep-Spec06 on the new multicore processor
We had access to a remote server with 2 x 6174 amd processor.
Preliminary measurements will be show
Some early measurements on an Intel 5600 worker node will be exposed.
(Univ. + INFN)
Hyperthreading influence on CPU performance
In this report we present the hyperthreading influence on CPU performance when running the HEP-SPEC2006 benchmark suite in a 2 quadcore CPU shared memory system
(HP BL460c G6). This study was performed as a function of the number of running
instances (from eight to sixteen), and with and without SPEC RATE. We concluded
that the elapsed application run time can be clearly reduced if hyperthreading is
on. The effect becomes stronger as the number of running instances increases
reaching up to 30% for the maximum number of tested running instances.
We intend to extend this work for other shared memory systems, and try to define an upper limit for the CPU performance increase as a function of running instances.
Operating Systems and Applications: I
Scientific Linux Status Report and Plenary Discussion
Progress of Scientific Linux over the past 6 months. What we are currently working on. What we see in the future for Scientific Linux.
Also we will have a Plenary discussion to get feedback to and input for the Scientific Linux developers from the HEPiX community. This may influence upcoming decisions e.g. on distribution lifecycles, and packages added to the distribution.
Operating Systems and Applications: II
Windows 7 Deployment at Cern
Windows 7 is officially supported at CERN since the end of March. The official Windows 7 service was preceded by pilot project that was started in December 2009. We will present our experience gathered during pilot phase as well as trends and deployment plans for Windows 7 at CERN.
TWiki at CERN: Past Present and Future
TWiki was introduced at CERN in 2003 following a request from a small group of software developers. The service soon grew in popularity and today there are over 7000 registered editors of 60000 topics.
This presentation takes a look at the current service ,the problems experienced and a look at future developments for TWiki at CERN.