Monte Carlo Production

ATLAS operations in the GridKa T1/T2 Cloud John Kennedy LMU/RZG, Germany
Simon Nderitu Universität Bonn, Rodney Walker, Günter Duckeck, Cedric Serforn, Julien de Gratt LMU München,
Gen Kawamura Universität Mainz, Jan Eric Sundermann Universität Freiburg,
Andrzej Olszewski INP PAN Krakow, Jiri Chudoba Institue of Physics Prague, Kai Leffhalm DESY-Zeuthen

Organization

In a distributed environment organization and communication play a vital	role.

Cloud Monitoring

Distributed Analysis

We focused on forming contact to the sites in our region and identifying the ATLAS Services which we needed to perform to ensure smooth operation in the cloud

Each service area has a coordinator who is responsible for the service and communication within the cloud and with ATLAS in general

Organization:

Monthly cloud video meetings including T1/T2 contacts and service coordinators

Weekly operations meetings between T1 and service coordinators

Wiki pages and mailing lists

Core Services:

Data Management DDM

Monte Carlo Production

T1 – contact,reprocessing

Distributed Analysis – development, tests, training events

Installation of ATLAS software

In addition to organisation we also

Participate in ATLAS wide computing challenges

Perform internal cloud tests

Ensure our presence in ATLAS computing meetings and working groups

Each of these areas is seen as critical to the cloud operation and we ensure redundancy

In addition a T2s coordinator role was recently defined to liaise with the T2s (and T3s)

Internal cloud developments:

Cloud Monitoring

Transfer Tests

Tools for frequent tasks

As a community we aid each other by developing and sharing tools and expertise

Data Management – DDM

The management of data is essential within the cloud and is one of the most challenging tasks.

Tasks within the real of data management include

Data Distribution

Ensuring space management is respected (space tokens at sites etc)

Data consistency and cleaning

Data aggregation from MC production

Replication of user data on request

The evolution of disk space at the T1 centre

Work within the DDM field requires good knowledge of many systems such as DDM, FTS, LFC, SRM and storage elements (dpm,dcache).

Development of internal cloud monitoring and tools

Provide a single view which shows cloud status

Gathering information from various sources

Provide links to sourced monitoring sites for deeper inspection

The Cloud monitoring development has eased the operations load within the cloud and provides managers and shifters alike with a simple overview of the health of the cloud

File Transfer Testing/Monitoring

Test and monitor file transfer between sites

Test different transfer mechanisms such as srmcp,FTS,lcg-cp thus allowing problems to be better identified

Persistify the data allowing views of distinct periods

Such tools aid cloud operations and can benefit the global community

Monte Carlo production is a continuous service within the cloud with:

Two pilot submission instances running within cloud to provide redundancy and scaling.

Strong contribution from T2s

T3s also starting to run production

The submission instances require supervision and good contact is needed to the site managers to aid with problem spotting and solving

MC production walltime in 2009

Job distribution per site in 2009

Monte Carlo production in 2009 has achieved a walltime efficiency of nearly 84%

With in excess of 7500 simultaneous jobs running within the cloud

Tier1 Operations

Staging tests with 9TB of data (3127 files) – 190MB/s

Data Reprocessing

Large scale retrieval from Tape is challenging

Numerous test activities during last 12 months

Through these tests a visible

Improvement in tape recall performance has been achieved

Recall tests in spring 2008

Recall tests in spring 2009

Showing marked improvement

Extensive monitoring for GridKa helps ATLAS

maintain a reliable service from the T1

Distribute analysis a cornerstone of our cloud services

Ensure cloud is ready for Distributed Analysis

Regular tests of distributed analysis in cloud

Good communication between Dist Analysis team and site contacts – dedicated tests for sites where problems seen.

Stress tests where appropriate

The Distributed analysis team within the cloud contribute to the development of the Ganga tool and provide support and tutorials for the user community both in the cloud and ATLAS at large.

We have strongly encouraged out user community to use the DA tools where available.

Regular testing leads to the solution to possible problems:

Here we see a case where data was being pooled on one disk, after re-distributing the data the cpu/walltime ration for analysis jobs is much improved

Cloud Participants:

>13 sites

5 countries

2 ROCs

before

after

See Presentation by Johannes Elmsheuser: Distributed Analysis in ATLAS using GANGA ,CHEP 2009

See Poster by Cedric Serfon: Data Management tools and operational procedures in ATLAS:

Example of the German cloud,CHEP 2009

Cloud Monitoring developed by Stefan Birkholz et al. Universität Goettingen

john kennedy

2003-07-23T17:38:45