Baseline Services Working Group

Date 23:02:05 

Present at CERN:

Ian Bird (chair)
Markus Schulz (secretary)
Philippe Charpentier
Dirk Düllmann
Julia Andreeva
Miguel Branco
Alessandro de Salvo
Latchezar Betev

Stefano

Remote Participants:

Peter Elmer
Anders Waananen
Flavia Donno
Peter Elmer, Stefano Lacaprara
Andrei Tsaragorodtsev
Steve Traylen
Razvan Popescu

The meeting started off with Ian giving the presentation that has been on the 
Agenda page: http://agenda.cern.ch/fullAgenda.php?ida=a051165

The minutes will not contain a detailed description of the slides, but outline the
discussions triggered by the slides.

The main objective of the presentation is to describe what the Baseline Services Working Group (BSW)
is and what mandate the group has 

Comments on the slides: 

Slide 0
-----------

Lists the members of the group.

Slide 1
----------- 

Goals

Time scale, full services in operation September 2006
Groups work is input to the LCG TDR and has to finish by end of April 2005.

Scope
avoid overlap with other activities in the project, focus on defining core services with targets for 
functionality and scalability/performance metrics

For areas where development efforts are needed the required resources have to be identified and 
the feasibility has to be checked.

Fallback solutions have to be described. 

The responsibilities of the experiments and the project have to be defined and the 
experiment plans and ideas exposed.

Slide 2
-----------

Guidelines

A clear identification of Experiment components and services provided by the project is 
needed. Where needed fallback solutions, which can't be development projects, have to be specified.

This is NO DESIGN EXERCISE, but draw on existing software.

Developments, if needed, have to be short term and the long term maintenance has to be taken into account

Performance metrics should be defined for all relevant services

It has to be clarified how the services will affect the regional centers and the operations infrastructure. 

Question by Philippe:

We might end in a deadlock in case we identify services that are wanted, but we have doubts that the
metrics will ever be met. What do we do with those?

Ian:
Clarifications:
 You ask for a service that we need in a year from now with these characteristic, but you don't 
 rust that this is there 

Phone: Do you refer to a generic, or specific service?

Philippe: generic

Slide 3
-----------

Scope and Timescale

This is not a middleware group 
split between experiments and projects in terms of provision of services and  functionality needed.

We have to report quickly, end March a first report is needed and by end of April 2005 the final report has to be provided 

Slide 4
----------------
Service Challenges

The timeline for the service challenges.

SC2 (now) SC3 1 month in summer, then the services will move into production in autumn.

Sep 06 the initial LHC service has to be in stable production

This is a very compressed time scale.
We have to build up form the bottom the services and have them in stable prod.

Slide 5
-----------

Meetings

The proposed frequency is every 2 weeks. To make rapid progress small subgroups have to 
be defined that create proposals to be discussed in the biweekly meetings.

The second meeting could take place at the end of next week.
Experiments present their view of the core services. 

We will use the results of the data management workshop and
other focussed discussions that we will trigger, for example on subjects like SRM functionality.
The service challenges will give very concrete input.

Slide 6
------------

Baseline services

Ian presented a list to start with. The list might be more than complete and if possible 
we can even get rid of a few items.

Phillipe: We should include the transparent (posix style) grid file access.

Some discussions sparked from this. It was centered around the issue of the locality 
of this I/O. It was agreed that the posix style I/O should be provided to local files and include 
links to the catalogue 

Some of the members of the meeting commented on several of the items.

Dirk: Storage management part should be separated from the catalogue issues  

ATLAS: Space management is very important for us.

Ian: Will be dealt with in the storage management section

Flavia: Left us here.......................

LHCb: Does the workload management include the CEs?

Ian suggested that we should discuss this one by one while going through the 
different services

Slide 7
-----------

Storage Management

It is clear that the Tier 1s need the SRM interfaces to their storage, but we need to put this in  a written form.

All sites should move to SRM,  but we need to define the required SRM subset needed.

Ian recommends to start with Jean Phillips report on SRM subsets (see link)

The storage management workshop can be used to discuss this. 
To do this  we have to go to the workshop with a list of prepared questions.

LHCb:  We have to finish this part by the start of the workshop?

Ian: Not completely, but we have to bring in the experiments view!

Slide 8
-----------

Reliable File Transfer

We are testing an implementation provided by gLite in the SCs.
This will help to understand the needed functionality/

File placement service
This is related, but maybe not a generic service. gLite proposes a generic FPS, CMS uses PhedEX.

Should we worry about this?

LHCb: Is registration included in the reliable file transfer.

Ian: The current model has an agent for each file catalogue to do the registration. 

Phone: PhedEX can move chunks of files in a coherent way.  Collections based transfer should
           be in the baseline definition of a service.

Ian: This is clear. Collections, groups, sets have to be supported. Otherwise you won't get any performance.

Slide 9
------------
Grid Catalogues 

The experiments should present their plans. There are already the metadata catalogues. 

What is nescessary to a grid wide catalogue?

What is needed from the services?

You might just ask us to run the ORACLE back-end.

Where should the metadata go? 

Dirk: We have to define what metadata is

Alice: We should not mix the metadata definition and the service.

LHCb: Something like: File location, date, owner, size and the physical file name should be there, but 
 information like the version of program that produced the data and what is in the file is experiment dependent and should not be part of a file catalogue. 

Ian:
We have to simplify, but get the interfaces defined correctly to interoperate with different services.
For example, there are DLI and SI. What are the interfaces for other components like POOL?
How can be brake this down and isolate these things?

Phone: Andrew 
Current line is to define interfaces as Web-services and use  Soap as a protocol. 
We have to be practical and not stick to the religion. 

(this was not very clear to understand)

Ian: We should define the function and performance. Then we see what meets this

Dirk and LHCb agreed and gave strong support to this concept.

Slide 10
------------

Workload Management

Is this a baseline service? Can this be done in an efficient way by the experiments?

The CE and the batch systems are a baseline service. Is the RB an agreed services?

Alice: If not the RB, then it is only the batch system to connect to...

LHCb: The CE should be simple, you can ask questions and it handles job submission.

Ian: Even now you can talk in different ways to a CE (condor, globus, ..) and can build your own RB using 
the provided interfaces

LHCb: You need a defined way to install an agent on the CE, or open the firewalls to give the experiments direct access to the WNs.

Atlas: We  use the RB

Ian: LHCb uses in a very experiment specific way to handle jobs

LHCb: We don't make real use of the RB.

Phone: There is a relation between the WLMS and the file access, this should not be forgotten.

Ian summarized:

Seperate the CE from the storage.

Ian: We need to specify what we can do with a CE

Phone: We can have multple WLMs, but only one CE

Ian: There was a workshop last week in Rome. It discussed a common CE definition including JDL scope.
       Mirco, Anders, glite, globus, egee, and norduGrid participated among others.

Anders: A common interface to a CE is quite in the future,....

Ian: But  we have already one, globus and condorG

Philippe (LHCb):
We have to be informed of workshops like this. The experiments are the only ones with experience 
in successful developing, implementing and operating WLMS. We are kept in the dark about discussions like this.

ATLAS: There have been people from Atlas at this event

Ian: It has been widely advertised.

Slide 11
-------------

Grid Monitoring

This has mainly answer the question:
 What happens to a job?

There was a discussion about what to do when there are no RBs.
Currently the RBs track the state of jobs.
If experiments use their own systems then there is no global monitoring view.

Phone: There is a problem, if each experiment is going to have it's own WLMS then grid monitoring is very hard. Since most information is lost.

Ian:  There is a set of things that you can get from the CEs

Phone: Yes, but monitoring needs to connect to the experiments WLMS to get the full picture

Ian: What if we have a RB and no one is using it?

Markus: 
The RB is not only doing matching jobs with resources, but provides tracking of jobs. The information in the Logging and Bookkeeping DB is essential for spotting problems with sites/VOs etc. This is essential information for the operations team.

ALICE:
We use our own (monitoring?), this is fine. However there is information that we collect from the batch systems that should not be shared by a global view. 

Ian:
What has to be provided by the CE and the LRMS?

LHCb: 
The state of the jobs has to be reported. 
But this is currently quite useless. Because you never know wether the "Finished State" means 
that the job ended successful or not. 

Anders: We have to specify more complex error codes

Ian: We have to state what we need functional and leave it to the implementer to specify the codes.

Phone:
Please, not another round of requirement gathering, we need to start telling what we think about certain implementations and start selecting.

Phone: The metric gives a different criteria

Dirk: 
Yes, but multiple might survive this, how do ensure that those interoperate?
The interfaces have to be specified.

Ian tried to summarize the discussion:
It is both, we have to understand what we want and where it will come from.

Phone:
We should not exclude anyone, but someone should be warned!

Slide 12
--------------

VO Management

Can be expanded to access rights.

Ruth left, but will call Ian back on slide 5

LHCb: 
The roles will be defined in the data management.

Dirk:
Defining roles is one thing, but implementation has to be possible with current systems

Slide 13
---------------
Installation of Application Level Software

GAG produced a rather complete document, but we have to prioritize it. 

Slide 14
--------------
How to proceed?

Experiments should describe what they are doing right now and what are the requirements originating  from 
these activities.

Clear picture for some of the services, but some are not so clear. We have to go through this one by one.
Then small teams work on proposals.

Ian suggested the following prioritization:

file transfer,
storage
catalogues 
WLM (here more input is needed)
monitoring and software installation are less important

Comments:
==========

Ian:  Next week would be good for the presentation of the status of the experiments view.

Alice:
We have next week an ALICE week and we can work only on the first one.

Ian:
The first three are somewhat coupled

ATLAS:
We can do the the first two.

ALICE:
We need more time for internal discussions.

Ian: 
If this triggers too much discussions inside each experiments we will never be finished in time.
Next Friday afternoon, and we limit it to storage and file transfers.
The experiments that can report on more are free to do so.

If there are constraints on the other items they you should be mentioned.

The next meeting will take place: Friday at 16:00

LHCb:
But everything has dependencies, how should we handle this?

Ian:
If you know already that you have your own catalogue then make this clear.