Minutes of the GGSD meeting - 12 April 2007
Status of SRM 2.2 (Flavia)
A status update of the SRM 2.2 implementations was given (please, check slides).
Experiments are invited to let Flavia know about possible use cases to test.
All implementations pass basic tests. Still working on some details with
use-cases. Copy tests need attention for the endpoints where copy is
implemented.
4 implementations provide deployable releases: dCache (v1.8Beta), DPM (v1.6.4),
StoRM (v1.3.15), BeStMan (v2.2.0.0)
It was asked to include S2 based tests for lcg-utils and GFAL.
This will be done.
It was asked to document use-case tests. This is work in progress.
BNL asked what is needed to "participate" to the S2 test.
As a minimum you need to:
- Subscribe to the srm-tester mailing list: srmtester@lbl.gov
- Register the following DN:
/C=IT/O=INFN/OU=Personal Certificate/L=Pisa/CN=Flavia Donno/Email=flavia.donno@pi.infn.it
- Send the full endpoint (srm://<full hostname>:<port>/srm/managerv2?SFN=<dteam path>
to srmtester@lbl.gov
lcg-utils status (Jean-Philippe)
Please, check slides. lcg-utils v1.5.1-1 and GFAL 1.9.0-2 available in
production.
SRM v1 is the default and SRM v2 used only if no SRM v1 published or
space token is specified. This behavior is not configurable.
No ReserveSpace functionality offered through lcg-util/GFAL.
BringOnline not possible with lcg-utils but it is with GFAL. If needed
BringOnline can be made available via lcg-utils.
Pinning file not available since it is not supported by all implementations.
In particular, the CASTOR garbage collector can be configured per VO, depending
on the needs. VOs need to contact the CASTOR support team.
Releasing files is possible with both lcg-utils and GFAL.
lcg-cp now takes also SURL as destination (only if source is a local file or
is a TURL with GSIFTP as transport protocol) In this case no catalogue
operations are performed.
At the moment there is no way to address GFAL to use a specific file access
protocol. If needed, this can be implemented.
fts status (Paolo T.)
FTS 2.0 offers support for SRM 2.2. It is under test since December 2006 on
pilot FTS testbed. Deployment at CERN foreseen for April 2007 and at T1s one
month after. For features introduced and plans, please check slides.
It was observed that standard gridftp/SRM point to point tests should be made
part of SAM in order to monitor overall performance of the network.
SRM v1 to v2 migration strategy (Stephen)
Please, check slides for a full introduction and details on the problem.
Recommendations:
Sites should use alias as much as possible and keep the names of the SEs
stable. The GSSD will try to push forward this recommendations.
On sites where the SRM v1 and v2 endpoints are disjoint (each with its own
name service), it may be necessary for the VOs to decide which v1 entries
need to be made available under v2. This implies making a physical copy
of the data concerned if the v1 entries cannot simply be added to the v2
name service. Examples: CASTOR v1-->v2 at CERN, dCache-->CASTOR at RAL.
On sites where the SRM v1 and v2 endpoints are front-ends for the same
name service, data written by v1 remain accessible by v2 and vice versa.
Data written by v1 can only have storage class hints expressed through the
name space: the SE admin can map a particular directory to a particular
storage class. To ease the transition from v1 to v2 VOs could design
name spaces such that the different storage class instances only appear
high up the hierarchy, so that there will be not so many. Here is an
example of how the namespace should be organized:
Let's assume the following:
RAW --> T1D0
DST --> T1D0, but different tape set
ESD_master --> T1D1
ESD_replica --> T0D1
AOD --> T0D1
The namespace SHOULD NOT be organized as follows:
/my_exp/data/year/run/reco_pass/stream/RAW
/my_exp/data/year/run/reco_pass/stream/DST
/my_exp/data/year/run/reco_pass/stream/ESD_master
/my_exp/data/year/run/reco_pass/stream/ESD_replica
/my_exp/data/year/run/reco_pass/stream/AOD
Instead it SHOULD be organized as follows:
/my_exp/data/RAW/year/run/reco_pass/stream
/my_exp/data/DST/year/run/reco_pass/stream
/my_exp/data/ESD_master/year/run/reco_pass/stream
/my_exp/data/ESD_replica/year/run/reco_pass/stream
/my_exp/data/AOD/year/run/reco_pass/stream
This will easy the transition from v2 back to v1 if necessary.
It is not advisable to go back to SRM v1 once the full functionality of SRM
v2 has been used by an experiment. The goal is to test v2 thoroughly and then
migrate to v2 and stay with it.
All implementations offer new features together with the SRM v2 interface.
Therefore, upgrading to SRM v2 can imply more bugs to be sorted out.
The experiments have agreed to collaborate to make v2 working fully.
We should be pay attention to situations where both SRM v1 and v2 endpoints
are published. If the v2 endpoint is not visible for some reason, there is
the possibility that client tools default to SRM v1, creating confusion and
potential problems.
Storage Issues from ATLAS (Miguel)
Please, check slides.
Many scalability issues identified during transfer when reaching CASTOR
LSF max num of pending jobs threshold.
Existing CASTOR s/w could not cope with current ATLAS load. ATLAS
pushed for earlier deployment of new CASTOR-LSF plugin to satisfy
Tier-0 requirements. CASTOR problems served as a motivation for
detailed low-level error analysis and to try to understand
interactions between all m/w layers: FTS->SRM->GridFTP->Storage
Elements. Network tests (GridFTP only) are now ongoing to BNL. ATLAS
suggested WLCG should take over this responsibility: to constantly
monitor state of the network.
Network tests (GridFTP only) are now ongoing to BNL. ATLAS
suggested WLCG should take over this responsibility: to constantly
monitor state of the network.
Problems with FTS doing the SRM handling while transferring. FTS 2.0 has a
DB schema change that allows for decoupling SRM handling from file transfers,
which is a high-priority item on the developer to-do list.
Noticed interference between servers and monitoring tools
(CASTOR and GridView)
dCache problem noticed in Lyon when disk full (dCache was not
counting big blocks allocated by the system for small control files).
The problem has been fixed.
dCache disk full led to many different error messages. This needs to
be addressed.
Similar dCache problem occurred at BNL with disk full because the log
rotation algorithm was not working properly. This reduces the Tier0
throughput as CASTOR transfer slots are blocked to retry the failed operations.
Storage Issues from LHCb (Nick)
Please, check slides.
srm_get (SRM v1) not working properly for dCache since it returns TURL of a
non-staged file. Problem acknowledged and fix being worked on.
Few authorization problems noticed at SARA but this could be a general issue
(please, check slides for details).
Question: Should gsidcap be allowed to be used remotely ?
Answer: The experiments can decide what to do. It is not recommendable to
use gsidcap for remote transfers.
LHCb intends to use gsidcap for implementing pre-staging as a temporary
workaround for the srm_get problem. SRM v2 offers functionality in this
respect. Generic higher level tools could be used.
(Check presentation by Jean-Philippe).
Roberto Santinelli sent the following report after the meeting:
"1 the gsidcap issue we had at NIKHEF (but this was rather a site configuration
problem that prevented LHCb to run reconstruction/reprocessing for many months
at SARA/NIKHEF, see https://twiki.cern.ch/twiki/bin/view/LHCb/DC06Activity
for more information
2. the CASTOR2 issue at CNAF (Angelo Carbone could give
you a fully exhaustive description of this issue although
https://twiki.cern.ch/twiki/bin/view/LHCb/PreviousDC06Problems also gives
an overview of the problem and its solution).
3. At a certain point we also suffered about a slowness of transfers
to/from CERN versus T2 centers and this was also due to a misconfiguration on the
HTAR routing. (see Olof's findings)
https://twiki.cern.ch/twiki/pub/LCG/SCActionList/tier-2_transfers.doc)
4. Very recently, but this has to be confirmed and further investigated,
we are experiencing a problem with dcache centers. It looks like, if under
heavy load, dcache storage system registers the file in the database
(read: in the dcache name-space /pnfs/...) but the replica is not
physically copied on the disk pools.
This problems seems to happen at transfer time (using lcg-utils) and this
is really worrying (if confirmed) because a large amount of data might be
lost (having sometimes only one single replica).
Of course there is a plethora of problems related to the high instability
of the SRM endpoints that brought a very low efficiency on our transfers
(as reported too many times at several bodies) and also a very bad
connection between application area (read:Root) and the middleware it self
(read: SRM) on the format of the turl returned by srm_get that is not
understandable by the application (see root:// rfio:// protocol saga at
CERN Castor2, the gsidcap protocol that wants prepended dcap: in order to
be understood by ROOT, format of the rfio protocol returned by Castor1
at PIC)........but on that last respect Philippe is the most adequate
person for a full exhaustive summary."
Rollout plan (All)
SRM v1 will be supported till required (this is true for CASTOR, dCache and
DPM).
The following test endpoints will be added to the S2 test suite:
dCache:
------
FNAL and DESY already there
BNL
FZK
IN2P3
This will happen within the end of April. All T1 endpoints will be available
on dedicated test hardware. No existing data will be made available.
Newly created data can be scratched. Experiments have agreed on this.
UK sites are interested in testing new versions of DPM and StoRM. Greig
will send new endpoints to Flavia as soon as possible to be included in
the S2 test suite.
New StoRM endpoint available in production at CNAF as of now. This will be
added to the test endpoints.
All available endpoints will be S2 [stress-]tested at least till 15th of May.
Endpoints will be added to FTS pilot testbed for FTS tests at the same time.
All endpoints will be published by the test UI (lxb0728.cern.ch) that is
available to the experiments to test lcg-utils/GFAL.
FTS 2.0 client will be also installed on test UI.
CASTOR2 with SRM 2.2 will be installed at CNAF in the next 2 coming weeks.
This endpoint will also be S2 tested and made available for
FTS/lcg-utils/GFAL testing on both the test UI and the FTS pilot testbed.
(This point has been checked and agreed with the pre-production testbed
coordinators.) All available endpoints can be included in the
pre-production testbed together with all available clients, even if
uncertified. The experiments can test their software against the new
versions of the middleware on the pre-production testbed. This inclusion in
the pre-production testbed can start during the first 2 weeks of May if
steps 1-8 above are concluded.
Tier-2s can use production resources for these tests. Experiments have to
be careful and aware that newly produced data can be lost.
On the pre-production testbed a dedicated LFC is available for tests. Also,
a dedicated BDII will be used.
Sites that install StoRM and join the pre-production testbed should also
make available some CEs with GPFS directly mounted. Such CEs and SEs should
be published properly in the pre-production BDII (CESEBind in particular).
This plan will be carried out and overviewed by the SRM effort at CERN.
Storage Monitoring Tools (Mirco C.)
Mirco reported on the analysis of the tools studied so far for retrieving
storage information that can be useful for experiments. This info can be
made available through the tools that are currently used for monitoring
(dashboards, GridView, SAM, etc.).
It was suggested to make a summary of the information commonly available
The report should be circulated to the VOs for comments
VOs should give input on what they need
A first wrapper should be made available and possibly integrated with
the existing monitoring tools.
Investigation on what is available but not at the moment retrievable
from the various implementations should also be carried out.
Next GSSD meetings
A tentative schedule is published
here.
There are minutes attached to this event.
Show them.
