Choose timezone
Your profile timezone:
Regional Operations Meeting DECH
→
Europe/Zurich
FZK
FZK
Description
Telephone conference. Details for connecting to the conference were distributed through the regional mailing list.
<HTML>
<HEAD>
<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8">
<TITLE>Minutes for Regional Operations Meeting DECH (January 26th, 200</TITLE>
<META NAME="GENERATOR" CONTENT="OpenOffice.org 2.0 (Linux)">
<META NAME="AUTHOR" CONTENT="Clemens Koerdt">
<META NAME="CREATED" CONTENT="20070126;14410000">
<META NAME="CHANGEDBY" CONTENT="Clemens Koerdt">
<META NAME="CHANGED" CONTENT="20070126;17260000">
<STYLE TYPE="text/css">
</STYLE>
</HEAD>
<BODY LANG="en-US" TEXT="#000000" LINK="#0000ff" VLINK="#800080" DIR="LTR">
Minutes for Regional Operations Meeting DECH (April 20th, 2007)
Attendance:
Sven Hermann <chair>, Clemens Koerdt, Helmut Dres (FZK)
Hans-Gunter Borrmann (Uni Freiburg)
Alessandro Usai (CSCS)
Uwe Ensslin, Yves Kemp, Andreas Gellrich (DESY-HH)
Andreas Haupt, Kai Leffhalm (DESY-ZN)
Günther Duckeck (LRZ)
Christian Peter (ITWM)
Yves Kemp (Uni Karlsruhe)
Renate Dohmen (MPPMU)
Apologies:
(Uni Wuppertal)
(RWTH-Aachen)
(SCAI)
Missing:
(Uni Dortmund)
(GSI)
1. Introduction
Last meetings minutes: no comments
Announcements:
All GGUS tickets should be at least updated on a weekly basis (also if in scheduled downtime!)
SLC3 support will cease this June
MySQL parts are now fully tested (PPS: several local instances, plus the global LFC at PIC run MySQL. Certification: Both flavours of LFC are tested and have permanent installations on the testbed.)
R-GMA/Job Wrapper Tests
recommendation to switch off the Jobwrapper Test (SAM) on all WNs, see link on agenda page
Tomcat Error Check Script for MON box (see link on agenda)
refreshed request to announce maintenance and downtimes in production properly (with broadcast in advance)
Several new Production updates: 19-21 (19: dcache/UI, 20: bdii, 21: yaim) (see also link on agenda page)
PPS update 25, 26 and 27 (mail "Release of new update to PPS: PPS-UPDATE 2n" to project-eu-egee-pre-production-service@cern.ch )
Update 26 with native SL4WN package – found serious bugs. segmentation fault with gridftp transfers, another issue: no clean upgrade path
Still no clear timescale for the release of native SL4 WN package into production (expect at least 3-4 weeks)
EGEE III preparations ongoing with draft proposal circulating
Action Items:
PPS certification (get info system and SAM tests working: #! GSI, ITWM) ->
GSI - no update, ITWM – no visible changes, currently busy with failover of gocdb3 installation, push the PPS system within 2-3 weeks. OPEN.
Dech VO: Supported by all sites in region? -> now only recently certified site LRZ not supporting Dech VO. OPEN.
Regional SFT-Server: done (see link on agenda page). CLOSED.
create a criteria catalog to classify MW update in "major" and "minor" ones to warn site admins regionally in advance, to better advise smaller sites -> expected next meeting. OPEN.
coordination meeting T1-T2 concerning storage/transfers to be organized -> no news yet. OPEN (or contact Andreas Heiss directly andreas.heiss@iwr.fzk.de )
check certification of MySQL components with OCC -> was clarified in one of last ops meetings (see also announcements above.) CLOSED.
Need for not completely dropping perl API (LFC). -> was forwarded to ROC-managers meeting. CLOSED.
2. Round the Sites
CSCS
completed installation of worker nodes
installed everything on SL4 – the SL3 compatible package
so far no segmentation faults experienced
plenty of jobs coming in and no major problems experienced
dCache in production: was announced
tripled number of CPU
by the end of the year: significant further expansion planned
DESY
experienced problems with RB beginning of week
RB became overloaded with huge output sandboxes by a user
standard setup of RB not appropriate, better create different partitions
users can create 'denial of service' attack in this way!
what is the policy?
contacting users?
SH: Would be helpful to discuss this case further. Please open a ticket describing the details for developers in order to make them aware of this.
created another RB dedicated to CMS only with larger partition of sandbox
its possible to contact a RB using aliases
you reach the network server
possible to do job submission via aliases
opting for the natively compiled SL4-WN package SL4 using YAIM
500 jobs usually on system, don't want to play around and test one patch after another
please put more pressure on SA3 to come forward with a decent version
update 21 : done relatively straightforward
minor points: do not want to fill the whole site-info.def for every new update
GRIS starting up with wrong ownership : wrong number of availably resources
should be possible to run a single configuration script and not all of them
problem with VO specific directory (had to hack one script)
SH: important to open a ticket if its fault of developers
PPS: had a look at SL4 WN package
managed to built quattor template (SL4), but not for external use yet!
not yet done functional tests for SL4 (so far only quattor template for DESY specific installation)
we are in contact with working groups on that, but haven't really discussed topic
migration wasn't painful to new version of yaim
Comment: Now requiring java 1.5! (..but SL4 comes with 1.4)
DESY-ZN
everything running smoothly
APEL problem finally solved
APEL provided new RPMs
now working with multiple CEs
recommendation to rerun all tests after update
AG: At DESY-HH we did not save all var/log/messages long enough, which is apparently needed (if you rerun APEL on accounting data) in order to register jobs to VO users using the grid (information received from Dave Kant on APEL accounting problems)
It seem also that some batch systems with local users will experience problems..?!?
FZK
dCache – new patches helped increase stability
RB – currently experiencing troubles, still investigation the reason
assumption: logging and bookkeeping problem
output sandbox OK
status reason not available
currently one should use a01-004-127.gridka.de (lcg-RB-3.0.10-1)
GSI
(no information received)
ITWM
production system running fine
scheduled last production update for the middle of next week
log keeping issues: we keep our var/log/messages for four weeks
no problems detected
LRZ
its first time in this meeting
finally got certified
received first atlas jobs
not yet deployed all our pool nodes – more changes to come
SL4 not a priority because WNs running on SUSE LINUX
using the tarball installation
SH: WN tarball installation based on SL4 already available
should support Dech VO within next week
MPPMU
problems with gLite installation
edg-job-submit working, but problems with glite-job-submit: anybody else using SGE?
AU: I think gLite not supported for SGE* (glite uses blah)
also using the WN tarballs on SUSE OS
middleware server running on Scientific Linux
*There is an SGE working group with mailing list:
project-eu-egee-batchsystem-sge@cern.ch
If you want to join contact: Gergely.Debreczeni@CERN.CH
RWTH
(received
by mail before the meeting)
dCache running stable again
Upgrade to patch1.7.0-34 and correction of configuration fault helped.
Apel Problem: solution is sight
Jobs under VOMS-Group cms:/cms/dcms are (were) not registered as CMS jobs but as DCMS-Jobs.
Torque now supporting secondary Unix-groups for the queue-ACLs. exchange of primary with secondary group for dcms accounts.
jobs running as expected.
control of APEL accounting still to be controlled.
Change active since 23.4. 12:30.
Scientific Linux 4
Test installation of a WN with SL4 (i386) successful (apart from O(5) middleware RPMs
jobs are running fine
now WN with SL4 (x86_64) same dependency problems
still running fine
upgrade of remaining WNs will be done only after request from CMS
SCAI
(received by mail before the meeting)
production running fine
problems with last two PPS updates now solved
was related to new version of YAIM and new parameters
release notes of Update 27 were much improved (thanks to Christoph at DESY)
PPS cluster now running on SLC4 WNs
production release 21 not yet installed
currently trying to setup a new WMS/LB for PPS
experiencing problems with submit “error reading answer from L&B direct server”??
SCAI is asking for clarifications on the issue of running multiple BDIIs on one cluster (EGEE/D-Grid)?
DESY: We are running two top-BDIIs of which one is internal.
Reason:
setup all services independent from CERN
support of experiments which are not LCG VOs
Method: site GIISs have to be registered in BDII. Synchronization to CERN BDII is done by looking up (daily) any changes and then adding those changes to DESY BDII
Questions remaining:
How do people know about the BDII? into from lcg-infosites? (but does not save people from understanding the reasons behind...?!)
Perhaps we should agree on a common policy within region?!!
But a common regional policy whould mean e.g. CSCS supporting all German VOs!
CSCS: you would also support ours then, for us this is no problem
DESY: In general, we want to have a unique grid infrastructure.
At the moment I cannot see what speaks against a separate topBDII
SH: thanks for clarifications!
Dortmund
(no information received)
Freiburg
reinstalled dCache
all files were lost, but since inaccessible since last September not much of a problem
RB problems: jobs were successfully submitted, but remained in status running
AU: perhaps not registered in Logging&Bookkeeping service?
Will reinstall RB from scratch.
on SL4: would prefer the native package
Wonder what is necessary for a dCache pool on a solaris system...
AU: information system is already supported by dCache
Karlsruhe
running again
tested SL4 with tarball version on one node
installation will be done on virtual machines
waiting for new machines to include into cluster in half a year or so
Wuppertal
(no information received)
3.
Feedback to the TCG
skipped
4. COD
last shift was right before Easter and without any significant problems
5. ROC-On-Duty
Handover DESY to FZK
DESY: period was generally quiet. lot of relatively old tickets in the system. tried to asked back for few of them, but got no answer. Lots of GridKa tickets.
FZK: no questions at the moment
SH: Statistics: 15 tickets were created and 19 tickets solved during the last two weeks.
6. AOB
------
AG: Who is going to Hepix and CHEP?
SH: few people from FZK go to Hepix and few others to CHEP
GD: What about possibility of bringing smaller Atlas groups into the grid, that have not the manpower for a complete site administration, but would rather have a local SE only?
setup a site with only SE – such to exchange.
AG: Feel somewhat uneasy about concept. Grid is about giving and taking jobs and this principle would be compromised. Using the advantages without needing to share the local resources. does not scale globally!
GD: Atlas has about 15 groups, a few with local clusters. Wouldn't it be overkill for them to run a whole grid components. And who is caring for twenty nodes or so?
AG. I think it helps a lot. from use cases those small sites proofed to be very useful!
Moreover, storage is the biggest problem at grid sites at the moment. I think it's worthwhile for very small clusters to enter the LCG/EGEE computing grid.
AU: I think concept (of single SE without WNs to access SE) technically problematic with HEP VOs. But for reduced expectations it might be possible...?
SH: would suggest to cut the discussions here for the moment and suggest to continue later
My opinion is that a lightweight version for a middleware installation might be a good thing, but will also ask our European colleagues on their opinion.
Next meeting will be in two weeks from now
</BODY> </HTML>
There are minutes attached to this event.
Show them.