Choose timezone
Your profile timezone:
WLCG-OSG-EGEE Operations meeting
→
Europe/Zurich
28-R-15 (CERN conferencing service (joining details below))
28-R-15
CERN conferencing service (joining details below)
Description
grid-operations-meeting@cern.ch
Weekly OSG, EGEE, WLCG infrastructure coordination meeting.
We discuss the weekly running of the production grid infrastructure based on weekly reports from the attendees. The reported issues are discussed, assigned to the relevant teams, followed up and escalated when needed. The meeting is also the forum for the sites to get a summary of the weekly WLCG activities and plans
Attendees:
OSG operations team
EGEE operations team
EGEE ROC managers
WLCG coordination representatives
WLCG Tier-1 representatives
other site representatives (optional)
GGUS representatives
VO representatives
Weekly OSG, EGEE, WLCG infrastructure coordination meeting.
We discuss the weekly running of the production grid infrastructure based on weekly reports from the attendees. The reported issues are discussed, assigned to the relevant teams, followed up and escalated when needed. The meeting is also the forum for the sites to get a summary of the weekly WLCG activities and plans
Attendees:
To dial in to the conference:
a. Dial +41227676000
b. Enter access code 0157610
OR click HERE
EGEE/WLCG Operations Meeting, November 20th 2006
Agenda: http://indico.cern.ch/conferenceDisplay.py?confId=7643
Attendees:
OSG ROCs: Absent
OSG GOC: Absent
US-ATLAS: Absent
US-CMS: Ian Fisk
+ EGEE ROCs
Asia-Pacific: Shu-Ting Liao
Central Europe: Marcin
CERN: Alexandre Duarte, Piotr Nyczyk, Veronique Lefebure, Maite Barroso, Stephen Traylen, Oliver Keeble, Judit Novak
DECH: Sven, Andreas, Torsten
France: Osman, Rolf
Italy: Alessandro Cavalli
Northern Europe: Jeff
Russia: Lev Shamardin
South East Europe: Kostas
South West Europe: Kai
UK/I: Philippa Strange, Jeremy Coles
+ WLCG Tier 1 sites
ASGC: Shu-Ting Liao
BNL: Absent
CERN: Piotr Nyczyk, Veronique Lefebure
Fermilab: Ian Fisk
GridKA: Sven
IN2P3: Osman
INFN: Alessandro Cavalli
NDGF: Michael Gronager
PIC: Kai
RAL: Philippa Strange
SARA /NIKHEF: Jeff
TRIUMF: Reda
+ GGUS: Thorsten
+ VOs
Alice: Absent
ATLAS: Absent (Gilbert sent apologizes)
BioMed: Absent
CMS: Ian Fisk
LHCb: Absent
Proactive: Peter Kunszt
Feedback from last meeting's minutes
No comments
---
EGEE Items
+ Grid-Operator-on-Duty handover (05') From ROC SWE (backup: ROC DECH) to ROC CERN (backup: ROC Italy)
Notes:
Due to a problem on the interface between GGUS and SWE system, ticket creation failed when a ticket was assigned to SW ROC.
[GGUS-Ticket-ID: #Array(CIC Helpdesk only)] - CE failure on egeece.ifca.org.es - (IFCA-LCG2)-
[GGUS-Ticket-ID: #Array(CIC Helpdesk only)] - CE failure on pprod03.lip.pt - (PPS-LIP)-
[GGUS-Ticket-ID: #Array(CIC Helpdesk only)] - CE failure on mallarme.cnb.uam.es -(CNB-LCG2)-
[GGUS-Ticket-ID: #Array(CIC Helpdesk only)] - problem at -(UB-LCG2)-
[GGUS-Ticket-ID: #Array(CIC Helpdesk only)] - gCE failure on ce05.pic.es -(pic)-
Once problems were solved, tickets were submitted again for those sites that had still alarm on.
---
TCG decision: gram on the gLite CE
Migrate the applications to condorG or to Globus GT4.
* About the status of CREAM:
Should require one year to achieve production quality.
The interface proposed by the CREAM CE is not a standard yet so can't predict when it will be available.
---
Resource request for Proactive VO
Peter Kunszt:
The Proactive VO asks for a significant amount of resources, starting from November, 27th until at least December, 7th.
More information about this available at the VO id card
This is a VO for integration work in the context of the Core GRid project.
The main objective is to integrate developemnt tools developed in France and Switzerland into the EGEE infrastructure.
The resources are to be used on two demos in integration events.
Maite: Is the request already being addressed by individual ROCs or this is a general request?
Peter: Not addressed yet. The project people does not have experience with EGEE.
Maite: Do you have any resources available by now?
Peter: We may have enough for the Swiss event but not to French one.
Ralph: The request was already forwared to ROC managers.
Please, all ROCs/sites interested contact Peter and Rolf to offer resources.
COPY OF THE REQUEST:
Dear site admins,
The Proactive VO is searching *urgently* supporting sites for a small period of time, from November, 27th to December, 7th,
2006.
This request is supported by the EGEE Project Office.
They need at least 1000 CPUs, no storage except the sandbox, outbound network connectivity from the worker nodes. Short
submission times would be very good.
Further information on the VO itself and its purpose is available from the VO ID card on the Operations portal
(see https://cic.in2p3.fr/index.php?section=vo&page=homepage, selecting the proactive VO in the scroll down list)
or at the VO''s homepage at http://www-sop.inria.fr/oasis/ProActive/ .
Your ROC could certainly help you if you have technical questions. The persons who are on the "cc:" field can give you
more information on VO specific topics.
Thank you for reading this. Any participating site is very welcome.
Cheers,
Rolf Rumler
EGEE OAG co-chair
---
EGEE issues coming from ROC reports
All ROC reports received.
No reports from Biomed
1. What we do with long standing GGUS tickets that are assigned and no progress is observed?
(I mean tickets raised by regional team, not COD on duty) (CE)
Maite: Escalate them to the Operations Meeting by including them in the ROC Report.
2. Regional Certification finished WMS update 10 certification:
http://wiki.grid.cyfronet.pl/RegionalCertification/WMSLB-3.0.2u10
It looks like the WMS is able to accept at most 20 parallel job submissions.(CE)
3. After SFT/SAM tests, it remains empty directories and some files on tested SE. Could it be possible to properly
clean up the SE after the SAM/SFT tests are ended (successfully or not) ?
In the meantime, could we set up a cron to clean up OPS/DTEAM storage spaces on SE ?(France)
Piotr: Some tests copy and register files from the local UI to SEs. There is another test that deletes the previous files
if everything goes ok.
Some files remains in the LFC and we run batch cleaning jobs but it sometimes doesn't work.
It could be a good idea to run cron jobs to delete the files but I'm not sure about the directories.
Conclusion: This only happens when the tests do not finish normally, so it is the site responsibility to clean them
regularly (e.g. with a cron job).
4. Announcement: Please note that in order to increase the electrical power of its computer room, IN2P3-CC should change
its main electrical board. This change implies a complete shutdown from 15 in the morning to 18 afternnoon of december
2006. Only the main public hosted grid services will be maintained during this period: Biomed/Egeode VOMS, Biomed LFC,
Site BDII, Local LFC, FTS. Even for those critical services a shutdown will be necessary (about a couple of hours).
We will keep you informed. (France)
France: All affected services will be hosted at CNAF.
5. Ticket #15284 ("Policy for security updates of third party software") is assigned (7.11.2006) but not in progress yet.
Suggestion was to start discussion via Security Mailing List to come to a conclusion(DECH)
Oliver Keeble: The policy is to maintain what is stated in the Torque distribution but we are doing our best to support
external contributions/patches.
The gLite integration team policy is: the external packages are not guaranteed to be maintained. They are provided for convenience. They are maintained by their providers.
The reality is that they will be maintained on best effort.
Sven: What is the opinion of the secutiry about this?
Oliver: We will make clear our policy and let them decide.
Maite: Open an action so this is clarified with the security team.
6. General question about FCR: several sites show "no result" in SAM for a given VO (e.g. several IT-ROC sites for ATLAS
VO). Is there any problem or is it a regular behavior? Should a ROC take care (= open a ticket) about sites removed
from a VO FCR?(Italy)
Judit: The VOs have to setup their critical tests to see the results.
Alessandro: We only see results from some sites, for other sites the results are Not Available.
Judit: It is a problem with the test submission for Atlas VO.
Maite: How the ROC should proceed about this kind of problems?
Judit: They should contact the Same Development/Support team, through GGUS tickets
7. We have added a new SE to our site, it is in the site BDII and is has now also been added to the GOCDB, however it has
not yet received any SAM ops tests - it has already been receiving SAM dteam tests successfully. What else needs to be
done?(UKI)
Piotr: The best is to submit a GGUS ticket with the hostname of the machine that is not being monitored.
It seems to be related to a possible bug in the bdii to SAM db synchronization script.
Kai Neuffer A similar problem with a Computing Element that was deleted from Goc DB but is still being monitored.
Piotr: Those deleted machines remain in our database for one week. If it remains for more than one week it can be a
synchronization problem.
8. We've got an issue to discuss on todays meeting. It is about the GGUS ticket
https://gus.fzk.de/pages/ticket_details.php?ticket=15682
The issue is critical from my point of view, and I haven't received any feedback about this issue yet. (Russia)
Maite: The support unit has already been contacted and are looking into it (production repository team)
In general, please, Wait a few days before scalating the ticket to teh operations meeting, also try to
comment on the ticket so you get a response. This one was reported last Friday and today is Monday.
---
OSG Items
None
---
WLCG Items
* WLCG Service Commissioning report and upcoming activities (15') (files document doc )
See new and updated information at https://twiki.cern.ch/twiki/bin/view/LCG/SC4ExperimentPlans
* WLCG related Issues coming from experiment VOs and Tier-1/Tier-2 reports (15') (files Tier-1 reports )
Reports were not received from:
> Tier-1 sites: BNL,
> VOs: Alice, Atlas, LHCb
* Site IFCA complains about CMS jobs with are performing a lot of "sleep 120" when they are not able to receive their
data needed to proceed (SWE)
Maite: Contacted CMS but didn't get an answer yet.
Ian Fisk : I believe that this is related to an issue in an outdated User Inteface. Could we get the DN of the user
submitting the jobs se we investigate more?
Maite: Kai can you follow this offline?
Kay Neuffer: Yes.
---
Review of action items
2006-08-14—2:
Maite: No news about that. We will check the current status.
2006-10-05—1:
Osman: It is in our todo list. It is not finished yet but in progress. I think that we can close this task because we have
it in our todo list in the CIC Portal.
Maite: Do you have a estimate on when it will be finished?
Osman: No because it is under Gilles's responsibility.
2006-10-05—2:
This action can be closed after Jeff's comments.
2006-10-16—2:
Maite: I will check offline if it is already done.
65:
In progress. We will check offline and update the status.
67:
Steve Traylen: The counter was restarted last friday, so it will go to PPS in 2-3 weeks from now.
69:
Maite: I will follow this offline after the meeting and solve it.
---
AOB
Sven: It would be good to be able to access the old ROC Reports from the CIC Dashboad, as it was available with the
previous version of the portal.
Maite: Osman do you have any estimation on when it will be available again?
Rolf: We are working on it but we don't have an estimate now.
There are minutes attached to this event.
Show them.