Operations meeting

Name: Operations meeting
Start: 2005-01-31T14:00:00+01:00
End: 2005-01-31T16:00:00+01:00
Location: CERN

Monday 31 Jan 2005, 14:00 → 16:00 Europe/Zurich

CERN

Ian Bird

Description

Phone: +41 22 767 7000 \"Grid Operations/Ian Bird\"

Hide

Weekly Operations' Meeting 2005-01-31

Agenda: http://agenda.cern.ch/fullAgenda.php?ida=a045849
Contact: project-lcg-gda@cern.ch
Operations' Manual: http://lcgdeploy.cvs.cern.ch/cgi-bin/lcgdeploy.cgi/lcg-docs/EGEE-CIC-Operational-Manual/opMan.pdf

Nick Thackray was ill. Ian Bird chaired this meeting.

CIC reports:

CIC-on-duty report (Piotr Nyczyk from CERN):
A full report is in the CIC-on-duty log, starting as [2005-01-31 15:32] - Piotr Nyczyk 9654. To navigate from the entry page http://cic.in2p3.fr, click on CIC views, On Duty Dashboard, FollowUp, FollowUp using LogFiles. This is the last entry on page 01/2005 : html

- The week started with 65 problematic sites, 48 were still bad at the time of the meeting. However, 16 were down due to pre-announced service changes. Overall, during the week, only 3 sites improved, the CIC-on-duty sent 16 emails and made 1 phone call.
- There will be a meeting shortly with savannah developers on statistics and some application interface like lapse time between ticket Open/Close.

CNAF CIC report (Alessandro):

IN2P3.FR CIC report (Helene Cordier and Frederic Schaer on the phone):

From pierre.girard@in2p3.fr Thu Feb  3 10:01:43 2005
Date: Mon, 31 Jan 2005 11:28:45 +0100
From: pierre girard 
To: Nicholas Thackray 
Cc: project-egee-roc-managers@cern.ch, project-eu-egee-cic-managers@cern.ch
Subject: Status of ROC France 24/01->31/01

Hi everybody,

Here's the status of ROC France for the past week:

- Number of bad sites at the start of the week: 3/8 sites

24/01->26/01: CCIN2P3-LCG2 :  Replica manager failed because of disk 
full on SE. As Atlas VO made a cleanup in this SE, the problem has been 
solved by itself.
24/01->NOW:   Both CEA-DAPNIA-SACLAY and IN2P3-CPPM sites are in setup 
step, and then should not be monitored for the moment.

- Number of bad sites at the end of the week: 4/8 sites

* both CCIN2P3-LCG2 and IPSL-IPGP-LCG2 sites are in scheduled downtime 
for upgrade to LCG2_3_0
* Both CEA-DAPNIA-SACLAY and IN2P3-CPPM still in setup stage.

- Major operational issues encountered

* Problem with RGMA client installation on all sites which had upgraded 
to LCG2.3.0

- Upgrade to LCG2.3.0 :

All french sites are either already in LCG2.3.0 or currently in upgrade 
maintenance.
Then, all french sites should be in LCG2.3.0 next week

- AOB

CCIN2P3 will add ~300 bi-pros (SL3) by upgrading to LCG2.3.0
CCIN2P3-LCG2 will change its site name for IN2P3-CC.

Cheers

Pierre

-- 
______________________
Pierre GIRARD
Grid Computing Team Member
IN2P3/CNRS Computing Centre - Lyon (FRANCE)
http://cc.in2p3.fr
Tel. +33 4.78.93.08.80 | Fax. +33 4.72.69.41.70 | e-mail: pierre.girard@in2p3.fr

UK CIC report (Philippa Strange):

From P.J.Strange@rl.ac.uk Thu Feb  3 09:55:29 2005
Date: Mon, 31 Jan 2005 12:42:48 -0000
From: "Strange, PJ (Philippa)" 
To: Nicholas Thackray ,
    project-egee-roc-managers ,
    project-eu-egee-sa1-cic-on-duty
    
Cc: project-eu-egee-cic-managers 
Subject: UK ROC Report 31/01/05


    Please find below a summary from the UK ROC:

     +  Most sites are now upgrading to 2.3 and or SL3
     +  Our sites at SL3 have not seen much use by VOs while the RH7.3 services are much fuller
     +  We continue to have firewall issues at several sites
     +  An upgrade of the BDII hardware has seen an improvement in test results for UKI
     +  Two sites would like to change their host names for various reasons but are being discouraged from this at the moment. The whole area of SEs getting full raises several issues that need to be resolved - the ability to replicate data stored at a site to another site and the implementation of volatile storage
     +  Many sites are not publishing accounting data but this is expected to improve as sites move to LCG 2.3 where APEL (the accounting package) is integrated. All ROC managers are requested to remind sites that accounting records must be kept. Details of what is required can be found here:

SW Europe ROC report (Gonzalo Merino):

From merino@pic.es Thu Feb  3 10:01:27 2005
Date: Mon, 31 Jan 2005 11:06:44 +0100
From: Gonzalo Merino 
To: Nicholas Thackray 
Cc: project-egee-roc-managers ,
    EGEE SWE 
Subject: SWE Status report 24-31 Jan 2005

Hello,

This is the Status Report for the SWE Federation RCs during the week
24-31 January:

Summary:

  -Sites where upgrade to LCG-2.3 is still not finalised [5/12]:
    CIEMAT, CNB, IFCA, IFIC, UB

Sites detail:

CESGA:
  -LCG version: lcg-2.3
  -Status:
   +A new site has been installed with LCG2.3 and SL. The data for the
machines in the CESGA-EGEE site in the GOCDB have been changed accordingly.
   +The plan is to move the WNs from the old LCG2.2-rh7.3 site to the
new one once this passes the configuration tests.
   +This new site is still under scheduled downtime and is currently
being tested.

CIEMAT:
  -LCG version: still lcg-2.2
  -Status:
   +Last week ok.
   +The site has been contacted and asked to report about the satus of
the upgrade to lcg-2.3. This activity is ongoing and the site foresees
to be at the new version today or tomorrow.


CNB:
  -LCG version: still lcg-2.2
  -Status: Last week ok.
   +The site has been contacted and asked to report about the satus of
the upgrade to lcg-2.3. This activity has been delayed until now because
they were wating for new hw to arrive. The upgrade should happen now asap.

IFCA:
  -LCG version: ? (currently upgrading to 2.3)
  -Status: All week down. Still down.
   NOTE: GOC DB Scheduled downtime, not up to date:
2005-01-24 11:53:14	2005-01-22 00:00:00	2005-01-28 00:00:00	Major
network changes and update

IFIC:
  -LCG version: still lcg-2.2
  -Status:
   +Last week ok.
   +The upgrade to lcg-2.3 has been delayed until now mainly because a)
the site was busy with DC jobs and b) the torque-maui new scheduling
system was being tested in an independent environment.
   +Today the upgrade of the site to lcg-2.3 keeping rh7.3 will be done.

INTA:
  -LCG version: lcg-2.3
  -Status: The job submission test have failed this weekend. Site will
be contacted.

LIP:
  -LCG version: lcg-2.3
  -Status: All week ok.

PIC:
  -LCG version: lcg-2.3
  -Status: Last week ok.

UAM:
  -LCG version: lcg-2.3
  -Status: Upgraded last week and is now ok.

UB:
  -LCG version: lcg-2.2
  -Status:
   +The upgrade to lcg-2.3 has been delayed until now because they were
waiting for an installation of new air conditioning system in the site.
This installation seems to be delayed, so the upgrade to lcg-2.3 will
happen now. The pbs queues have been closed and drained during the
weekend to be able to proceed with the upgrade.


UPV:
  -LCG version: lcg-2.3
  -Status: Last week the job submission problems were solved. Now ok.

USC:
  -LCG version: lcg-2.3
  -Status: Last week ok. This morning's TestZone reports job-list-match
failure. Site will be contacted to investigate the origin of this.

-- 
Gonzalo Merino (merino@pic.es)
Port d'Informació Científica, PIC
Edifici-D, Universitat Autònoma de Barcelona
08193 Bellaterra (Barcelona) SPAIN
Tel: +34 93 5813322 / Fax: +34 93 5814110

SEE ROC report (Kostas Koumantaros):

From kkoum@grnet.gr Thu Feb  3 09:55:12 2005
Date: Mon, 31 Jan 2005 13:25:34 +0200
From: Kostas Koumantaros 
To: Nicholas Thackray ,
    project-egee-roc-managers ,
    project-eu-egee-sa1-cic-on-duty
    
Cc: project-eu-egee-cic-managers 
Subject: SEE ROC Report 31/01/05

Hi,

Here is the SEE report, apologies for the delay.

Thanks

Kostas

********

- Tests results 2005-01-24
Bad: CY-01 (tracking the problem), ROGRID-ICI (tracking the problem),
TAU (some upgrades), WEIZMANN (upgrade)

- Tests results 2005-01-31
BAD: BG01-IPP (CT), CY01-LCG2 (CT), WEIZMANN-LCG2 (Network problems)

MAINTENANCE: TAU-LCG2 (Upgrading to LCG2-230)

OK: GR-01-AUTH, HG-01-GRNET, ROGRID-ICI


BG01-IPP: is having some recplica managment problems, its site admins 
are lookin into them, propably fixed today.

CY-01: is having some recplica managment problems, its site admins are 
lookin into them.

- All sites 2.3.0; TAU upgrading now
- HG-01 supports BioMed (gate), and also MPICH, supporting network tests
- 1 new site in BG (BG-02-IM), about to be integrated before the end of 
this weeek.

CE ROC report (Marcin Radecki):

From radecki@agh.edu.pl Thu Feb  3 10:01:36 2005
Date: Mon, 31 Jan 2005 11:09:12 +0100 (CET)
From: Marcin Radecki 
To: project-egee-roc-managers 
Cc: project-eu-egee-cic-managers 
Subject: Status ROC Central Europe 24-31.01

Dear All,

Please find below the report.

 Status of the Central Federation:
 =================================

 Total: 8 sites
 Status at the 25.01: 4 ok / 2 with alerts + 2 maintenance
 Status at the 31.01: 7 ok / 1 with alerts

 6 sites running LCG 2.3.0, 2 sites are on LCG 2.2.0

 Failure reasons:
 ================

 1. Two sites had transitory Replica Manager problems (purple
 alert), even if not changed anything.

 2. POZNAN-LCG site is running experimetal 64-bit installation, it's based
 on LCG_2.2.0, but still has problems with Replica Manager (purple alert).
 Working on that.

 3. WARSAW-LCG reported that functional tests reached timeout once due to
 full pbs queues.

 Maintenance reasons:
 ====================

 1. Maintenance at one site (24-26.01) was caused by air conditioning
 system failure, the second site has been finalizing LCG version upgrade.
 (25-26.01)

 Regards,
 Marcin Radecki

NE ROC report (Jules Wolfrat):

From wolfrat@sara.nl Thu Feb  3 09:55:38 2005
Date: Mon, 31 Jan 2005 11:51:10 +0100
From: Jules Wolfrat 
To: "project-egee-roc-managers (EGEE ROC managers and alternates)"

Cc: per@pdc.kth.se, jules.wolfrat@sara.nl
Subject: Status ROC NE

Nicholas,

NE sites:

HPC2N : nothing to report
NIKHEF: RB problems due to BDII problem and host certificate expiration
NSC: Open ticket (#1561) on missing ACL of MON BOX. Problem solved but 
just noted that the ticket has yet to be closed.
SARA: CE problem since Saturday 2:21 am because of DIMM giving problem on CE node, DIMM will be replaced.

Per and Jules

Russia report (Alexander Kryukov at CERN):

From kryukov@theory.sinp.msu.ru Thu Feb  3 09:54:59 2005
Date: Mon, 31 Jan 2005 17:33:19 +0100
From: Alexander Kryukov 
To: egee-roc-managers 
Cc: egee-cic-managers ,
    egee-cic-on-duty 
Subject: Update: Draft status report 31/01/05

Hi,

Below the updated status of Russian RC's

                       -- Alexander

=================================

Russian status report on 31 Jan. 2005.

RC's status
=================
(Y) - Yellow dot on the monitoring page
(G) - Green dot on the monitoring page 

SINP(Y)
1) Upgrade to SLC-3+LCG-2.3.0
	- RB still LCG-2.2.0/RH	1) RH-7.3+LCG-2.3.0
	- RB still LCG-2.2.0/RH-7.3
	- UI still LCG-2.2.0/RH-7.3
2) globus job run is working well
3) edg-job-submit - faild
	- ???
4) YAIM does not make everything correctly
5) kickstart+dhcp installation SLC does not configure host name
correctly
	- hostname - non full qualified domain name
	- /etc/hosts - the same
	=> does not work host-based authentication for ssh. 
6) We are preparing for installation of RLS.

IHEP(G)
1) RH-7.3+LCG-2.3.0
	- PBS + AFS
2) GOC monitoring: B-->G
3) Problem "Black hole" on the monitoring GOC map was solced by changing 
	ldap://lcg15.sinp.msu.ru:2170 --> ldap://lxn1189.cern.ch:2170
		* Disappear message "Site Deployment Information:  alert_history "
			http://goc.grid.sinica.edu.tw/gstat/ce001.m45.ihep.su/
		A.K> We have to investigate this point more carefully. 
		Other sites use SINP LDAP server without problem.
4) R-GMA problem on the WN. /opt/edg/sbin/test/edg-rgma-check:
	| 
	| 
	| 
	| cannot service request, client hostname is currently being
		blocked
			~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	| 
	| 
		
		* On the CE, SE, UI - OK.
		* WN has a outbound connection via NAT 194.190.165.210 
			(m45-ext.ihep.su)

ITEP(G)
1) Upgrade to RH-7.3+LCG-2.3.0
	- Partially LCFGng, partially YAIM
2) GOC monitoring: Y-->G dot:
	- Change BDII from lcg15.sinp.msu.ru to lxn1189.cern.ch

KIAM(G)
1) RH-7.3+LCG-2.3.0
2) Preparing documentation for installation of LCG-2.3.xin Russian
	
RRC KI(Y)
1. RH-7.3+LCG-2.2.0
2. Experiments with QUATTOR. Not finished yet.

PNPI(Y)
1) Install VO server for North-West region of Russia
	- Preparinf web site
2) The PNPI site does not include in TestZone BDII yet.

IMPB(Y)
- No reply yet

JINR(G)
1) RH-7.2+LCG-2.2.0
	- try to install torque+MAUI
		* AFS
		* for PBS the problem was solve successfully
2) Hardware problem with some WN's - fixed
3) there was MDS problem on the CE.
	- HD space (/var) was exhausted - fixed
4) LHCb is working successfully. It occupied 50% of RC permanently.

General points.
==================

1. Bad network connection
	- A lot of problem with ISP during lase 2 weeks
		* Mostly - instability connection

2. YAIM is a good solution but can not consider as a replacement of
LCFGng or similar tools

3. PNPI (GIIS: cluster.pnpi.nw.ru:2135, Name: pnpilcg2) does not include
in BDII either SINP BDII or CERN.

Open questions
===============
1. What is a status 2.3.0
	- Plan of issued of 2.3.1 with bug fixes
	
2. Consistences SLC-3.0.3 and LCG-2.3.0

3. Compatibility between LCG-2.2.x and 2.3.x services
	- RB(2.2.x)+CE(2.3.x)
	- RB(2.3.x)+CE(2.2.x)

4. Single entry points for bug reports
	- ROLLOUT
	- Savannah
	- GGUS
	- What else?

DE & CH report (Sven Kassemann, Sven Hermann):

From Sven.Hermann@iwr.fzk.de Thu Feb  3 10:01:09 2005
Date: Mon, 31 Jan 2005 10:24:16 +0100
From: Sven Hermann 
To: Nicholas Thackray 
Cc: project-eu-egee-cic-managers ,
    project-egee-roc-managers 
Subject: Status ROC D/CH

Hello,

here's the status of ROC D/CH for 24.-31.1.2005.

Regards

Holger and Sven

*********************************************
Status GermanySwitzerland:

- Start of the week: 1 + 2 + 4 bad sites Monday 24.01.2005
- End of the week: 2 + 1 + 3 bad sites Monday 31.01.2005
- 1 + 3 sites remained bad through the week (1: Status purple - LCG 2.3. 
3: Testing and scheduled maintenance)
- Testing of new sites ongoing (Fraunhofer ITWM, CSCS-LCG2, HU-BERLIN)
- 6 of 10 sites upgraded to LCG 2.3 but not all up and running.
   The other 4 sites are currently working on the migration.
- HU-Berlin, ITWM und CSCS are in the list for D/CH now.


-- 
Dr. Sven Hermann                                 sven.hermann@iwr.fzk.de
Forschungszentrum Karlsruhe                        Tel.: +49-7247-828632
Institute for Scientific Computing / Inst. f. Wissenschaftliches Rechnen
Hermann-von-Helmholtz-Platz 1,  76344 Eggenstein-Leopoldshafen,  Germany

Security report (Ian Neilson):
There was a root compromise on the PBS batch machine at the Royal Holloway site, still being investigated. ROCs should still send the security information of newly registered sites in their region to support-lcg-deployment@cern.ch, for us to update the list of security contacts that we maintain.

Next CIC-on-duty:

The CIC-on-duty this week is RAL.
The actual work is be done by the ROCs and the follow-up by the CICs. The CIC-on-duty just does the monitoring.
The ROC managers should remember that when a problem at a site is solved the site needs to go through a period of quarantine before joining production again. This is mentioned in the Operations' Manual.
The CIC site (requires personal certificate loaded) is http://cic.in2p3.fr .
Link to the daily Site Status reports.
Savannah should be used for the Tasks' hand-over and follow-up. The Operations' Manual should be the basis.
When the CICs report also as ROCs, they should do this in separate reports.
Sites' status of updates can be checked in the daily reports e.g. http://cern.ch/lcg-testzone-reports/cgi-bin/lastreport.cgi

Action List:

(*** ACTION 2004-11-29--1 ***)Vincent Breton to check with the other application managers the "Migration to SLC3 plan" of non-HEP VOs. OPEN

(*** ACTION 2004-11-29--5 ***) Steve Traylen, with help from Laurence Field to document on the Wiki page for the sites on how to block users, when necessary. OPEN

(*** ACTION 2004-12-06--1 ***) Sites should accelerate migration to SLC3, at least on the service nodes due to security considerations. OPEN

(*** ACTION 2004-12-13--1 ***) Escalation procedures in the Operations' Manual should clarify that sites running outdated LCG2 versions or don't respond to CIC-on-duty action prompt will be disclosed in this meeting. OPEN

(*** ACTION 2004-12-13--2 ***) Savannah accounts were created for all the ROCs. New users had to activate these accounts but as some failed to do it in time, their accounts have now expired. Now A. Kryukov has to re-create the Russian ROC account(s). OPEN

(*** ACTION 2004-12-13--4 ***) Nick Thackray to send email to the 3 relevant ROCs about Helene Cordier's reminder to UK, IT and RU to submit their test suites as agreed in The Hague. OPEN

(*** ACTION 2005-01-03--1 ***) Grid Infrastructure Support Section to give generic (functional) DNS aliases to important service nodes, ensuring transparent service changes. OPEN

(*** ACTION 2005-01-10--1 ***) Frederic Schaer will enter a bug in savannah, project=lcgoperation, requesting a proxy at site level protecting from asynchronous updates of the CA list on various nodes at a site (RB, WNs etc) that now cause credential verification failure. OPEN

(*** ACTION 2005-01-10--2 ***) Laurence Field or Markus Schulz to prototype a dump of the GOC database every few hours and provide a read-only copy for the community. OPEN

(*** ACTION 2005-01-10--3 ***) The asian sites' support needs to be formalised. Requested by CERN at the relevant CIC-on-duty report. Not assigned to anyone. OPEN

AOB

The next regular telephone meeting will take place on Monday Febuary 7th 2005 from 14:00 to 16:00 in conference room 28-R-015.

Maria Dimou, IT/GD, Grid Infrastructure Services

There are minutes attached to this event. Show them.

- 14:00 → 15:00
  
  Operations reports 1h
  
  CERN France Italy UKI Germany/Switzerland Northern Central South West South East Russia
- 15:00 → 15:15
  
  Security Issues 15m
- 15:15 → 15:30
  
  AOB 15m