Minutes of Bild meeting 11.10.2011

CERN: Vlad,Joel,Zoltan,Antonio,Andrei,Andrei,Matvei,Krzis,Mario,Fede,Elisa
phone: Adria,Ricardo.


Dirac-DiracWeb released
Andrei
v6 practically ready, running on volhcb13
before, a test transformation should be done.
Proposed to prepare a MC workflow
This should be then used for testing every new Dirac release.
Difference between pre7 and pre8 : some bug fixes.

News from other VOS:
some fixes from Stephane about the file catalog selection.

LHCbDirac releases:
Joel: some fixes of transformation,Bkk in recent releases.
Dirac lcg bundle: should be re-built to include the new lcg_util API, for python 2.6.
 Ricardo: lcg-bundle is done with a script.
Script provided to Joel and Elisa, they will check the correct versions and prepare the bundle.
SLC4: some sites still have slc4 nodes, but they are incompatible with pilots and are banned.
Joel only compiles for slc5, not slc4 (the applications are only slc5!)
but outside lcg , many sites are still with slc4, so Dirac binaries will still be compiled for slc4
 
LHCbWEb
released as git , to be clearly defined how to back port to SVN tags.

New tools to prepare LHCbDirac release will be used after Dirac v6. Adria and JOel will try the procedure
with some pre-release, should be tested even before v6 is out.

News from IT:
Joel: the TEG security group will do the first report at tomorrow's GDB. No news, just an 
overview of the current situation.

----------------------------------
Reports from developers:

Core/Framework
Adria. 
no big news.

WMS:
Andrei.
no big news.

DMS:
Ricardo:
some new tasks in github, with Daniela. Some important changes in the stager logic.
github task https://github.com/DIRACGrid/DIRAC/issues/226 

other task: fix for the case when a transfer fails because the destination file already exists
this in some case blocks the FTS transfer. So, the fix would be to remove the destination.
github task: https://github.com/DIRACGrid/DIRAC/issues/227 

Removal agent and other data management agents: Krisz working on the way they use proxies, it
should work for agent with many threads. Ongoing. https://github.com/DIRACGrid/DIRAC/issues/223
Would go to rx x>0

Elisa:
LFC permissions: in DIRAC/Resources/Catalog/LcgFileCatalogClient: new version of the getPathPermissions
which fixes the problem of authorization when a user has more than 1 DN. Tested locally on my private
installation and pull request done: https://github.com/DIRACGrid/DIRAC/pull/218
-DataManagement/StorageHistoryAgent: improving performance: https://savannah.cern.ch/task/?22885
some margin for optimization querying the StorageUsage service in bulk.
-consistency SE vs lfc: the agent has to be adapted to run the checks short after the storage dump
creation. For CNAF, as it is created daily, the is no problem. For other sites, it is very important 
to implement a check on the creation data of the storage dump and run the checks only if the elapsed 
time is less than a max. threshold.


RSS:
Mario: stomp is now in the binaries (needed for using ActiveMQ). 
Mario send a detailed report (see at the bottom of minutes)
Vincent: agents with sls sensors ready (needing the new lcg bundle)
https://savannah.cern.ch/task/?21239 


Production: 
Fede: re-writing all the modules. They will go to the first or second LHCB dirac version on top of 
Dirac-v6 https://savannah.cern.ch/task/index.php?23196

ByRunFileTypeSizeWithFlush:  https://savannah.cern.ch/task/index.php?23192

Antonio:
porting a script by Marco Adinolfi to set data quality
Lots of testing needed.

Bookkeeping:
Zoltan: sent a report before the meeting (see at the bottom of the minutes).
General review of tables partition, to be ready for the 11g , to take max advantage of Oracle
 potentiality.
All the optimizations are in Zoltan private area. It would be good to keep this into a safer place.
Documentation would be very useful to share knowledge about the Bkk DB.



Web portal:
Matvey:
production management page can be splitted into the LHCb specific and the general one

----
Presentation on DIRAC file catalog, by Andrei.
slides provided.
Relatively new component in Dirac.
How this can be integrated into LHCb CM and which advantages LHCb can profit of it.
2 years ago it was decided to provided inside Dirac a file catalog, needed by communities that 
didn't have an external FC like the LFC.
Dirac FC  is similar to LFC.
From LHCBDirac point of view, exactly same interface. it could be used in parallel to the LFC.
Main features:
it is a standard replica catalog. All usual functionality available.
Bulk queries optimized, especially optimized for LHCb use case
Exploits naming conventions, allows to have a light db footprint
Native support for ancestors.
Security, can be set to different levels.
Interface:
standard like LFC. Also web interface.
ILC has used it in production for > 1 year now

Proposal is to set it in parallel to lfc, and populate it with existing LHCb data.
It would only write, and not for reading, unless explicitly specified.
Particular care should be taken for the population and synchronization:
the sync will be ensured by a failover mechanism.
The plan is to first populate it and to test how it performs.
Populating it, should not put a big load on the LFC.
OPened a Savannah:
https://savannah.cern.ch/task/?23352


CHEP abstracts:
the twiki is hosting the abstracts and everybody should read them and comment.
https://indico.cern.ch/conferenceDisplay.py?confId=155300

---------------------------------------


Reports sent by developers before the meeting

Mario:
===> RSS
* ConditionDB integrated on ResourceStatusDB, to be tested
* NagiosTopology agent deployed
* Utility to setup RSS CS for other VOs, ?
* Policies rearrangement ( DIRAC / LHCbDIRAC ), ?
* Synchronizer refactoring, tested completely ?

* ResourceStatusDB/Handler/Client completely re-factored
* ResourceManagementDB/Handler/Client completely re-factored
* New unit tests written for ResourceStatusDB ( ~3500 asserts )
* Prepared small infrastructure to run tests with arbitrary fixtures,
and reporting the
   results somewhere else.
* Prepared small tests ( still working on that ), run as unittests, to
analyze code quality
   metrics ( cyclomatic complexity, lines of code, lines of
documentation, global and
   unused variables, etc.. ). Let's see what is the outcome of this..
* Written a small module for writing SQL statements, instead of
hard-coding them on the
   *DB.py modules. It makes easier to debug, and to write the DB
modules. Question for the experts:
   why such thing is not provided by default by DIRAC ??
* Stomp has been included on the $LHCBDEV externals by Joel. This week
we will have updates
   on the NagiosConsumer task.

Zoltan:
Tasks:
#22892: the method which returns the runs file has to be changed: closed
-fix the filetypes
-change the query
-move the query to the Oracle server side

# 23186 review of the oracle tables partitions -ongoing
-analyse the tables and choose better partition
-first try a a small different, but it required more study

#22919 improve the performance of the getFilesWithGivenDataSets -ongoing
- the heavy queries optimised, but needs more investigation
- MC steps have no visible file type while the file table it is vßisible.
  
BUGS:
#87222 Correct counting of events for MC requests that do not have a merging step -closed
-only count the last step
-performance tuning(oracle stored procedure)
-optimised the query

#87585 Feicim File dialog window TCK save -closed
- The save method did not taken account the TCK filter.
- add the TCK info to the method which used by Ganga