Minutes of Bild meeting 11.10.2011 CERN: Vlad,Joel,Zoltan,Antonio,Andrei,Andrei,Matvei,Krzis,Mario,Fede,Elisa phone: Adria,Ricardo. Dirac-DiracWeb released Andrei v6 practically ready, running on volhcb13 before, a test transformation should be done. Proposed to prepare a MC workflow This should be then used for testing every new Dirac release. Difference between pre7 and pre8 : some bug fixes. News from other VOS: some fixes from Stephane about the file catalog selection. LHCbDirac releases: Joel: some fixes of transformation,Bkk in recent releases. Dirac lcg bundle: should be re-built to include the new lcg_util API, for python 2.6. Ricardo: lcg-bundle is done with a script. Script provided to Joel and Elisa, they will check the correct versions and prepare the bundle. SLC4: some sites still have slc4 nodes, but they are incompatible with pilots and are banned. Joel only compiles for slc5, not slc4 (the applications are only slc5!) but outside lcg , many sites are still with slc4, so Dirac binaries will still be compiled for slc4 LHCbWEb released as git , to be clearly defined how to back port to SVN tags. New tools to prepare LHCbDirac release will be used after Dirac v6. Adria and JOel will try the procedure with some pre-release, should be tested even before v6 is out. News from IT: Joel: the TEG security group will do the first report at tomorrow's GDB. No news, just an overview of the current situation. ---------------------------------- Reports from developers: Core/Framework Adria. no big news. WMS: Andrei. no big news. DMS: Ricardo: some new tasks in github, with Daniela. Some important changes in the stager logic. github task https://github.com/DIRACGrid/DIRAC/issues/226 other task: fix for the case when a transfer fails because the destination file already exists this in some case blocks the FTS transfer. So, the fix would be to remove the destination. github task: https://github.com/DIRACGrid/DIRAC/issues/227 Removal agent and other data management agents: Krisz working on the way they use proxies, it should work for agent with many threads. Ongoing. https://github.com/DIRACGrid/DIRAC/issues/223 Would go to rx x>0 Elisa: LFC permissions: in DIRAC/Resources/Catalog/LcgFileCatalogClient: new version of the getPathPermissions which fixes the problem of authorization when a user has more than 1 DN. Tested locally on my private installation and pull request done: https://github.com/DIRACGrid/DIRAC/pull/218 -DataManagement/StorageHistoryAgent: improving performance: https://savannah.cern.ch/task/?22885 some margin for optimization querying the StorageUsage service in bulk. -consistency SE vs lfc: the agent has to be adapted to run the checks short after the storage dump creation. For CNAF, as it is created daily, the is no problem. For other sites, it is very important to implement a check on the creation data of the storage dump and run the checks only if the elapsed time is less than a max. threshold. RSS: Mario: stomp is now in the binaries (needed for using ActiveMQ). Mario send a detailed report (see at the bottom of minutes) Vincent: agents with sls sensors ready (needing the new lcg bundle) https://savannah.cern.ch/task/?21239 Production: Fede: re-writing all the modules. They will go to the first or second LHCB dirac version on top of Dirac-v6 https://savannah.cern.ch/task/index.php?23196 ByRunFileTypeSizeWithFlush: https://savannah.cern.ch/task/index.php?23192 Antonio: porting a script by Marco Adinolfi to set data quality Lots of testing needed. Bookkeeping: Zoltan: sent a report before the meeting (see at the bottom of the minutes). General review of tables partition, to be ready for the 11g , to take max advantage of Oracle potentiality. All the optimizations are in Zoltan private area. It would be good to keep this into a safer place. Documentation would be very useful to share knowledge about the Bkk DB. Web portal: Matvey: production management page can be splitted into the LHCb specific and the general one ---- Presentation on DIRAC file catalog, by Andrei. slides provided. Relatively new component in Dirac. How this can be integrated into LHCb CM and which advantages LHCb can profit of it. 2 years ago it was decided to provided inside Dirac a file catalog, needed by communities that didn't have an external FC like the LFC. Dirac FC is similar to LFC. From LHCBDirac point of view, exactly same interface. it could be used in parallel to the LFC. Main features: it is a standard replica catalog. All usual functionality available. Bulk queries optimized, especially optimized for LHCb use case Exploits naming conventions, allows to have a light db footprint Native support for ancestors. Security, can be set to different levels. Interface: standard like LFC. Also web interface. ILC has used it in production for > 1 year now Proposal is to set it in parallel to lfc, and populate it with existing LHCb data. It would only write, and not for reading, unless explicitly specified. Particular care should be taken for the population and synchronization: the sync will be ensured by a failover mechanism. The plan is to first populate it and to test how it performs. Populating it, should not put a big load on the LFC. OPened a Savannah: https://savannah.cern.ch/task/?23352 CHEP abstracts: the twiki is hosting the abstracts and everybody should read them and comment. https://indico.cern.ch/conferenceDisplay.py?confId=155300 --------------------------------------- Reports sent by developers before the meeting Mario: ===> RSS * ConditionDB integrated on ResourceStatusDB, to be tested * NagiosTopology agent deployed * Utility to setup RSS CS for other VOs, ? * Policies rearrangement ( DIRAC / LHCbDIRAC ), ? * Synchronizer refactoring, tested completely ? * ResourceStatusDB/Handler/Client completely re-factored * ResourceManagementDB/Handler/Client completely re-factored * New unit tests written for ResourceStatusDB ( ~3500 asserts ) * Prepared small infrastructure to run tests with arbitrary fixtures, and reporting the results somewhere else. * Prepared small tests ( still working on that ), run as unittests, to analyze code quality metrics ( cyclomatic complexity, lines of code, lines of documentation, global and unused variables, etc.. ). Let's see what is the outcome of this.. * Written a small module for writing SQL statements, instead of hard-coding them on the *DB.py modules. It makes easier to debug, and to write the DB modules. Question for the experts: why such thing is not provided by default by DIRAC ?? * Stomp has been included on the $LHCBDEV externals by Joel. This week we will have updates on the NagiosConsumer task. Zoltan: Tasks: #22892: the method which returns the runs file has to be changed: closed -fix the filetypes -change the query -move the query to the Oracle server side # 23186 review of the oracle tables partitions -ongoing -analyse the tables and choose better partition -first try a a small different, but it required more study #22919 improve the performance of the getFilesWithGivenDataSets -ongoing - the heavy queries optimised, but needs more investigation - MC steps have no visible file type while the file table it is vßisible. BUGS: #87222 Correct counting of events for MC requests that do not have a merging step -closed -only count the last step -performance tuning(oracle stored procedure) -optimised the query #87585 Feicim File dialog window TCK save -closed - The save method did not taken account the TCK filter. - add the TCK info to the method which used by Ganga