EUChinaGrid workshop, Beijing 12-14 June 2006 ============================================= Summary of issues to be evaluated by PEB ---------------------------------------- EUChinaGrid proposes to EGEE two collaboration activities: 1) interoperability of gLite with the CNGrid middleware GOS 2) IPv6 compliance of gLite C.Grandi and s.Monforte are the contacts for both activities now. 1) interoperability There are several middleware flavours in projects supported by the Chinese Accademy of Schence (CAS). The one with which EUChinaGrid is poposing to build interoperability is GOS by CNGrid. gLite and GOS follow quite a different approach. The majors are: - Access to the infrastructure in GOS is thorugh a portal via username and password. The proxy is upoloaded there for further use. In gLite the user proxy is delegated by specific services at the need and stored on myProxy only if specified - AuthZ of GOS is based on obtaining tokens for a specific resource while gLite is based on VOMS - The choice of the resource is done in GOS on the portal in an automatic way. User cannot specify requirements. In gLite the CE may be accessed directly or via the WMS but with the possibility to specify requirements - The files are stored in GOS on a distributed file system, in gLite on Storage Elements The lower impact interoperability activity identified is to address submission from EGEE to CNGrid considering CNGrid resources as a unique CE and SE. This would imply: - develop a CE with a BLAH plugin that talks to the GOS submission APIs - develop an information provider (GIP) for this CE - develop a system for moving information from VOMS to AGORA - build an SRM interface in front of the GOS distributed file system EUChinaGrid is asking EGEE to collaborate with CNGrid for this. The minimal request is that the work is done by EUCHinaGrid and/or CNGrid, while EGEE provides only consultancy 2) IPv6 compatibility The network calls that may bring to non-IPv6 compliance are well known. It is proposed that - CNGrid produces a short (maximum 2 pages) document with rules and guidelines - CNGrid optionally produces a codechecker that identifies the potential problematic calls in source code - CNGrid optionally produces a list of calls in the gLite middleware (equivalent to the output of the codechecker) - CNGrid produces a description of the possible problems in the major external dependencies of gLite (in particular GT4 and other VDT 1.3 omponents) These are provided to EGEE during the summer and in any case before the EGEE conference JRA1 does a first round of modifications in the codes following the recommendations At the EGEE Conference we discuss the current issues and possibly the results and agree on how to proceed. In the following you can find the notes taken during the meeting for most of the presentations. Unfortunately they are not complete, but give a feeling of the meeting 12/06/06 morning ---------------- * Welcome - Chen, H.(Director of IHEP) * Keynote Speach - Gagliardi F. (Microsoft) Overview of grid projects. Good EGEE presentation. Activities from Microsoft (e.g. multicore CPUs <-> clustering) * EUchinaGRID Project - Ruggieri F. (INFN) 1.3 MEuro over 2 years - interoperation of EGEE and CNGrid - IPV6 - IPV4 interoperability - Applications: EGEE, ARGO YBJ, never born proteins Now 622 MBps link ORIENT will upgrade EU-China (GEANT-CSTNET/CERNET) link to 2.5 Gbps Now testbed has 6 sites Issue: need gLite middleware usable on IPV6 infrastructure * EGEE an European GRID Infrastructure - Grandi C. (CERN/INFN) Got some interest on avian flu challenge - Chinese GRID Projects ----------------------- * e-Science and Grid in CAS - Nan, K. (CNIC-CAS) Descritpion of the China Accademy of Sciences (CAS) activities - SDG (Scientific DataGrid) project Application driven: e-Science virtual labs are the basic unit of research activity based on SDB (Scientific DataBase) project (since 1983) accumulates data from different sciences data distributrd on ~20 chinese sites ~ TB data Rewources: 236 CPU 20 TB disk, 50 TB tape SOA middleware Data Access Services (DAS): interface to information and storage read only interface on a web page. Every site has its ownn server MappingBuilder & Dataview www.sdg.ac.cn - Scientific Computing Grid: supercomputing center 111 Gflops commercial and free application software ~ 300 users on the 2 systems - applications include HEP: YBJ-ARGO, 155Mbps link Beijing-Tibet, building a system based on LCG Avian Birrd Flu Alarming system (integrate DBs, surveys, analysys, alarming, ...) geoscience: East Asia Resource Environment Collaborative Research Network (network + databases integration) disaster reductions, env. science (sim. of Yangtze river, earthquake predictions), ... * NSFC Grid Initiative - Hu, C. (Beihang University) National Science Foundation of China: 3 years project network R&D, middleware and testbed, e-science applications Overview on CROWN project focused on grid middleware, SOA, grid security, grid enabling tools, dist. reasource manag. appl. - middleware suit, portal for China e-science, research Node Server: (a gatekeeper based on GT4) RLDS: info system (similar to Globus MDS, tree-like architecture) gateways t other grids Scheduler: submits requests from users to RLDS (java based application framework) based on JSDL job persistency, hierarchical job scheduling job execution (posix, *ws*, pbs jobs) Monitoring: of all resources CROWN Designer (based on Eclipse) for application and site programming) support for WSRF/WS-I (generate scheleton for java web services Workflow engine based on ...) Portal Security Identity Mapping service to map identities from one security architecture to another (i.e.l not only PKI) Open source in progress - installation based on Installanywhere Testbed (no production level system): several (7?) sites in China, 1 in UK and 1 in Australia Applications: AREM: simulations disrtibuted among sites biological applications (blast, cap3, ...) MDP: massive multimedia data processing (audio & video content recognition) digital sky survey gViz, support remote medical co-diagnoses In future: 2-tiers resource organization (backbone + non-stable resources) add a virtual layer above resource and middleware (via virtual machines?) * CNGrid - Qian D. (Beihang University) may 2002 to dec 2005 - application driven, multilateral cooperation - develop testbed 11.6M funding 8 sites 18 Tflops, 200TB disk - HPC: high perf computers 45M funding developed two >4Tflops machines (Itanium and AMD64 in Beijiong and Shanghai) HPC evcaluation center at Tsinghua Univ. - grid software 12.6 funding GOS middleware running in Unix, Linux, Windows. Everything through http port layered architecture service virtualization GSML grid language environment more details in presentation tomorrow - application grids 24M funding - Resource and environment - NGG National Geological Survey ground water resources evaluation in north China reduced the time of processing from 3 years to 3 months, calculations more accurate mineral resource evaluation - DFG Digital Forestry Grid: turning the farmland back into forrests (evaluation & planning) - CMAG China Metereoligical Appliaction Grid: weather forecasting - Research (SDG, biological appl., ddg, chinagrid) - SDG scientific DataGrid - DDG Drug Discovery Grid: using HPC in p2p mode (parassitic use of unused CPU cycles) - BAGrif Bioinformatics Applied Grid: computing data an knowledge for bioinformatics research - ChinaGrid (see next talk) - Services (weather forecast, intelligent transoprtation of information) - Manufacturing (aviation, simulation for space agency) - AviGrid: Aviation Manifacture Grid: design, simulation and manifacturing of aircrafts etherogeneous env (linux, aix, windows) International cooperations - UK e-science (also OMII-China) - EGEE (FP6) (EUChinaGrid) - Globus Future: - improve GOS software (efficiency, robustnessm user-friendlyness - identify killer applications - management (including security, monitoring - improve network performance (CNGI IPV6 backbone) - adopt a buisness model - policies and regulations Need continuous government support HPC and grid will continue wiht bigger investments more involvement of the industry * ChinaGRID - Jin H. (Huazhong University of Science and Technology) ChinaGrid based on China Education and Research Network Campus grid platform, common platform, application platform overall system: ~ 20 sites, 15 TFlops, 150 TB CGSP middleware follows WSRF, OGSA - first release in 10/1/2005, v2 beta released 25/4/2006 v1 installed as campus grid in all universities and campuses survey of who downloaded the middleware (by region0. A lot form US > 16000 hits for downloading the packages tree-like topology new version will be more complete, documented, robust. CGSP container based on GT4 + packages developed for deployment domain Manager Info Center (IS and visualization) Data Management: Grid File System Exec manager: binary exec, service invok, workflow exec (BPELv4WS1.1) - based on JSDL Heterogeneous DB: virtual to physical table mapping Portal: AG4ChinaGrid. English version much smaller than chinese one... more than 50K hits per day GridPPI: programming interface (MPIU-like API) Security CGSV supervision layer (monitoring) AG4CG (application layer) Applications: bioinfo, ImageGrid, computational Fluido Dynamics course on-line (courses from universities shared by grid User survey: more than 25000 independent IPs, millions of hits massive information processing grid (e.g. digital museums, UDMGrids) 12/06/06 - afternoon -------------------- * Biomedical Applications - Breton V. (CNRS) Summary of EGEE and EGEE applications Biomed achievements in EGEE: - organization and deployment in the first year - Medical Data management, Large scale data challenges (malaria, bird flu) in the second year Applications: medical immage processing, pharmacokinetics, Simulation of MRI images, segmentation of radioligical images, GPS@ bioinformatics portal, electron-micrioscopic images processing Still a few users but big impact on the community Detailed description of DICOM and WISDOM MDM: stress of legal and privacy requirements, data disrtibution description of SRM/dicom, fireman, gliteIO, amga, hydra higher level services rely on DICOM: bronze standard application description of the demonstration done at the review concern for the lack of MDM in gLite 3.0 In silico drug discovery against neglected and emergency diseases reduced cost and accelerating factor for R&D description of research for malaria (WISDOM) and bird-flu Next steps: WISDOM II, HIV, SRAS, bird-flu in EGEE II 13 partners in 13 countries. Need to start providing services (MDM, simulations, data mgm) Comments again on the lack of SDM components in gLite 3.0 (also from other people in the audience). Answered that AMGA and Hydra are going to be certified now and that GFAL and LFC with the needed functionalities are foreseen in the next months and that JRA1 is providing direct support in the meantime. * ARGO-YBJ Data Moving and Analysis - Galeazzi, F. (INFN - Roma Tre) Cosminc ray telescope made of RPCs installed at 4700 mt in YBJ, Tibet reconstruct the shower front, distribution and direction. 200 TB/year of data Data transfer to both China and Italy, and data processing (MC and reco, hopefully user analysis) Only 2 sites (IHEP and CNAF that need the full dataset Data tranfer 7.5 MB/s sustained. Current rate is 2 MB/s. Slow network to IHEP. No network for CNAF (by flight from Beijing) Link to Beijing upgraded to 155 Gb/s in april. China to Europe will be upgraded to 2.5 Gb/s Use grid infrastructure, but simple configuration in YBJ (no computing experts!) prototype based on Gilda testbed. 2 FTS channels per physical channel (only going out YBJ 2 FTS servers and 2 LFC catalogues at IHEP and CNAF Files deleted form YBG only when present in both LFC, otherway start tape backup Data pushed from YBG, pulled from IHEP and CNAF (by each other) if data does not appear at your site but the other site has them, then pull from there Still some troubles in FTS channels set up. Thinking to user AMGA or RGMA for DB data. MC simulation: Corsika (based on fluka) -> experience from MAGIC to build jobs ArgoC (based on G3) Reconstruction: process data asap. Only one person taking care of both sites. Only one DB (AMGA?) replicated at both sites (replica and slave). Critical data are read-only in the slavem non critical data are read/write Need precise error handling in job submission Issues: ARGO VO OK. need multiple acceptance of CA. how many BDIIs RBs? checksum of files at the end of transfer? sync of DBs? ownership of files and FTS channels? need LFN convention, DB structurem SW version and configuration * Bioinformatics & Health Grid Node of China National Grid - Chen J. (Beijing Genomics Institute - CAS) HPC -> algorithm and SW develop -> data analysis and mining - datagrid: genome of rice, chicken and silkworm - computing grid - knowledge grid: share sequences of genome Have now 1 TB data, 10K hits/day, 2000 jobs/day Only 1 DB, 5 sites. OGSA interface for security Need to sequence 100 K sequences/day (50 MB?) * Grid Activity in Taiwan - Chih-Chiang Chang (ASGC) ASGC scope is to develop and dfeploy gid middleware, operate a grid to benefit researchers Using EGEE after testing different flavours of middleware. In NA3, NA4, SA1 and JRA1 in EGEE (?!?) 2.5 GBps link to Geneva, 10 GBPs link to Amsterdam now. Links to Asia-Pacific T2s Participated to LCG SCs (file transfers, now stable at 120 MB/s) In activity since a long time: LCG-0 (march 2003), LCG-1, LCG-2. Now have a VOMS and are part of PPS Are in WLCG (MoU signed in Dec 2005) as a Tier-1 Contribute to ARDA. Other applications: - Grid Application Framework - mpiBLAST-G (parallel gene alignment software run on the grid) - digital archive datagrid - atmosphere datagrid - participated to avian flu drug discovery service challenge Dissemination: - International Symposium on Grid Computing every year - tutorial - printed material (including chinese version of GridCafe) Industrial cooperation - NSC-Quanta Project (evaluate and tune performance of blade systems) - Microsoft collaboration project (port gLite middleware to MS Windows) Operations: - CIC and AsiaPacific-ROC (APROC) - serving AP Tier-2s (Taiwan, Japan, China, Pakistan) - running GStat (Global Information Monitoring System) - ASGCCA (Certification Authority), joined APPMA in 2004 - VO Services for Atlas, CMS, biomed. Deployed a VOMS for TWGridVO, AP...VO - APROC give first line support to GGUS * Round Table and Wrap-up ... Participation of Asia Pacific sites to MWSG and JSPG CA for countries that do not have it in AP region 13/06/06 - morning ------------------ - Workshop on GRID Middleware Interoperability ---------------------------------------------- * gLite Middleware - Grandi C. (CERN/INFN) gLite process status and plans for gLite components * GOS Middleware - Wang Y. (Beihang University) requirements: naming mechanisms, process or state mantain, virtual organization, programming model focus on architecture , not protocols hosting env: tomcat and in future: omii, gt4, websphere, weblogic, .net core: agora (AAA, info) and grip (user info and interaction) high level services EVP address space: effective address space: e.g. eres://service/metaservice physical add. space: e.g. http://... must be used outside GOS virtual add. space: start eith vres:// v <-> p 1:1, e <-> v n:m Security Use x.509 and CA and proxies, SAML tokens for auth, GOSContext SSL/TLS in the trasp. layer, WS-security Axis handler chains. grip applications may use or not security depending on config. message passing: soap msg over SSL/TLS (https) AuthN: Agora: convert username and password to token AuthZ: SAML proxy uploaded to portal via username and password and available to grip user accesses grip with username and password agora provides user manag and AAA user gets access to resources Core services Agora (tomcat + axis) AuthN, AuthZ Grip (tomcat + axis) state info for end user username + pw passed to agora that returns proxy eres name passed to agora that returns vres and token for the chosen resource Router virtual to physical resource mapping router services can talk each other and eschange info Resource discovery find eres, convert eres in vres, convert vres in physical address Application services GFI (Grid File Infrastructure) meta services -> global user file space name mapping from eres to physical global file information, including ACLs data services -> distrib file storage, file transfer as soap message map user id to local file directory store files in local file system file transfer by servler form distributes user file space Grid Batch system local batch system driver (inferface to LRMS) grid batch service batch job description in xml but not JDSL requirements? in future will use gridSAM, support jsdl, support ftp and gridftp jobstates, support openpbs and LSF Grid Accounting system * Inter-operation with EGEE Infrastructure - Fattibene, E. (INFN) Description of EGEE Operations (ROC, GOC) and monitoring proposal for building an operational unit for the China resources * Discussion on Interoperability & Interoperation Proposed to start with the use case of using cngrid resources from EGEE job submission from glite to cngrid consider cngrid as a CE transfer authZ info from VOMS to AGORA build a BLAH plugin for cngrid build a GIP for cngrid develop a system to get the cngrid token other use case (later stage) is data transfer consider the cngrid file system as a SE build an SRM interface to cngrid file system in a second phase try to have the vice versa. Salvo and Claudio are contacts for EGEE now. - Workshop on GRID/IPv6 Compatibility ------------------------------------- * gLite Middleware on IPv6 - Gautier J.P. (CNRS/UREC) IPV6 advandages: addressing (included private networks), enhanced multicast, hierarchic addressing plan includes IPSec security, mobile IP geant2 and tein2 (in asia) are IPV6 compliant. Problem for the "last mile" WG at GGF on IPv6 in the middleware shopuld avoid use of IPv4 hardcoded addresses. APIs are IPv6 compliant in gLite there are several components not IPv6 compliant. [...] * GOS and CNGrid on IPv6 - Wang R. (Beihang University) Plans for portig in place [...] * First tests of gLite/IPv6 compatibility - Carcione V. (GARR) [...] * gLite WMS and IPv6 - Monforte S. (INFN) summary rules for IPv6 compliant network programming presented survey of IPv4 explicit calls in gLite packages may be more difficult for non-gLite software * Discussion on IPv6 compatibility of Grid Middleware (Gabriella) Has IPv6 a future in Grid, given that the middleware is not compliant? (Jean-Paul) Yes! It is pormoted by EU, there are many infrastructures around with IPv6. The middleware needs to be made compliant (Wang R.) ... (Valentino): strong investment in EU on IPv6. Now the middleware has to be made available on IPv6. (Federico): large coutries like China and India are deploying IPv6 becasue the suffer from lack of IP numbers. mobile connectivity is using IPv6. No other way... need to do it now that we are not yet pressed than when we cannot avoid (Gabriella): we should also try to use the new options offered by IPv6. Packet forwarding will be faster (Jean-Paul): Mobility is not a need for grid now but it can be addressed only by IPv6 (Federico): Can we develop a "code checker" that verifies the IPv6 compliance of the code? (Jean-Paul): would be a good idea. nothing available now (Salvo): grep is enough... (Federico): an automatic tool would be useful for developers, also if you need to go to libraries (Salvo): reverse engineering is too tricky (Gabriella): developers must have IPv6 machines. Who is using IPv6? -> no one!!! (Federico): it is good if you don't know! (Gabriella): during last IPv6 forum Microsoft said that VISTA will use IPv6 as main network protocol and some applications will only be IPv6 compliant. (Federico): how many man-months are needed to port the code to IPv6 (Wang): no direct interaction with the network layer in GOS. The DBs are not IPv6 compliant and will change that. (another chinese): 8 man months for a first evaluation. A lot of time is wasted on 3rd party software (Salvo): for the WMS with dependencies, a couple of months including testing. (Roberto): Propose to present something at the EUChinaGrid conference in rome and EGEE conference in Geneva. (Salvo): there are external dependencies (Roberto): documentation and survey for the gLite components (Giuseppe): need to have a plan. First deadline in September. Recipe for migration, some code checker (Claudio): recommendations in a couple of pages + code checker based on grep and/or survey + some advice on external dependencies that we are changing now (VDT 1.3) (Federico): Want a deadline for developers (Jean-Paul): interesting for JRA1 to have guidelines (Gabriella): we needed to: 1. set-up IPv6 in our environment. done but not complete 2. interoperation between middleware w.r.t. IPv6/IPv4 (Valentino): deadline for gLite developers to give first fixed code in september (Giuseppe): let EGEE have the guidelines asap and ask the reaction (Valentino): doing IPv6 at the same time of 64-bits may be disruptive (Claudio): 64-bits is a priority. I think we can do it at the same time, not before (Jean-Paul): There is also SA3 that has to do integration and Testing (Gabriella): change the deliverable to include the guidelines document for developers (Roberto): EUChinaGrid/EGEE-TCG meeting at next EGEE conference to discuss this (Claudio): yes. Also on interoperability. If the work starts immediately after the conference we may have something by the new year. (Roberto): Today is the deadline for having the slot at the EGEE conference -> Federico is going to ask (Federico): 1) guidelines, 2) code checker, 3) survey, +) ask for gLite plan