Hackathon: Broadly topics discussed were: 1) Main goal was to discuss the RUCIO codebase in a little more detail among people in the room Covered the general code structure within the project, some conversation about how the different components interact. 2) Some questions about coding styles, testing and general code quality 3) Some discussion started about how the tests are structured after looking into the test directory. Andrew mentioned that some of the tests have hidden dependencies as well as some test periodically failing or potentially failing to run when repeated against the same DB instance. Teng highlighted that after running the tests against the RUCIO instance for DUNE it left a lot of files in a live system. 4) It was suggested one of the first pieces of work for Wenlong would be to look into the teardown process of the tests to make them clean up after themselves more completely. This seems well defined and would be a good starting point 5) There was a discussion about which DB instance to use for production and why chose different technologies. Ian said that from the developers and experience that the choice of DB is mainly down to scalability and prostgresql is favoured with this in mind 6) There was a brief summary of how the deamons work and how they can be launched as one-shot processes so run on demand. 7) There was some discussion about how FTS and RUCIO communicate and how FTS permissions work. I(Rob) am interested in having some documented examples of how to manually run transfers using all of the protocols available as a reference as well as how to use FTS manually. Ian suggested that it should be possible to just use the FTS hosted at RAL for any/all of our needs. 8) There was some discussion about what RUCIO does and how it compares to DIRAC. This led to a question of what can be done to support RUCIO/DIRAC integration at Imperial. Curious to know the timeline for first test instance and we should probably contact Imperial to offer RUCIO support. Discussed that at the DIRAC workshop at Imperial the feeling was that full RUCIO/DIRAC integration might be a further away then 6 months but we discussed whether they would be using 1 RUCIO instance per VO or would they consider using the RAL multi-VO instance. If they'd want to use the latter it's not clear at this stage if anything is required from the RAL installation. If they need any RUCIO changes they may need help getting these back into mainline, I think people said they could potentially offer support there but not on the DIRAC side, 9) I (Rob) probably managed to confuse people asking questions as I was walking through the permissions model in RUCIO for file management. (trying to understand this from the perspective of a Tier2 admin) After some discussion I managed to understand how it compared to other storage management systems and how data is managed at sites as a result. 10) There was some discussion about how Edinburgh has an interest in working toward features which are good for all users of RUCIO multi or single VO but we will treat DUNE as a working example or use case when trying to develop new features. Talks: Rob: Discussion after the talk about what we mean by different sized VOs and that we only envisage VOs managing huge amounts of data to deploy their own RUCIO instances. There were some questions about the dev instance of RUCIO at Edinburgh and I clarified that there is interest in having a dev instance which is updated quite frequently which is used for trying to transfer files would be one goal of setting this up. Ultimately setting up a test instance is more of a training exercise in how RUCIO works, how it's setup, permissions, etc. There was some discussion of what work Wenlong can do when he starts in Edinburgh in late August. It was suggested that Edinburgh might be in a position to work with DUNE developers approaching metadata catalogue from the physics perspective so it would be useful to see how this would integrate with RUCIO metadata and what is possible or what is missing that can potentially be added to RUCIO. It was mentioned that Wenlong might first look at how the current system scales and just test how well the current technology works but that we'd likely approach developing this from a DUNE first perspective with an eye to supporting other VOs. If SKA have any clear requirements for metadata support within RUCIO it would be useful to compare these with DUNE and potentially ATLAS/CMS to work out what is common and where work is potentially needed. Not overly clear who to talk to on the various experiments if we want to explore this but we should consider who to talk to before Wenlong starts. Teng: Discussion about how the differen components work and what chages may potentially have to be made to support multi-VO monitoring. Changes at the point of ingress are potentially needed. James: Discussion about how the signing of URLs works as well as future plans for this work. Andrew: Discussion on the amount of changes that need to be approved for this work to be completed as well as what is required to make RUCIO support multi-VO. Some questions about experiences in using docker to develop/deploy. Some questions about potential multi-VO RUCIO support post September. Ian: Discussion about uptake of RUCIO by new communities or the alternative technologies which they may select over RUCIO. (Good to understand why not RUCIO when they don't chose it.) Some discussion about how to deploy RUCIO into production as well as some questions about how the rucio-auth handles security. Discussion: Some further discussion about what testing is done by RUCIO devs as part of release/deployment testing cycles. Pete was keen to emphasise how RUCIO and work to help new/smaller VOs was very welcomed by funding councils and how this is good work we should continue. There was some discussion about different meeting where UK-RUCIO work should be presented: - I(Rob) suggested GridPP43 (agenda/timetable permitting). - Pete suggested that it would be good to present something at the IRIS Tuesday meeting, probably toward end of August/September which would fit well with RAL work on multi-VO support being completed on this time scale. - It was agreed we are probably well represented in Thursday RUCIO meetings and on RUCIO slack. - Has been suggested it would be good to have people attend the RUCIO coding camp which appears to be at CERN in October https://indico.cern.ch/event/819753/overview Hopefully Wenlong will be able to attend so it would be good to make introductions then. There was some further discussion about Edinburgh's commitment to help RAL with setting up RUCIO monitoring. As this has been requested by SKA Edinburgh has agreed to now make this a higher priority to assist RAL in setting up monitoring.