SWrep cleanup and CDB commit speed up

Thursday 27 Mar 2008, 16:00 → 17:00 Europe/Zurich

513-1-021 (CERN)

Veronique Lefebure (CERN)

We have memory problems on CDB compilation, and suspect it is due to the enormous size of the repository templates. Cleaning them up may help to get rid of the memory issue and to fasten compilation.

Slides Matthias_slides.pdf Matthias_slides.ppt

Present:
Matthias Schroder, German Cancio, Ulrich Schwickerath, Ricardo Silva, Jan van Eldik, Veronique Lefebure

• See attached slides written and presented by Matthias
• Comment from German: technically it would be possible that swrep checks itself if an RPM present on a repository is used or not by a machine, but it would be very expensive. Also cleaning up the rep. automatically would not allow roll backs. Finally, if one RPM is removed from the repository, nothing would prevent someone to put it back with the same name but with a different content, which would lead to a mess.
• Question by Ricardo: what about having a repository for the noarch packages ? German's answer: there used to be one but it was removed to ease maintenance. Also, lots of "noarch" packages are not really arch independent.
• ACTIONS:
  • attack first the most frequent packages
    
    • candidates: castor, CA's, kernels,*debug
      • *debug should stop being mirrored at all. Do we still have a lot of them ?
  • create a list of them by parsing the .xml profiles
    • we could also use CDBSQL: on a separate database, copy ONLY pkg info (to the opposite of what we currently do). That would be very easy to query. Easy to do: we have all tools available.
      • ! L4C don't have a CDBSQL backend and xml are not accessible to us, but they use OUR swrep !
  • decide how old packages we keep (we need to let people the possibility to roll backs
    • Question from Jan: do we know how many packages are really used ? and how much we will gain by cutting the swrep down to a number of x packages ? This should be measured.
  • kernels themselves should be cleaned up. But this is out of the scope of this discussion. Topic for another meeting
• Summary:
  • by end of next week (04.04.08):
    
    • get a cleanup repository template and test it (for each platform) (Matthias and Veronique)
    • mv packages away from the swrep if test above ok (Veronique)
  • Implement an archive option for swrep-soap-client (Marco Emilio (ask him) or Veronique) and present the idea before to project-quattor
  • every xx months, clean up swrep
  • setup a CDBSQL for packages only (Veronique)
• Topic for next meeting: Ulrich pushes for the washdog