JCOP Fw Installation Tool discussion

Tuesday 5 Sept 2017, 14:00 → 16:00 Europe/Zurich

42/3-002 (CERN)

Discussion about requirements for the fwInstallation tool that need to be completed with priority, prior to the redesign of the Framework

fwInstallation Rework 2017.pdf

fwInstallation_update_20170904.pdf

Reduced JCOP FWWG on the Installation Tool

Present: Andre Augustinus, Peter Chochula, Slava Filimonov, Clara Gaspar, Frank Glege, Piotr Golonka, Lukasz Goralczyk, Mariusz Suder, Fernando Varela

• Fernando and Piotr explained that the efforts will be divided into 2 phases:
  • before the nearest YETS, the most important features will be developed and delivered, after which the current fwInstallationTool and fwConfigurationDBSystemInformation will enter maintenance mode
  • the second phase will be started in the scope of consolidation/facelift/redesign of frameworks; it is at this stage that the remaining requests and suggestions will be treated
• Rafal Kulaga will take over the responsability for the fwInstallation tool from Lukasz.   Mariusz Suder will be the main developer.
• Lukasz presented the current state of fwInstallation tool (see slides);
  • some bugfixes already included, namely for the memory leak related to the use of some database resources (bug reported by ALICE and LHCb); to be tested/confirmed, in particular with the new patched version of the Oracle Instant Client
  • Request of CMS to store the "attached values" for the project (e.g. DB connectivity parameters, etc): initial work started with estimated development time of 2 weeks; Frank requests the release of the tool with this feature as soon as it is available
  • agreed to have fwInstallation-8.0.6 with the "attached value" feature available in ~2 weeks (end of September). In any case this is too late to have any release suitable to be deployed in the nearest Technical Stop.
• Lukasz presented a document (see slides) listing the available features of the tool and pending fixes/improvements and pending features. Lukasz proposes the list of features that, from his point of view, would be desirable in future versions; this triggered discussion during which the following points were raised (listed in random order)
  • rollback feature is probably not achievable due to, for instance, dependencies between components and many others; major conceptual work would be needed; in practice, one achieves "rollbacks" at the moment by making a backup of projects
  • "transparent installation" : the tool should be intelligent not to be too intrusive, and for instance detect that only files need to be copied. Lukasz commented that component developers should focus on the component main functionality, rather than the its packaging; the installation steps should be done for him by fwInstallation tool functionality, with all smartness required
  • Clara suggests that the components should be done such that installations should be easy. It  should  be clearly documented for every version what has changed, and what is happening during the upgrade of the component
  • A tool that makes a diff (allowing for a review whether the change was safe to apply) : it is used for PSEN project and a simiar approach is also at LHCb (windiff of complete framework);
  • for LHCb, component upgrades are done by Clara without the use of fwInstallation, hence all the postInstall scripts need to be replayed manually... The steps exec'd by postInstall should be avoided, if possible, or well documented and easy to apply
  • General problem of the use of DPL files to store various settings. At the component upgrades the settings are overwritten. A better mechanism would be welcome, plus review of this in all components.
  •  "restricted" operations that prevents certain operations, e.g. on modifications of certain internal DPL via ASCII manager, etc.
  • DB redesign: an abstraction layer to avoid direct writes to database and induced inconsistencies. DB is already very convoluted due to its history of evolution; technologies: PLSQL or views to be considered
  • CMS suggests the use of "tags" to determine which configuration is for production, which for tests, etc
  • agreed that the "massive administration" (e.g overview and manipulation of all dps in all systems, etc) is not in the scope of the fwInstallation.
  •  the CDBSI needs ergonomics facelift, e.g. when one needs to declare a new version of Framework to be massively upgraded on many systems, or a change in the component group should be applied only to the explicitely targetted systems (some mechanisms are in place, and CMS has scripts to do that already; would be nice that the tool provides them in its panels)
  • ALICE: When checking files, hashes should be calculated at installation time for the files being installed, and they should be cached instead of being recalculated on the "source" avery time; should be possible to recalculate them; there is this intermediate hashing to which comparison should be done, and it should respect the order of project paths (to avoid unnecessary alarms if a "overparameterized" file is actually not effective; System Overview does not use proper checksums and shows errors even though installation tool is OK;
  • ALICE (still about hashes): it should be possible to easily update the hash of a single file in DB, in scenarios where someone opened the file in editor and saved it without really introducing any chance; this would calm down the warning;
  • ALICE suggests that the fwInstallationAgent should have parameterizable setting on how often it is started

  • ALICE: requesting that the overparameterized files should show details: this is already available by clicking in details column. General comment that "overparameterization" word is strange, yet we follow the nomenclature of WinCC OA here...

  • ALICE: installation tool does not recognize UNC paths (//server/folder/); workaround is to map a drive (this is what CMS is doing); not sure if WinCC OA supports UNC at all; to be clarified

  • General comment about error handling and logging: very inconsistent and requires careful study of log files

    • ALICE suggests that major faults during installation (such as "violated FK in database") are not only dropped to the log but in addition stored in the DB, whereby System Overview Tool could pick it up and present

    • CMS chimes-in requesting that all the log output are much better structured explaining what happens and for what components; all the log output should then be made somehow permanent (eg. in a datapoint) and displayed in a panel, showing clearly the progress of installation of the component. Would be nice that component installation would gather some profiling info (how long things take) and logs it.

    • Fernando clarifies that fwInstallation_log() function already exists for this purpose, yet it is not used consistently by all components; this function already stores the texts in DB, and displays them in the panel

  • When installing a new version of component, the files belonging to old versions of components should be cleaned up; apparently a non-trivial problem in particular when installing DLLs that are copied by postInstall scripts, for instance. At the moment one may uninstall and install again, and this clears up the files but only those that component knows about (ie. not the DLL copies that postinstall does). We may need more effort to provide better fwInstallation functions for component developers to eliminate the problem at the source. Certain components (such as fwAccessControl) used to have special code for cleanup in their postInstall scripts.

  • Proposal of distributing the "core part" of fwInstallation as a WinCC OA Patch; approved as a good idea, yet technical details need to be considered (project pathes already have fwInstallation; how to "undo" it, etc

    • The idea is to split the tool into a core "bootstrap" part that could download/upgrade the main part of the tool easily

    • LHCb would need to handle installation without having the agent. The bootstrap of the fwInstallation (creation of internal DPs, etc) would need to be run somehow.

  • Commandline mode for fwInstallation: would be welcome for LHCb

    • Piotr inquired about the interest in having a common commandline tool to create projects, etc, quoting the WPM tool of BE-ICS already presented in FWWG. Seems no interest at the moment as everybody already has and maintains their own tools.

    • For "commandline" installation of components, one should look at what could become the repository of components. For BE-ICS it is Nexus, CMS alrady use SVN tags, others: to be looked and discussed

• Lukasz demonstrated the APEX web tool that is a frontent to BE-ICS System Overview (CDBSI/InstallationDB); ALICE has a very similar tool.

  • APEX likely to disappear within next 5 years due to Oracle Weblogic being phased-out from CERN... BE-ICS is moving towards web applications on top of AngularJS, hosted by CERN services.

• Agreed that ALICE requirements concerning the hashing would be implemented in the next-to-next version of the fwInstallation, before YETS.