Go to the Indico Home Page Up to category Newer event Newest event Printable version
 
Focus on:  Hide Contributions
  • Compact style
  • Indico style
  • Indico style - inline minutes
  • Indico style - numbered
  • Indico style - numbered + minutes
  • Indico Weeks View
    • Switch to the management area of this event
    Login

    ATS-IT Consolidation of Server Provisioning

    → Europe/Zurich
    31/2-028 (CERN)

    31/2-028

    CERN

    8
    Show room on map
    Enzo Genuardi (CERN), Jose Castro Leon (CERN)
    ATS-IT Consolidation of Server Provisioning
    Zoom Meeting ID
    66883354831
    Host
    Jose Castro Leon
    Useful links
    Join via phone
    Zoom URL
    Hide

    ATS-IT Consolidation of Server Provisioning Meeting minutes - 04-08-2025

     

    Attendance:

     

    • Enzo GENUARDI (ATS)

    • Giuseppe SIMONETTI (ATS)

    • Nikolaos TSIPINAKIS (ATS)

    • Thomas OULEVEY (ATS)

    • Karolina PRZERWA (IT) (online)

    • Vincent DUCRET (IT)

    • Nikos PAPAKYPRIANOU (IT)

    • Jose CASTRO LEON (IT)

     

    Introduction to the Project

     

    • Jose Castro Leon: We are just starting with a kickoff meeting describing the project at a high level. Describing processes that happen in IT and then in ITS so then later we can dig into each of the individual steps concerned. We can do later do an evaluation of the tools and set up a testbed.

    • Jose Castro Leon: We can just show the PSO draft, we aim to present it on the Technical committee meeting on Aug 26, 2025 There are no major changes, we just need to fill some blanks left.

    • Thomas: The document was split into several phases with Steven?

      • Jose Castro Leon: This is the phase1 just for the provisioning of baremetal resources, its evaluation and set a testbed

      • Jose Castro Leon: Then in the upcoming phases, we can target the resources provisioned for kubernetes.

    Description of IT server provisioning processes

    • Jose Castro Leon: Brief of the process to provision resources and which is the tool that we use

    • Jose Castro Leon: Describe the different steps of the process that have not changed much since.

    • Nikos: These slides come from 2021

    • Jose Castro Leon: The automated process starts with the Acceptance tests

    • Thomas: Do you always take 8 weeks to provision hardware?

      • Nikos Papakyprianou: No, it depends on the hardware that we receive.

      • Thomas: Enzo, do we run those burning tests for a week?

      • Enzo: We run this for 4-5 days depends

      • Nikos Papakyprianou: This is up to 8 weeks to trigger the payment. (2 weeks of installation and 6 weeks of acceptance)

      • Jose Castro Leon: This only reflects the worst case scenario

    • Jose Castro Leon: Then we receive those and we commission them and make them available for end-users

    • Jose Castro Leon: Once they are in production, they can be targeted for firmware upgrades, replace parts, etc through servicing.

    • Jose Castro Leon: And a year before they get out of warranty, we notify end-users that the machines will be decommissioned

    • Jose Castro Leon: Now we are going to look at the tool that we are using

    • Jose Castro Leon: We as admins we maintained a pool of resources and users can provision/deprovision from the pool

    • Jose Castro Leon: We have the same image that applies both for VMs/Physical Instances (w GPT support, UEFI, software RAID)

    • Jose Castro Leon: We see now the full picture of the state machine of Ironic and how the server lifecycle is managed

    • Jose Castro Leon: Since quite some time we have changed to Redfish as our default driver

      • Thomas: Do you still have the issue with IPMI that you need a random user and a random password? Can you use just redfish?

      • Nikos Papakyprianou: We can disable IPMI and just use redfish

      • We don’t do it yet, as we have still legacy nodes, and we are just run both, and randomize the credentials

      • Nikolaos: Do you store the credentials in teigi or where?

      • Nikos Papakyprianou: We have an API/DB in which we stored them, and we can use it in case of a major event

      • Thomas: We can try to favor Redfish only for this project

      • Jose: The tool that we have uses both, and it’s nice to have something in case we need to onboard hardware

      • Nikos Papakyprianou: We still have some tools that need IPMI

      • Thomas: From IPMI you can’t access the device without root access, but this may have changed for redfish?

      • Nikos Papakyprianou: Exposes a web interface locally and you can still do the same

      • Jose Castro Leon: Focussing on newer tools it’s nice but we also need to keep the window open to add older hardware in the future

      • Enzo: Is there any difference between IPMI and Redfish

      • Nikos Papakyprianou: Redfish do not support everything we need, but most of it. But it’s nicer in the output (json based) than ipmitool

      • Jose Castro Leon: Direct console access

      • Thomas: For icinga, we may need to change how we are monitoring it?

      • Jose Castro Leon: Yes, but you can use IPMI for that at the beginning and change that part later into the project.

      • Nikos Papakyprianou: We have a tool that has the randomizing of the IPMI

      • Nikolaos: If you can share with us this randomizing tool it would be really helpful

    • Jose Castro Leon: Now looking into the autoregistration flow

      • Thomas: Do we have a way to replace a server in case of failure?

      • Nikolaos: In some situations, we replace a faulty server and we clone the MAC address on the spare machine

      • Jose Castro Leon: Because you need to keep the same IP

      • TODO Jose Castro Leon:  Evaluate the use case for replacing servers in-place

      • Vincent: The server will boot and it’s not known so it gets an IP on both interfaces that will update the details in lanDB

      • Thomas: Do you know if we can add this setup on the TN?

      • Vincent: It’s doable

      • Vincent: We can also check if we can split the MGMT from the DATA

      • Nikolaos: Both are connected to the same switch

      • Vincent: We’ll try to split them into different routers, VLAN, but it could be doable

      • Nikos Papakyprianou:: What are you using for provisioning?

      • Nikolaos: We use kickstart based installation to our own PXE instance

      • Enzo: For machines in IT, we use AIMS to forward it to our PXE instance

    • Jose Castro Leon: Now looking at introspection and how the information is propagated from the nodes

      • Thomas: Which is the source of truth for location information?

      • Nikos Papakyprianou:: This is now handled by EAM and then distributed to OpenDCIM

      • Giuseppe: We have used with an specific field to show it on the back or on the front

      • Thomas: We use the openDCIM plugin to do affinity/anti- on the applications

      • Jose Castro Leon: The solution implemented was the only option at that time

      • Jose Castro Leon: We have different ways now to allow you to be really specific on how the nodes are deployed.

      • Nikolaos: How can we target a specific node?

      • Jose Castro Leon: We have the possibility to target individual nodes if you need to

    • Jose Castro Leon: Introspection rules can be used to raise alarms if some of the nodes do not meet the requirements

      • Thomas: How do you handle SAS arrays with data on it?

      • Jose Castro Leon: We have a feature that’s called safeguards that avoid to touch disks that you don’t want to.

      • Nikolaos: Is LanDB updated with the information on provisioning?

      • Jose Castro Leon: Yes

      • Enzo: If we have multiple registrations? Where does the serial number come from? Is it autogenerated?

      • Nikos Papakyprianou: We request to the vendor to add ASSERT_TAG and the PRODUCT_TAG inside the bios to generate the serial number

    • Jose Castro Leon: Now about burning, we run in parallel on all nodes (cpu,memory,disk) and we pair nodes to validate the network setup.

    • Jose Castro Leon: We benchmark all nodes with HEPSPEC and HEPSCORE

    • Jose Castro Leon: And now for retirement, we tag the nodes to be retired so then on cleaning the nodes will not be brought back into the available state

      • Nikolaos: Can we move the cleaning step to be done at provisioning?

      • Jose Castro Leon: I don’t think this can be changed, what are you trying to avoid is to prevent accidental deletion?

      • Jose Castro Leon: There is not much we can do, and the recovery will need multiple layers of the stack

    • Jose Castro Leon: And now on the resources that have been managed by Ironic since its beginning.

     

    Q&A

    • Nikolaos: You mentioned that we have a solution for kickstart files?

    • Jose Castro Leon: We do have a solution within Ironic to use kickstart files to deploy nodes, developed within the community, but this is not actively used.

    • Thomas: We may be interested in bootc, for both VMs and physical machines.

    • Jose Castro Leon: This is in the master branch of Ironic and we are not far from there

    • Nikolaos: Do we need you to operate the equipment and also the control plane?

    • Jose Castro Leon: We just need to agree on the support levels.

    • Nikolaos: Disk on delete, to prevent accidental deletion of servers

    • Jose Castro Leon: Locking will help partially

    • Nikolaos: This is to introduce a step to prevent deletion of important data

    • Jose Castro Leon: Available is the state on really to be deployed

    • Jose Castro Leon: It’s not that easy to undelete with all the DBs involved

    • Jose Castro Leon: I understand the use-case but if you introduce a manual step to validate, it also breaks the automated process

    • Thomas: For dedicated machines with timing cards, can we have flavors for that?

    • Jose Castro Leon: Yes, with flavors

    • Thomas: Are we going to have a new region for the PoC?

    • Jose Castro Leon: Ideally it’s better to avoid any interference with existing TN1 region

     

    Next Steps

    • Jose Castro Leon: Please have a look on the PSO and add your input

    • Jose Castro Leon: Here there is a link to the shared material of the project

    • Jose Castro Leon: And the next meeting is in Prevessin in 2 weeks 

      • Enzo: Please remind that the entrance to the room is from the external door, the cafeteria will be closed.



    The meeting finished at 11:30am

    There are minutes attached to this event. Show them.