\documentclass[a4paper,
               %boxit,        % check whether paper is inside correct margins
               %titlepage,    % separate title page
               %refpage       % separate references
               %biblatex,     % biblatex is used
               keeplastbox,   % flushend option: not to un-indent last line in References
               %nospread,     % flushend option: do not fill with whitespace to balance columns
               %hyphens,      % allow \url to hyphenate at "-" (hyphens)
               %xetex,        % use XeLaTeX to process the file
               %luatex,       % use LuaLaTeX to process the file
               ]{jacow}
%
% ONLY FOR \footnote in table/tabular
%
\usepackage{pdfpages,multirow,ragged2e} %
\usepackage{hyperref}
%\usepackage{draftwatermark}
%\SetWatermarkText{DRAFT}
%\SetWatermarkScale{1}
%
% CHANGE SEQUENCE OF GRAPHICS EXTENSION TO BE EMBEDDED
% ----------------------------------------------------
% test for XeTeX where the sequence is by default eps-> pdf, jpg, png, pdf, ...
%    and the JACoW template provides JACpic2v3.eps and JACpic2v3.jpg which
%    might generates errors, therefore PNG and JPG first
%
\makeatletter%
	\ifboolexpr{bool{xetex}}
	 {\renewcommand{\Gin@extensions}{.pdf,%
	                    .png,.jpg,.bmp,.pict,.tif,.psd,.mac,.sga,.tga,.gif,%
	                    .eps,.ps,%
	                    }}{}
\makeatother

% CHECK FOR XeTeX/LuaTeX BEFORE DEFINING AN INPUT ENCODING
% --------------------------------------------------------
%   utf8  is default for XeTeX/LuaTeX
%   utf8  in LaTeX only realises a small portion of codes
%
\ifboolexpr{bool{xetex} or bool{luatex}} % test for XeTeX/LuaTeX
 {}                                      % input encoding is utf8 by default
 {\usepackage[utf8]{inputenc}}           % switch to utf8

\usepackage[USenglish]{babel}


%
% if BibLaTeX is used
%
\ifboolexpr{bool{jacowbiblatex}}%
 {%
  \addbibresource{jacow-test.bib}
  \addbibresource{biblatex-examples.bib}
 }{}
\listfiles

%%
%%   Lengths for the spaces in the title
%%   \setlength\titleblockstartskip{..}  %before title, default 3pt
%%   \setlength\titleblockmiddleskip{..} %between title + author, default 1em
%%   \setlength\titleblockendskip{..}    %afterauthor, default 1em

\begin{document}

\title{General Technical Services: Overview, improvements, remaining issues
}

\author{L.~Serio\thanks{luigi.serio@cern.ch},  \\
\& TIOC, (CERN, Geneva)}
	
\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
The technical services have contributed to the performances of the LHC machine.  TIOC has been instrumental in analyzing events, propose consolidations and coordinating interventions to minimize downtime and improve performances. The presentation will provide an overview of the most significant TIOC actions and summarize technical services overall availability and reliability. For the most significant major events the root causes, impact on operation and required repairs and consolidations will be outlined. An outlook for run 3 will be provided based on lessons learned and status of the consolidation program. 
\end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction
}
CERN’s Technical Infrastructure is a large and complex system of systems providing essential services for the reliable operation of accelerators and experiments (\cite{TI}). 
The Technical Infrastructure is made of the following critical systems: access infrastructure and system, cooling and ventilation, electrical network, IT services.
Any failure of the TI impacts directly on accelerators performance, availability and reliability.
Diagnosis of failures, analysis of major events (incident or near misses on a technical system with at least an accelerator, an experiment or the computer center stopped or potentially be stopped) and prevention of risks due to maintenance and operation intervention is complicated by the complexity, the intricated systems functional dependencies and the continuously evolving (physical and functional modification and consolidation) modifications of system inter-dependencies.
Furthermore the aging and budget constrained maintenance and consolidation program requires adequate monitoring, analysis and follow-up to assess systems and equipment and, anticipate and coordinate major events to minimize and mitigate impact.
All major events in the TI are followed up and analysed by the Technical Infrastructure Operation Committee. TIOC performs a systematic root cause analysis to identify causal relationships involved in the failures with a formal failure reporting, analysis and corrective action system.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Systems performances during run 2}	

Over the past years TIOC has monitored, recorded, analysed events related to the infrastructure systems serving the accelerator complex, the experiments and the computer center. TIOC recommendations have initiated and implemented several consolidation actions to correct situations originating from the reduced maintenance, as well as detected non-conformities or weaknesses of systems and equipment of the technical infrastructure. Furthermore, TIOC has coordinated (Fig.~\ref{FIG:TIOC}) several large and complex technical interventions and incidents or intervened via a special best effort on-call for immediate action and coordination. 

\begin{figure}[htbt]
\begin{center}
\includegraphics[width=0.6\textwidth]{Figures Evian Serio/TIOC.png}
\caption{TIOC interfaces and coordination.}
\label{FIG:TIOC}
\end{center}
\end{figure}
%%%%%%%%%%%%%
\subsubsection{Overall Performances Assessment}

The LHC accelerator downtime due to the TI equipment faults has been steadily reduced during Run 2.
TIOC has been instrumental in the availability improvement achieved by guiding the analysis of the data acquired during occurred major events and launching the required consolidation programs to eliminate the failures or, if not possible due to schedule or missing budget, implementing operational procedures to minimise the impact of failures or risks for failures during planned interventions on the TI.

The overall results of TIOC work, coordination, analysis and consolidation program is described in Fig.~\ref{FIG:TIPerformance} with an overall reduction in number of faults and in particular the reduction in downtime that reached in 2018 less than 100 h. Availability of the TI reached well above 99 \% with an average over the whole Run 2 of 98.6 \%.
 
\begin{figure}[t]
\begin{center}
\includegraphics[width=0.5\textwidth]{Figures Evian Serio/Overall_run2_availability.png}
\caption{Run 2 overall number of faults, downtime and availability.}
\label{FIG:TIPerformance}
\end{center}
\end{figure}

Fig.~\ref{FIG:TIPerformance2} shows the details of the major contributing TI systems to the overall downtime during Run 2. It must be noted that without the large impact in 2016 of the electrical system large downtime due to a short circuit provoked by a weasel, the typical pattern of increase in availability after a Long Shutdown after an early 'infant mortality' is clearly depicted. It shows the increase toward a plateau of maximum availability or minimum constant (random) failure rate as expected. This is probably reaching the best achievable performances. Further improvement could in principle only be reached by increasing redundancies, updating and consolidating existing systems and components, improve the analysis capability and reduce intervention time (e.g. automatic fault detection, AI to help detect root causes faster, clean up alarms for non critical components, increase cross system SCADA development, add  a completely new alarm console, auto masking alarms from logbook to make the alarm screen cleaner, among other things). 

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.5\textwidth]{Figures Evian Serio/Run2_individual_systems.png}
\caption{Run 2 individual systems contribution to the downtime.}
\label{FIG:TIPerformance2}
\end{center}
\end{figure}


Fig.~\ref{FIG:TIPerformance3} shows the pareto of total downtime, root cause downtime and individual system inherent downtime for all the TI systems. The Electrical System that will be further analysed in the following chapter is clearly the major contributor to the impact of the TI to the machines performances.

\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1.00\textwidth]{Figures Evian Serio/Pareto.jpg}
\caption{Pareto of all Technical Infrastructure Systems plotting the total, root causes and inherent systems downtime.}
\label{FIG:TIPerformance3}
\end{center}
\end{figure*}

%%%%%%%%%%%%%
\subsubsection{Electrical Network}

The electrical network availability during Run 2 is shown in Fig.~\ref{FIG:Electrical}. Not taking into account the weasel incident it corresponds to expected performances over the past runs and a steady state constant failure rate (random failure) that can only be improved by updating or consolidating existing system. Nevertheless over the coming years an increase in failures can be expected due to aging of components and sub-systems and limited or delayed maintenance and consolidation program.
 Fig.~\ref{FIG:Electrical1} shows the pareto of the total downtime highlighting, as expected, the main 400 kV supply and the overall distribution system as the most frequent faults.The 18 kV and 3.3 kV impacting respectively the Power Converters  and cryogenic system of the LHC have been failing in line with previous years performances. The decreased impact on machine unavailability is mainly due to improved procedures, processes and faster reaction time. The longer fault duration for the 3.3 kV supply is linked to the required recovery time for cryogenics after compressors stop.
The most significant faults encountered during run2 were 
\begin{itemize}
	\item the fault of the 66/18 kV transformer fault in Pt8 due to a weasel short circuit impacting for more than 6 days the operation of the LHC machine,
	\item the multiple failures on HTA transformers in 2017 accounting for a total of 3 days of unavailability,
	\item the power outage due to a software intervention on the control electrical supervision system PSEN in 2017.
\end{itemize}
2018 was a particularly bad year in terms of thunderstorm with more than 90 days or about 98 \% of summer days experiencing thunderstorm activities with power cuts, glitches and floodings over the whole CERN site. Nevertheless the number of electrical glitches slightly reduced over the years and their impact significantly decreased due to the improvement of  the power converter electronics sensitivity to the electrical network supply stability. 


\begin{figure}[b]
\begin{center}
\includegraphics[width=0.5\textwidth]{Figures Evian Serio/Electrical.png}
\caption{Run 2 electrical sub-systems contribution to the downtime.}
\label{FIG:Electrical}
\end{center}
\end{figure}

\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1.00\textwidth]{Figures Evian Serio/Electrical1.jpg}
\caption{Electrical sub-systems downtime contribution and frequency of failures.}
\label{FIG:Electrical1}
\end{center}
\end{figure*}
%%%%%%%%%%%%%
\subsubsection{Cooling and Ventilation}

The cooling and ventilation availability during Run 2 is shown in Fig.~\ref{FIG:Cooling}. It corresponds to expected performances over the past runs and a steady state constant failure rate (random failure) that can only be improved by updating or consolidating existing systems and components. Nevertheless over the coming years an increase in failures can be expected due to aging of components and sub-systems and limited or delayed maintenance and consolidation program.
 Fig.~\ref{FIG:Cooling1} shows the pareto of the total downtime highlighting the cooling tower system as the most impacting sub-system in terms of machines downtime. This is mainly due to the direct impact and stop of the cooling water for the cryogenics and power converter systems and subsequent beam loss and extensive recovery time.
The most significant failure was the flooding in point 3 during 2016, due to a malfunctioning relief valve and reject in the UJ and lift pit. It provoked a machine stop of 3 days and damaged several systems and cabling. The consequences of the risk of inondation due to the LEP water cooling network configuration in the LHC were afterwards mitigated for potentially future events by redirecting the discharge flow of the safety valve and increase the height of equipment potentially at risk (cabling and electrical cabinet).
Due to the aging of the water networks several flooding and stops of water circuits were experienced also in other installations. In addition several electrical cabinets for the raw water circuits (supply for cooling towers and fire hydrants networks) are now operating beyond their expected operational life. There is therefore a high risks of breakdown and extended downtime for the affected installations.

\begin{figure}[b]
\begin{center}
\includegraphics[width=0.5\textwidth]{Figures Evian Serio/Cooling.png}
\caption{Run 2 cooling and ventilatilation sub-systems contribution to the downtime.}
\label{FIG:Cooling}
\end{center}
\end{figure}

\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1.00\textwidth]{Figures Evian Serio/Cooling1.jpg}
\caption{Cooling and Ventilation Sub-Systems downtime contribution and frequency of failures.}
\label{FIG:Cooling1}
\end{center}
\end{figure*}


%%%%%%%%%%%%%
\subsubsection{Lift System}

The lifts replacement project launched in 2014 had a reliability goal of 15'000 trips between failures. The total downtime per year is shown in Fig.~\ref{FIG:Lifts} with a significant improvement after 2014. The planned reliability goal hasn't been reached mainly because of misuse, teething problems and the required initial training of technicians for the new lifts.The increased of breakdown in 2018 shall be further analysed as it couldn't be related to an aging problem.

\begin{figure}[hb]
\begin{center}
\includegraphics[width=0.5\textwidth]{Figures Evian Serio/Lifts.jpg}
\caption{Lifts total downtime and origin of failure.}
\label{FIG:Lifts}
\end{center}
\end{figure}


%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed consolidations and upgrades
}

The aging of the equipment and the limited resources available will increase the risk to jeopardize the reliability of the installations for the coming years.
An analysis of available resources and most critical items to consolidate and/or upgrade has been performed and will be implemented during LS2.
The EN-EL group will perform during the LS2 the following major consolidation in order to minimize the impact of aging equipment, some of which is already beyond its expected lifetime (e.g. transformers): 
\begin{itemize}
	\item the Prevessin's 400 kV substation BE,
	\item the SPS high voltage network,
	\item  the UPS and 48 V DC systems,
	\item  the main general services substation ME9,
	\item  the auto-transfer system, the old diesel generator in the NA BB81,
	\item   the West Area substations,
	\item  the LHC18 electrical network,
	\item the LHC PLC diesel and replacement of the UPS batteries in the LHC.
\end{itemize}

In addition, EN-EL will:
\begin{itemize}
	\item upgrade of the 400 kV protection system BE1,
	\item  install a new 400/66 kV substation in Bois-Tollot BE2,
	\item   install several new SPS high voltage electrical substations
	\item  upgrade the main general services substation ME9,
	\item  install the new diesel generators and secured network substation ME91.
\end{itemize}

The EN-CV group will perform the consolidation of
\begin{itemize}
	\item  the PS complex (PSB cooling plant and chilled water production,
	\item   Lina3 HVAC,
	\item   AD target cooling,
	\item  PS Central building and main magnet cooling,
	\item  Siemens S5 PLC obsolescence),
	\item  the North Area (power and controls of cooling towers and chilled water network),
	\item  the SPS (BB3 primary water system, HVAC and piping for fire safety and the HVAC of surface buildings),
	\item the LHC controls renewal on the HVAC of CMS and ATLAS. 
	\end{itemize}
The aging water network is a huge and expensive problem to fix. Investigations on the most critical issues will be performed for some consolidation during LS2. An adequate financing shall be made available in order to consider the consolidation of the whole water network at CERN to eliminate risks of many more serious incident as the one experienced during Run2.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Outlook post LS2
}

Limitations in resources (budget and manpower), time and the complexity of some equipment and intervention, have required to postpone or cancel some of the consolidations or upgrade that were planned in order to maintain the level of availability of the technical infrastructure to the standards reached in Run 2.
The EN-EL group has therefore proposed not to perform for the time being the following consolidations: 
\begin{itemize}
	\item  the SPS low voltage electrical network,
	\item  the North Area electrical network,
	\item  the main substation supplying the medical services, HUG and the Fire Brigade (ME13),
	\item  the substations supplying PSB/ISOLDE (ME49 and ME76),
	\item  the oil transformers not compliant with current regulations and the LHC 48 V DC systems, 
	\item  the 18 kV protection relays and the HTA transformers (GEAFOL that failed during run 2).
	\end{itemize}

The impact will be a probable stop of the concerned equipment during run 3 with the required additional corrective maintenance intervention, increased operation and maintenance cost and related downtime.
The EN-CV group has proposed to postpone the water network less critical aging circuits with the impact on users that will be affected by the elevated risks of failure and breaks. 
The delayed intervention on the raising pumps and raw water network obsolescence's will further increase the risk of flooding and material damage in underground areas. The inspection and maintenance on power electrical switchboards is also postponed. 
Overall, the proposed down scoping and delay in maintenance and consolidations will fit the available resources envelope but will most probably impact availability and the longer term costs of operation due to increased corrective maintenance and interventions.
A comparison based on data extracted from INFOR EAM and industrial maintenance and reliability best practices (\cite{Maintenancebestpractices}) clearly show that CERN's technical services average Mainteance Costs over replacement asset value ratio is about 1 \%. This is significantly lower than the targets in industry varying between 3 and 5 \%.
The outlook and perspective for Run 3 are therefore a significant risk to reduce reliability and therefore availability due to aging and obsolences not mitigated by an adequate maintenance and consolidation program, There is also a significant risk to experience more single major events leading to long downtime due to major breakdown (e.g. transformers failures and water piping breaks). At least at the beginning of Run3, there will be a lower reliability, as in the past, of several systems for which major modifications have been performed, before reaching again nominal reliability values.
A new tool  (\cite{ML}) based on mining with Machine Learning algorithms previous years data (maintenance work orders, costs and planned budget, systems overall downtime and availability performances) indicates a decrease in machines availability due to technical infrastructure decreased reliability figures inferred from the planned maintenance budget and past reliability figures. This is in addition to further decrease due to major incidents, breakdowns  and aging beyond expected life time related issues. The overall assessment estimates shows a 1 to 2 \% decrease in the machine inherent availability due to the technical infrastructure. The figures represent an estimate that doesn't take into account recovery time for the other systems (e.g. cryogenics), major breakdowns and failures due to aging and operational error and environmental conditions. It is therefore to be taken as a minimum expected reduction of the reliability performance of the TI.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusions
}

The excellent performance reached by the technical infrastructure system during Run 2 were mainly due to an adequate maintenance program implemented during LS1, the systematic root cause analysis and the formal follow-up  of corrective actions  and  implementation of tools and procedures to minimize or mitigate failures as well as coordinate major interventions, planned and not-planned events. TIOC was instrumental on all the above allowing a formalised and common approach for all technical infrastructure systems.
Despite the good performance there is still room for improvement and in particular the need to continue the development of the necessary analysis tool for the complex and intricated systems dependencies. Some work is ongoing to use Machine Learning algorithms to model, assess and predict nominal and non nominal operation of the infrastructure as a whole  (\cite{ML}).
It is also recommended to review against industry maintenance and reliability best practices the level of financing of consolidation and maintenance activities for the technical infrastructure while guiding and monitoring the results via appropriate performance indicators that would be extremely useful on the long term process of optimising resources and activities.
For Run 3 we expect a decrease in availability due to reliability issues. The aging and obsolescence risks have significantly increased and there is a potential risk of single and high downtime events due to a major failure affecting one of the main non consolidated and not redundant sub-systems. Budget restrictions during LS2 and in the future years are not allowing the full deployment of preventive maintenance plans that are required to be able to reach the nominal level of performances and availability experienced during Run 2.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Acknowledgements
}

All the work reported is the result of an efficient collaboration among all groups and departments member of the TIOC and in particular the valuable support of the BE-OP-TI section.

\ifboolexpr{bool{jacowbiblatex}}%
	{\printbibliography}%
	{%
	% "biblatex" is not used, go the "manual" way
	
	\begin{thebibliography}{99}   % Use for  10-99  references
	%\begin{thebibliography}{9} % Use for 1-9 references

	\bibitem{TI}J. Nielsen and L. Serio, "Technical Services: Unavailability Root Causes, Strategy and Limitations", in Proc. 7th Evian Workshop on LHC beam operation, Evian Les Bains, France, December 2016.
	
	\bibitem{Maintenancebestpractices} Maintenance \& Reliability Best Practices, R. Gulati: 3-9\%  Physical Asset Management Handbook, J. Mitchell: ~3\%, Asset Life Cycle Engineering, S. McNair: 2-4\%. 

\bibitem{ML} L. Serio et al., Machine Learning-Based System for the Availability and Reliability Assessment and Management of Critical Infrastructures, AISIS2019. 
	 
	\end{thebibliography}

} % end \ifboolexpr
%
% for use as JACoW template the inclusion of the ANNEX parts have been commented out
% to generate the complete documentation please remove the "%" of the next two commands
% 
%\newpage

%\include{annexes-A4}

\end{document}



%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[width=0.49\textwidth]{figures/PSB_brightness_standard_2.png}
%\caption{default}
%\label{default}
%\end{center}
%\end{figure}
