\documentclass[a4paper,
               %boxit,
               %titlepage,   % separate title page
               %refpage      % separate references
              ]{jacow}
%
% CHANGE SEQUENCE OF GRAPHICS EXTENSION TO BE EMBEDDED
% ----------------------------------------------------
% test for XeTeX where the sequence is by default eps-> pdf, jpg, png, pdf, ...
%    and the JACoW template provides JACpic2v3.eps and JACpic2v3.jpg which
%    might generates errors, therefore PNG and JPG first
%
\makeatletter%
	\ifboolexpr{bool{xetex}}
	 {\renewcommand{\Gin@extensions}{.pdf,%
	                    .png,.jpg,.bmp,.pict,.tif,.psd,.mac,.sga,.tga,.gif,%
	                    .eps,.ps,%
	                    }}{}
\makeatother

% CHECK FOR XeTeX/LuaTeX BEFORE DEFINING AN INPUT ENCODING
% --------------------------------------------------------
%   utf8  is default for XeTeX/LuaTeX 
%   utf8  in LaTeX only realises a small portion of codes
%
\ifboolexpr{bool{xetex} or bool{luatex}} % test for XeTeX/LuaTeX
 {}                                      % input encoding is utf8 by default
 {\usepackage[utf8]{inputenc}}           % switch to utf8

\usepackage[USenglish]{babel}			 

\usepackage[final]{pdfpages}
\usepackage{multirow}
\usepackage{ragged2e}

\usepackage{units}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{pgfplots.groupplots}   

%
% if BibLaTeX is used
%
\ifboolexpr{bool{jacowbiblatex}}%
 {%
  \addbibresource{jacow-test.bib}
  \addbibresource{biblatex-examples.bib}
 }{}
\listfiles

%
% command for typesetting a \section like word
%
\newcommand\SEC[1]{\textbf{\uppercase{#1}}}
\def \cwidth {82.5mm}

%%
%%   Lengths for the spaces in the title
%%   \setlength\titleblockstartskip{..}  %before title, default 3pt
%%   \setlength\titleblockmiddleskip{..} %between title + author, default 1em
%%   \setlength\titleblockendskip{..}    %afterauthor, default 1em

%\copyrightspace %default 1cm. arbitrary size with e.g. \copyrightspace[2cm]

% testing to fill the copyright space
%\usepackage{eso-pic}
%\AddToShipoutPictureFG*{\AtTextLowerLeft{\textcolor{red}{COPYRIGHTSPACE}}}

\begin{document}

\title{Machine Protection during Run 2016 - Review of MP strategy}

\author{D. Wollmann\thanks{daniel.wollmann@cern.ch} CERN, Geneva, Switzerland} 
	
\maketitle

%
\begin{abstract}
In this contribution the strategy for the initial intensity ramp-up, the ramp-ups after short stops and the machine protection validations (loss maps, asynchronous beam dump tests) will be reviewed and improvements proposed. Operating the LHC with important systems in a degraded mode will be reviewed and discussed on the basis of two examples. A new fast failure case, causing orbit kicks on the beam in case of a quench will be presented. Finally, the machine protection classification and approval strategy of machine developments (MDs) is reviewed.
\end{abstract}

\section{Introduction}
2016 has been an excellent year with the LHC surpassing the design value for instantaneous luminosity and delivering about 40 $fb^{-1}$ integrated luminosity to the two high luminosity experiments. Besides many others, the LHC machine protection systems and the teams responsible for them were an important part of this success. The well established machine protection strategies and procedures, which were enforced and documented for standard operation and machine developments, ensured the safe operation of the LHC in 2016. This contribution analyses and reviews critically the 2016 machine protection strategy and proposes further improvements.


\section{Intensity ramp-up strategy and machine protection validations}
As in previous years, a step wise increase of the stored beam intensity was performed in 2016 following the beam commissioning. For each intensity step a minimum of 3 fills and 20 hours of stable beams were required. The correct functioning and response of all machine protection relevant systems was carefully analysed for these fills - from beam injection to dump - and the results were documented in so-called check lists. The defined intensity steps were 3/12, 48/72, 288, 570, 860, 1200, 1700 and 2300 bunches. The increase of stored beam intensity was interleaved with the increase in the length of the injected bunch trains, to avoid increasing both simultaneously. The first intensity step (3/12 bunches) is in general focused on establishing the LHC machine cycle. The second part (48 - $\sim 1000$ bunches) is designed to verify the correct functioning of all machine protection systems and to identify and mitigate potential issues. Intensities above 1000 bunches are usually dominated by intensity related limitations, like e-cloud, RF heating or UFOs. During the latter the increase of the stored beam intensity is not performed in big steps, but rather by incrementally adding one bunch train more from fill to fill, until limitations appear. In this phase, check lists are performed on a 6-weekly basis and are then called \textit{intensity cruise} check lists. 

Excluding a several day long stop of the whole accelerator complex due to a problem with the powering of CERN's Proton Synchrotron (PS), intensities $> 1700$ bunches (equivalent to stored beam energies above 200~MJ) were reached only 15 days after the start of the intensity ramp-up (see Fig. \ref{fig:IntensityRampup2016}). In total, 7 intensity increase and 4 intensity cruise checklists were filled during the proton - proton run in 2016 and documented in EDMS~\cite{EDMS}. 

\begin{figure}[h]
   \centering
   \includegraphics[width=1.05\columnwidth]{../paper/plots/IntensityRampup2016}
   %\hspace*{0.01\columnwidth}
  \caption{Intensity ramp-up 2016 from the end of the beam commissioning until technical stop one (TS1). The blue and red lines indicate the stored beam intensity in the two LHC rings in charges. The dashed lines symbolise intensity check lists before stepping to the next intensity. The stored number of bunches per beam is shown in orange.}
   \label{fig:IntensityRampup2016}
\end{figure}
For restarts after stops of nominal operation longer than 48~h two scenarios were defined in 2016. Scenario one applies to stops with little hard- or software interventions and requires in total two ramp-up fills. In this case the LHC cycle has to be revalidated with pilots bunches, if additional optics measurements are required. Otherwise a fill with 2-3 nominal bunches should be performed. In addition, to disentangle wrong settings, de-conditioning etc. from the intensity dominated effects at full pre-stop intensity, a $\sim 600$ bunch fill with 2-5 hours stable beams is required. Afterwards, operation can be continued at pre-stop intensity. \\ The second scenario covers a stop with numerous hard- and software interventions, as in a usual technical stop, and requires three to four fills: one fill with either pilot or 2-3 nominal bunches for cycle validation (as above), one fill with $\sim 50$ bunches and 1-2 hours stable beams, one fill with $\sim 600$ bunch and 2-5 hours stable beams and, finally, in case intensities $> 2000$ bunches had been reached before the stop, a fill with about half the maximum achieved number of bunches with about 5 hours stable beams. Afterwards, operation can be continued at pre-stop intensity. The correct behaviour of all machine protection critical systems needs to be carefully monitored during these short intensitiy ramp-ups to allow a quick identification and mitigation of possible issues. 

In 2016, scenario one has been successfully applied to the ramp-up after the stop due to the PS powering problem, technical stop one, machine development block one (MD1), MD2 and MD4. Scenario two was used for the ramp-up after MD3 / technical stop two (TS2). Figure ({\ref{fig:RampUpAfterShortStops_2016}}) illustrates the use of the two scenarios in 2016. As the experience with the two scenarios in 2016 was very positive it is proposed to also apply them during the LHC run in 2017.

\begin{figure}[h]
   \centering
   \includegraphics[width=1.025\columnwidth]{../paper/plots/RampUpAfterShortStops_2016}
   %\hspace*{0.01\columnwidth}
  \caption{Ramp-up of the intensity after short stops from nominal operation without (scenario 1) and with (scenario 2) massive hardware interventions. The first was for example applied after technical stop one (TS1), the latter after technical stop two (TS2) followed by the $\beta^* = 2.5$ km run.}
   \label{fig:RampUpAfterShortStops_2016}
\end{figure}
Figure (\ref{fig:LossMaps_AsynchDumps_overview_2016}) shows a table with the number of loss maps and asynchronous beam dump tests performed for validation of cleaning and passive protection during the 2016 run. In total, the impressive amount of 204 betatron loss maps, 38 off-momentum loss maps and 40 asynchronous beam dump tests have been performed as part of the initial beam commissioning, re-validation campaigns after technical stops and the different validations during the proton - ion run ~\cite{Mirarchi+2017}. These machine protection tests have been followed-up systematically, regularily and timely and were essential to gain confidence in the safe operation of the LHC with $\beta^*=40$ cm, relying for the first time on the phase advance between tertiary collimators respectively triplets and the dump kickers in IP6. \\ It is important to point out that betatron loss maps have a significantly smaller operational footprint than off-momentum loss maps and asynchronous beam dump tests, as the beams are usually lost as part of the latter. The use of the so-called gentle off-momentum loss maps allowed to perform mutliple off-momentum loss maps with the same beam, which increased the operational efficiency of these tests. One of the reasons for the comparably big number of validation tests performed in 2016 was the subdivision of the LHC cycle. A simplification of the cycle would allow to reduce the number of loss maps and asynchronous beam dump tests significantly. In addition, so-called continuous loss maps were performed during the ramp and the squeeze beam process, which provided important data on the cleaning performance during these parts of the LHC cycle. Based on the 2016 experience the standard, minimal scenario should be reviewed and possibly optimised. Furthermore, performance studies should be consequently seperated from machine protection validations. For the future it should be studied, if the analysis of loss maps and asynchronous beam dump tests can be further automatized, and how regular physics fills and their dumps can be used to validate the correct settings of the protection devices. Finally, the DOROS BPM interlocks implemented for the tertiary and IP6 secondary collimators should be unmasked and an automatic analysis implemented.

\begin{figure}[h]
   \centering
   \includegraphics[width=\columnwidth]{../paper/plots/LossMaps_AsynchDumps_overview_2016}
   %\hspace*{0.01\columnwidth}
  \caption{Overview table of loss maps and asynchronous beam dump tests performed in 2016.}
   \label{fig:LossMaps_AsynchDumps_overview_2016}
\end{figure}


\section{Operation of systems in degraded mode}
During the 2016 run there were two prominent examples, where a system was operating in a degraded mode, potentially impacting machine protection. In both cases a detailed risk analysis and extensive tests were performed, operational system parameters were adapted, interlock levels tightened and additional interlocks for short/mid-term mitigation were implemented. This was then complemented by a vigilant supervision of the respective systems. Therefore, time consuming repairs could be delayed to the extended year end technical stop 2016/17 (EYETS).

\subsection{Nitrogen leak in the LHC dump block (TDE)}
Figure (\ref{fig:NitrogenPressure_TDE}) depicts the nitrogen pressure in the TDE line (TD68.DB) during the LHC run 2016. A leak developed at the beginning of April and was discoverd a few days later. The discovery was followed by a period of investigation and step-by-step adjustment of the operational pressure before a stable situation was reach by then end of May 2016. In addition to the existing LBDS-XPOC injection inhibit of the TDE nitrogen pressure, SIS interlocks and BigSister announcer warnings were introduced for the two beam dump blocks.
\begin{figure}[h]
   \centering
   \includegraphics[width=\columnwidth]{../paper/plots/NitrogenPressure_TDE}
   %\hspace*{0.01\columnwidth}
  \caption{Nitrogen pressure in TDE line (TD68.DB) during the 2016 run.}
   \label{fig:NitrogenPressure_TDE}
\end{figure}
During the EYETS 2016/17 the leak rate was reduced following mechanical interventions. Studies on the criticality of repeated high energy beam impacts on the carbon dump blocks in the presence of a nitrogen leak are ongoing. The implementation of a hardware warning for the TDE nitrogen pressure, possibly later complemented by a hardware interlock, as proposed by the 134th MPP~\cite{MPP134th}, has been prepared and will be finalised in a technical stop during the 2017 run. 

\subsection{Suspected inter-turn short on main dipole A31L2}
On June 10th and August 3rd 2016, unusual voltage signatures in the main dipole A31L2 were detected by the quench protection system during a ramp down of the circuit and triggered the firing of the quench heaters. The two events took place at 547~A and 295~A respectively. The measured voltage over the magnet ($U_{diode}$) and the derivative of the circuit current for the first event are depicted in Fig. (\ref{fig:MBA31L2_quench_signature}). With simulations it was shown, that this signature could be explained by the (dis-)appearance of an inter-turn short in this magnet. An inter-turn short poses the risk of magnet and collateral damage in case of a quench or a fast power abort in the concerned magnet or circuit. Following the second event, the powering of this circuit was stopped, special detection equipment was installed and powering tests were performed. As no further indication of an inter-turn short was observed, the operation was resumed. The special measurement equipment was left in the LHC tunnel to improve the supervision of this magnet and the data were regularily analysed. To reduce the probability of a magnet quench or fast power abort in the main dipole circuit of sector 12 the so-called global protection mechanism of the powering interlock controller (PIC) was deactivated for this sector. The beam loss monitor thresholds of the dipole magnets in sector 12 were reduced significantly below the quench level, accepting additional UFO dumps~\cite{Lechner+2017}. Furthermore, the triggering threshold of the quench protection system was increased for the concerned dipole magnet. Ultimately, the main dipole A31L2 was replaced during the EYETS 2016/17.

\begin{figure}[h]
   \centering
   \includegraphics[width=\columnwidth]{../paper/plots/MBA31L2_quench_signature}
   %\hspace*{0.01\columnwidth}
  \caption{dI/dt and $U_{diode}$ versus time as measured during a suspicious quench of dipole A31L2 at 547~A on 10.06.2017.}
   \label{fig:MBA31L2_quench_signature}
\end{figure}

\section{New fast failures: quench heater firing with circulating beam}
During the analysis of an UFO quench of the main dipole C28L5 on June 12, small losses in the IR7 collimation system were discovered just before the beam dump. An orbit oscillation of $\sim 10 \mu m$, caused by a skew dipole field due to the discharge of the quench heaters, was identified as the source of these losses (see Fig. (\ref{fig:Orbit_oscillation_QH_firing})). Further investigations around previous quenches and a dedicated beam experiment in a machine development block confirmed that the LHC beams are still circulating for 35 turns or $\sim 3$~ms after the triggering of the quench heater discharge by the quench detection system, before they are dumped. Table (\ref{tab:quench_heater_kick}) summarizes the expected kicks for a subset of most relevant superconducting magnets respectively sets of magnets in the LHC. The expected kicks of single magnets are non-negligible at 450 GeV but small at 6.5 TeV. In case of the simultaneous firing of quench heaters in three neigbouring dipoles or a quench in one of the triplets, the kicks reach 0.49, respectivley 1.1 $\sigma$. This will cause significant losses in the LHC collimation region. For the new HL-LHC magnets, especially the Nb$_3$Sn triplets, the kick will be even stronger. Thus, it is important to chose a quench heater layout, which minimizes the skew dipole field and to ensure that the quench heaters and comparable protection equipment are only fired after the beams have been dumped~\cite{Valette-2017}. 

\begin{figure}[h]
   \centering
   \includegraphics[width=\columnwidth]{../paper/plots/Orbit_oscillation_QH_firing}
   %\hspace*{0.01\columnwidth}
  \caption{Vertical rms orbit change after quench of main dipole C28L5 at 6.5 TeV}
   \label{fig:Orbit_oscillation_QH_firing}
\end{figure}


 \begin{table}[hbt]
   \centering
   \caption{Simulated orbit kick due to the discharge of the quench heaters for a subset of most relevant superconducting magnets in the LHC}
   \begin{tabular}{ccc}
       \toprule
      \textbf{ Kick in $\sigma$} & \textbf{450 GeV}                     & \textbf{6.5 TeV}  \\
          $\sigma$ & ($\sigma$)           & ($\sigma$)    \\
       \midrule 
         Main dipole & 0.82 & 0.21 \\
         three main dipoles       & 1.92  & 0.49       \\
          D1 (IP 2\&8)      & 0.82    & 0.22    \\
          D2 (IP 2\&8)      & 0.58    & 0.16  \\
         Triplet  & 0.8 & 1.1 \\
       \bottomrule
   \end{tabular}
   \label{tab:quench_heater_kick}
\end{table}

\section{Machine Developments}
As in previous years detailed procedures were submitted for each requested machine development test (MD) by the requestors of the MD. Based on the detailed information in the procedures, MDs were classified by machine protection experts in three classes:

\begin{itemize}
	\item class A: set-up beam ($< 5 \times 10^{11}$ protons at 450 GeV and $2 \times 10^{10}$ protons at 6.5 TeV) using nominal settings on all protection systems.
	\item class B: high intensity beam with nominal settings in all protection systems
	\item class C: high intensity beam and changes of settings of protection systems
\end{itemize}
Of all MDs in 2016 6\% were classified class A, 68\% class B and 26\% class C. All class C MDs were discussed in the restricted Machine Protection Panel (rMPP) and the procedures were approved and documented in EDMS~\cite{EDMS}.
Overall this approach worked well in 2016. Nevertheless, vigilance is required from all involved players to ensure the optimal preparation and safe performance of the MDs. The cumulation and re-scheduling of MD blocks to accomodate unforeseen limitations was challenging for the MD teams as well as for the rMPP validation process. Although the communication between the MD team and the engineer in charge (EIC) link person during the preparation phase worked in general well, an earlier involvement of the EIC link person would be beneficial.

Several ad-hoc end of fill MDs at the end of the proton - proton run, which required a last minute check and approval by the rMPP, clearly illustrated that a proper machine protection validation can only be ensured if all MDs follow the usual process of approval by the MD coordination team, classification and validation by rMPP and implementation by the operations crew.

\section{Changes to the core of the machine protection system}
During the ion run in 2016 amplifiers were added on the interlocked BPMs in IR6 to shift their sensitivity region to lower bunch intensities in anticipation of limitations. This change had previously been discussed, but its implementation was only foreseen, if unnecessary beam dumps by the IR6 BPMs occured. When reviewing the impact of the changes, it was discovered that the additional amplifiers reduced the overall reliability of the system. Therefore, the amplifiers were removed at the next occasion.

The above example shows that any changes to the core of the machine protection system in the LHC should be discussed in and approved by the Machine Protection Panel (MPP), which comprises experts from all different machine protection systems and allows for an independent feedback. In that way the consequences of changes can be first fully evaluated before they are implemented. In the future, the MPP will ensure to be more proactive during situations as described above.

\section{Conclusion}
The strategy for the intensity ramp-up after the beam commissioning in 2016 proved to be very efficient and allowed to reach an intensity of $\sim 1700$ bunches, i.e. about 75\% of the maximum stored intensity, after only 15 days. The correct functioning and response of all machine protection relevant systems was carefully analysed and documented. Therefore, it is proposed to apply the same strategy for the 2017 intensity ramp-up. Two standard scenarios for ramp-ups after short stops have been defined and used successfully in 2016. Thus, it is proposed to apply these scenarios also in 2017.

An impressive amount of loss maps and asynchronous beam dump tests has been performed, analysed and validated in 2016. To reduce the load on the respective teams and to further reduce their foot print on machine availability, a simplification of the LHC cycle and a critical review of the minimum sets required during beam commissioning, after technical stops and optics changes has been proposed. 

The classification and approval process of machine development tests worked well in 2016. In the future, all end of fill and parallel MDs should be covered by the same process.

In 2016 two important systems were operated in a degraded mode (suspected inter-turn short in main dipole A31L2 and nitrogen leak in the LHC beam dump). The implementation of mid-term mitigations following a detailed analysis allowed to postpone lengthy repairs to the EYETS 2016/17. Such cases can also be expected in the future and require a case-by-case analysis.





\section{acknowledgements}
The author would like to thank A. Apollonio, C. Bracco, G. Bregliozzi, A. Lechner, A. Mereghetti, D. Mirachi, R. Bruce, S. Redaelli, R. Schmidt, M. Valette, D. Valuch, J. Uythoven, A. Verweij, J. Wenninger and M. Zerlauth for their valuable input and the colleagues from the restricted Machine Protection Panel (rMPP) for their advice and support.

\iffalse  % only for "biblatex"
	\newpage
	\printbibliography

% "biblatex" is not used, go the "manual" way
\else

%\begin{thebibliography}{99}   % Use for  10-99  references
\begin{thebibliography}{9} % Use for 1-9 references

%\bibitem{accelconf-ref}
%	C. Petit-Jean-Genaz and J. Poole,
%	``JACoW, A service to the Accelerator Community,''
%	EPAC'04, Lucerne, July 2004, THZCH03,  p.~249,
%	\url{http://www.JACoW.org/e04/papers/THZCH03.PDF}

\bibitem{EDMS}
		EDMS, \url{https://edms.cern.ch}
		
\bibitem{Mirarchi+2017}
	D. Mirarchi et al.
	"{Collimation: Experience and Performance}"
	These proceedings
	
\bibitem{MPP134th}
		134th MPP, \url{https://indico.cern.ch/event/571908/}
		
\bibitem{Lechner+2017}
	A. Lechner et al.
	"{BLM thresholds and UFOs}"
	These proceedings
	
\bibitem{Valette-2017}
	M. Valette et al.
	"{Effect of quench heater and CLIQ firing on the HL-LHC circulating beam}"
	In proceedings of IPAC'16, Copenhagen, Denmark



		
\end{thebibliography}
%\null  % this is a hack for correcting the wrong un-indent by package 'flushend' in versions before 2015


\end{document}
	