\documentclass{JAC2003}

%%
%%  Use \documentclass[boxit]{JAC2000}
%%  To draw a frame with the correct margins on the output.
%%
%%  Use \documentclass[acus]{JAC2000}
%%  For US letter paper layout
%%

\usepackage{graphicx}
\usepackage{amsmath}


%%
%%   VARIABLE HEIGHT FOR THE TITLE BOX (default 35mm)
%%

\setlength{\titleblockheight}{35mm}

\begin{document}

\newcommand {\invfb}   {\mathrm{fb}^{-1}}
\newcommand {\lumunit}   {\mathrm{cm}^{-2} s^{-1}}
\newcommand {\betas}   {\beta^{*}}
\newcommand {\lumint} {{\cal L}_{int}}
\newcommand {\lump} {{\cal L}_{p}}

\title{Experience with MPS during the 2010 run}

\author{J. Wenninger, CERN, Geneva}

\maketitle

\abstract{In 2010 the LHC stored beam energy was pushed to 25 MJ, ten times above TEVATRON, in little over 6 months. No machine protection issues were recorded, and the reliability of the machine protection system (MPS) did not impact beam operation in a significant way. After an initial phase of low intensity beam operation that was used among other things for the commissioning of the MPS, the intensity was increased in steps of a factor 2 up to 2 MJ. Following a stability run at 2 MJ, the intensity was increased in steps of around 3 MJs every few days during train operation. The intensity steps and upcoming MP issues were approved and discussed in the restricted Machine Protection Panel (MPPr) composed of representatives from the main MP sub-systems. Two reviews of the MPS were organized in 2010, one internal and one external review. This presentation will discuss the performance of the MPS, the experience from the MPPr and of the intensity increase and the outcomes of the reviews.}

\maketitle                 % Produces the title block including the abstract

\section{Introduction}

March and parts of April 2010 were largely devoted to commissioning with beam of the LHC MPS following the predefined procedures. The test plans where uploaded on WEB pages and the test results were filled by the MPS experts. The MPP chairman checked the results and ensured that no steps were skipped or forgotten.
The discipline for filling in test results was good and the plans were followed.
No major issues or availability problems encountered in this phase.

The same period saw the first collimator setups, including validations with loss maps and de-bunched beams (asynchronous beam dump simulations). The setups were verified periodically, but not at the predefined rate of once per week. The alignment (or at least retraction) of TCT collimators was verified by a MPP responsible for each fill of the high intensity period using the post-mortem data. The performance of collimation and protection systems was stable along the year, outlining the excellent stability of the machine (orbit, optics and collimator positioning).
But it must be noted that the stability is not yet sufficient for nominal tolerances.

\section{Steering the intensity increase}

The intensity increase was steered in 2010 through the restricted Machine Protection Panel (MPPr). It was 
composed of MPS experts from the main MP sub-systems, the head of the LHC OP group and the LHC physics coordinator (R. Assmann, B. Goddard, J. Uythoven, B. Dehning, M. Zerlauth,  A. Siemko, R. Schmidt, J. Wenninger,  M. Lamont,  M. Ferro-Luzzi). The MPPr provided recommendations on the MPS envelope (maximum intensity) to be approved by the LMC.
From the beginning the plan foresaw 3 phases:
\begin{itemize}
  \item Low intensity for commissioning and early experience.
  \item Ramp up to 1-2 MJ followed by a stable operation period of around 4 weeks that intensity.
  \item Breaking the World record of stored beam intensity and move into 10’s of MJ regime.
\end{itemize}
The actual intensity ramp up in 2010 is shown in Fig~\ref{fig:evol2010}, and the three phases are clearly visible.

\begin{figure}[pth]
   \centering
   \includegraphics*[width=\columnwidth]{JW_6_01.f1.eps}
   \vspace{-2mm}
   \caption{Evolution of the peak stored beam energy in stable beams in 2010.}
    \label{fig:evol2010}
\end{figure}

Figure~\ref{fig:evolplan} compares the intensity ramp up plan approved by the LMC in February 2010 with the achieved ramp up. There are a number of notable differences. The plan assumed that commissioning for the different phases (for example single bunches to trains) would be performed ‘in the shadow’ of physics operation. Train operation was assumed to be run with 50~ns trains of $8 \times 10^{10}$~p/bunch. The reality turned out to be rather different:
\begin{itemize}
  \item Commissioning was not transparent and could not be done in parallel to physics operation. Dedicated periods were devoted to commissioning of higher bunch charge (June) and trains (September).
  \item Higher bunch charges were eventually used (up to $1.2 \times 10^{11}$~p/bunch).
  \item In the last phase the intensity followed a much steeped slope than anticipated because no problems were encountered, see also Fig.~\ref{fig:evolplanlin}.
\end{itemize}
In the final phase the slope was four times steeper than what had been planned: this was possible thanks to the excellent performance of the entire machine and in particular of the collimation~\cite{ASSMANN} and MPS.

\begin{figure}[pth]
   \centering
   \includegraphics*[width=\columnwidth]{JW_6_01.f2.eps}
   \vspace{-2mm}
   \caption{Evolution of the peak stored beam energy in stable beams in 2010 compared to the plan outlined at the LMC in February 2010.}
    \label{fig:evolplan}
\end{figure}

\begin{figure}[pth]
   \centering
   \includegraphics*[width=\columnwidth]{JW_6_01.f3.eps}
   \vspace{-2mm}
   \caption{Evolution of the peak stored beam energy in stable beams in 2010 compared to the plan outlined at the LMC in February 2010 on a linear scale. In the last high intensity phase the achieved slope is 4 times steeper than the planned slope.}
    \label{fig:evolplanlin}
\end{figure}

\subsection{Too fast or too slow?}

When everything went well it is easy to conclude (a posteriori) that we could have progressed faster!
We tend to forget that we had a steep but also sometimes rocky learning curve (OP + MPS) in parallel to the intensity increase .
MPPr recommendations were the outcome of agreements (or compromises) among ALL MPPr members – some more conservative, some more aggressive.
In many cases operational issues played a significant role (QFB versus damper, orbit stability…).
‘Afterglow’ of the TT40 incident was still on some minds.
More aggressive colleagues and coordinators were a bit frustrated…

The intensity increase in the last phase corresponded to stored energy steps of ~3 MJ every 3 fills + 20 hours collisions.
Within a factor 2 of a super-aggressive rate: 1 fill of 10 hours.
Issue of controlling UFOs in this phase:
BLM threshold increase first by a factor 3, towards the end even by a factor 5.
We could have considered larger steps towards the end when the fractional increase became rather small.

The intensity increase plan was reasonable given that we were in a commissioning year.
Overall the progress followed recommendations of MPPr.
MPPr was over-ruled twice. Intensity within ‘factor 2’ of recommendations.

\section{Reviews}

Two reviews of the LHC MPS were organized in 2010, first an internal review~\cite{INTREVIEW} in June 2010, and later an external review~\cite{EXTREVIEW} in September 2010. The internal review was help before increasing the intensity towards one MJ, and it also served as a preparation for the external review. The external review took place after the longer operation period in the range of 1-2~MJ and before ramping up the stored beam intensities to new World records.
The external review committee was composed of MPS and operation experts from FNAL, BNL, GSI, DESY, SNS and CERN. It was chaired by R.~Bacher of DESY. The external review provided a detailed snapshot of the MPS state. The review committee made 11~recommendations:
\begin{itemize}
  \item None of the recommendations was a show stopper for the intensity increase.
  \item The committee expressed strong concerns around configuration and sequencing. As a consequence a major sequencer clean-up was made by the OP group under the super-vision of L~Ponce~\cite{PONCE}.
  \item All points have been (or will be) addressed.
\end{itemize}
In parallel to those main reviews, two sub-system reviews have been organized:
\begin{itemize}
  \item BLM system FPGA code review.
  \item LBDS TSU review (Trigger Synchronization Unit).
\end{itemize}

\section{Surprises}

\subsection{Quench and damage}

A real surprise of the 2010 run was the absence of ACCIDENTAL beam induced quench was with circulating beam. This outlines the excellent performance of BLM and collimation systems.

The only (known) damage to an LHC machine component is the beam2 wire-scanner that almost evaporated during a quench test, when the wire speed had to be reduced to 5~cm/s (from 1~m/s) in order to quench the D4 separation dipole~\cite{SAPINSKY}.
This test was almost fatal to the wire: the Carbon wire diameter was reduced from 30 to 17~$\mu$m over a length corresponding to size of the beam.

\begin{figure}[pth]
   \centering
   \includegraphics*[width=\columnwidth]{JW_6_01.f4.eps}
   \vspace{-2mm}
   \caption{Damaged beam2 wire. Over a length corresponding to the width of the beam, the wire thickness is reduced to 17~$\mu$m (nominal 30~$\mu$m).}
    \label{fig:wsdamage}
\end{figure}

\subsection{UFOs}


The very fast beam loss events (time scale of 1~ms) in cold regions of the machine have been {\bf THE other surprise} of the 2010 run, see Fig.~\ref{fig:ufoexa}. Those events have been nicknamed UFOs (an acronym  borrowed from nuclear fusion community where similar events are observed in plasmas). 18 dumps were triggered by UFO-type events, and more than 100 events that remained below the BLM dump threshold have been found in the logging data. The most likely cause of the UFOs are
small $\mu$m sized objects (dust…) ‘entering’ the beam. Some events were correlated in time and space to roman pot movements. Depending on the mass, it is possible that the particles charge up by ionization and are re-expelled from the beam. More details can be found in Reference~\cite{SAPINSKY}.

Figure~\ref{fig:ufoint} shows the correlation of the number of UFOs that dumped the beams and the integrated circulating beam intensity. After the increase of the BLM threholds by a factor of 3, the number of UFO triggered beam dumps per integrated intensity decreased by a factor 4.1. A simple extrapolation for the 2011 run with 950 bunches leads to 
one UFO induced dump every 10 hours

\begin{figure}[pth]
   \centering
   \includegraphics*[width=\columnwidth]{JW_6_01.f5.eps}
   \vspace{-2mm}
   \caption{Time evolution of the losses for a UFO that dumped the beam. One bin corresponds to a 40~$\mu$s time interval.}
    \label{fig:ufoexa}
\end{figure}

\begin{figure}[pth]
   \centering
   \includegraphics*[width=\columnwidth]{JW_6_01.f6.eps}
   \vspace{-2mm}
   \caption{Correlation between the number of UFOs and the integrated circulating beam intensity. The slope change occurred after an increase of the BLM thresholds by a factor~3.}
    \label{fig:ufoint}
\end{figure}

\subsection{Asynchronous dumps}

The first asynchronous dump  was recorded for beam1 on Friday November 19th at 450~GeV with a circulating pilot bunch. The event therefore occurred in rather ‘favorable’ conditions as seen from MP. The event was initiated by a fault on a trigger fan out unit. Both diagnostics and reactions to the event were correct:
\begin{itemize}
  \item The faults were detected by the LBDS IPOC and XPOC systems.
  \item A test dump revealed a missing trigger (reduced redundancy).
  \item Access to repair followed by revalidation.
\end{itemize}

The dump was however ‘doubly’ asynchronous since it involved 2~MKD kickers and not one as expected. This was due to a  change in the trigger fan out signal distribution (with respect to the initial design) following a reliability analysis. The cabling of the trigger fan outs will be restored in 2011 to the nominal ‘specifications’.

\section{Statistics}

A detailed analysis of the protection dumps was performed by M. Zerlauth at the Evian Workshop in December 2010~\cite{ZERLAUTH}. We  present here only some selected points.

Above injection energy 47 of 370 (13\%) of protection dumps were triggered by the BLMs.
Most of the dumps occurred prior to the increase of the BLM thresholds on various cold and warm elements.
The causes of the BLM dumps are shown in Fig.~\ref{fig:blmdumps}: the UFOs were dominant, other triggers occurred mostly during MPS tests and setups such as loss maps, wire scans and quench tests.
All failures were captured by the BLMs before quenching any magnet (the QPS providing the ‘ultimate’ redundancy).

The dependability and availability of the machine protection systems has been a major design criteria. It was subject to extensive studies using the FMECA approach (Failure mode, effects and criticality analysis).
The MPS dependability studies  were confirmed in 2010, see Table~\ref{tab:falsedumps}.


\begin{figure}[pth]
   \centering
   \includegraphics*[width=\columnwidth]{JW_6_01.f7.eps}
   \vspace{-2mm}
   \caption{Initiating events for the protection dumps triggered by the BLM system.}
    \label{fig:blmdumps}
\end{figure}

\section{Conclusion}

The LHC Machine Protection Systems have been working extremely well during 2010 run thanks to the commitment and rigor of operation crews and MPS experts. Most failures are captured before effects on the beams are seen, no quenches occurred with circulating beam.
	
Controlling (and understanding) UFOs could become a main issue in 2011. The BLM thresholds may have to be adjusted and possibly increased to probe the limit of the quench.

Steering of the intensity increase through MPPr should be pursued in 2011. The intensity increase plan for 2011 must be be defined, and the experience of 2010 should be integrate to optimize the plan.

An improved tracking system for ALL MPS changes must be put in place for 2011, in particular in view of the MD periods:
a safe recovery and pre-flight MP compatibility checks will become essential.

\begin{table}[bth]
\begin{center}
\begin{tabular} {|l|c|c|} \hline
 System  & Expected & Observed \\ \hline \hline
 LBDS    & 4 & 9 \\
 BIC    & 0.5 & 0.5 \\
 BLM  & 17 & 3 \\
 PIC   & 1.5 & 2  \\
 QPS   & 16 & 11 \\
 SIS    & -- & 4.5  \\ \hline \hline
 Total & 41 $\pm$ 6 & 31 \\
   \hline
\end{tabular}
\caption{Expected and observed number of 'false' (internally triggered) dumps for each of the main MPS sub-systems. One event is shared and BIC and SIS. The observed dumps correspond all to energies above 450~GeV.} \label{tab:falsedumps}
\end{center}
\end{table}


\begin{thebibliography}{10}

\bibitem{ASSMANN} R. Assmann, {\it these proceedings}.

\bibitem{INTREVIEW} Internal review of the LHC MPS, June 17th-18th 2010,
http://indico.cern.ch/conferenceDisplay.py?confId=97349

\bibitem{EXTREVIEW} External review of the LHC MPS, Sept. 6th-8th 2010,
http://indico.cern.ch/conferenceDisplay.py?confId=103908

\bibitem{PONCE} L. Ponce, {\it these proceedings}.

\bibitem{SAPINSKY} M. Sapinsky, {\it these proceedings}.

\bibitem{ZERLAUTH} M. Zerlauth, {\it Proceedings of the Evian Workshop}, December 2010.

\end{thebibliography}

\end{document}                        % End of document
