
\documentclass[a4paper]{jacow}
%%
%%  This file was updated in April 2009 by J. Poole to be in line with Word tempaltes
%%
%%  Use \documentclass[boxit]{JAC2003}
%%  to draw a frame with the correct margins on the output.
%%
%%  Use \documentclass[acus]{JAC2003}
%%  for US letter paper layout
%%

\usepackage{graphicx}
\usepackage{booktabs}

%%
%%   VARIABLE HEIGHT FOR THE TITLE BOX (default 35mm)
%%

\setlength{\titleblockheight}{27mm}

\begin{document}
\title{TUNE AND ORBIT FEEDBACKS}

\author{L. Ponce, J. Wenninger, K. Fuchsberger and S. Jackson, CERN, Geneva, Switzerland}

\maketitle

\begin{abstract}
  Major refactoring of the Tune and orbit Feedbacks systems has been performed during LS1
  to improve stability and performance. The talk is presenting the changes implemented
  in the system with the testing Framework put in place. The main issues encountered
  during the beam commissioning are also reminded as well as the configuration used
  all along the LHC cycle. Finally, performance of the Orbit Feedback system is
  summarized through some orbit stability analysis.
  
 
\end{abstract}

\section{NEW FEEDBACKS FOR 2015}
\subsection{Description of the systems}
The tune measurement system is comprised of 4 independent acquisition systems:
\begin{itemize}
\item {\bf On demand} used to perform measurement requiring changes in the acquisition chain 
  settings or beam excitation,
  \item {\bf Continuous GATED} and
{\bf Continuous High Sensitivity} used for nominal bunch intensity beam for tune 
feedback and continuous tune measurement,
\item {\bf Dev} system used for beam studies and as hot spare.
  \end{itemize}

\begin{figure}[htb]
   \centering
   \includegraphics*[width=60mm]{figures/TuneSystem.eps}
   \caption{Tune measurement systems architecture.}
   \label{QFB}
\end{figure}
In principle, the Tune Feedback System (QFB) can
be configured to use any of the devices, but in practise, only the 2 CONTINUOUS devices (GATED and High Sensitivity)
were used during 2015 as feedback functionality implies that acquisition settings are fixed

The Orbit Feedback (OFB) system is comprised of 2 different parts, implemented into 2 different 
software: the LHC Feedback Service Unit, so-called BFSU (formally OFSU) and the Orbit Feedback Controller, 
so called OFC.

\subsection{Feedbacks after LS1}
During Long Shutdown 1 (LS1), the Orbit Feedback system went through major re-factoring due 
to mandatory software migration and change of main developers.

The first important change concerned the Hardware. 
The old machines (4xCore, 32-bits running SLC5) have been replaced by new machines (24xCore with
 64-bits processor) ported to SLC6.
An extra timing receiver card has also been added to the OFC.


The BFSU class has been migrated from FESA2 to FESA3. More than 35000 lines of code  have been 
ported mainly manually as the standard migration tools were not fully mature at the start 
of the process in July 2014. This represented a major effort of 3-4 months (without testing) for the BE/BI-SW team.
As the code functionality and content were mostly unknown in advance, a systematic approach has been taken.
The team had to understand, document and use JIRA for managing the changes resulting  from the porting process.
At least 1 issue was created for the porting of each action (now > 130 issues for BFSU porting alone)
Code commits were (and still are) tagged with the issue number, this allows for automatic cross-referencing
of issues and SVN commits.


Specific porting technicalities were encountered in the FESA2->FESA3 porting process:
many issues with inherent 32-bit assumptions in the original code, 
handling of arrays changed from FESA2 to FESA3, many data-consistency issues 
with settings-properties setting acquisition-fields which used to be allowed in 
FESA2 but not in FESA3, etc...


The Controller (OFC) has also be ported to FESA3 with 2 main changes. 
Explicit types have been declared and
the ROOT based logging has been replaced with standard logging system which allows
 to now use standard BE/CO tools for logging analysis.

In parallel of the porting, several changes have been made to improve stability.
As a result from taking standard SLC version of ROOT, the code no longer needs to be built on operational hardware.
Both Service Unit and Controller code have been cleaned-up: 
un-needed features (e.g BCT subscription/concentration) have been disabled from controller and 
several bugs and unchecked string manipulation
have been identified and fixed.
Most of the critical settings have been moved into LSA settings 
for less reliance on FESA persistency for settings.
And finally, one of the major source of Ethernet overload was removed by balancing better the settings 
exchange between the Service Unit and the Controller over 10 seconds intervals and by reducing the size  
data structure.


\subsection{Feedbacks Testing Framework}

 In parallel of code refactoring, a Testing Framework 
was put in place as a collaboration between BE-CO, BE-BI and BE-OP groups. The idea was to 
set-up a controlled testing outside operational environment to have less impact on 
operations and being able to test the systems before beam is back.

Different types of tests have been identified and then implemented to exercise the FESA mechanics, 
communication between the different components and the Control loop behavior. 

The principle is to inject directly tests through UDP layer~(Fig.~\ref{TB}), allowing a free choice of
 the language. Most of the tests have been implemented in JAVA, based on Junit test. 
This gives the advantages of a bigger community to write test, an easy interaction with the other 
parts of the control system and allow a first experience with JAVA real time behavior.

\begin{figure}[htb]
   \centering
   \includegraphics*[width=80mm]{figures/testbed.eps}
   \caption{Testing Framework principle}
   \label{TB}
\end{figure}

Thanks to the TestBed Framework, functional tests of the Controller and the Service
Unit software have been performed allowing to test new control algorithms and simulate 
non-ideal beam behavior. Various operational scenarios have also been tested: simulation of 
instrumentation faults, simulation of (intermittent) network faults, simulation of packet corruption,
delays, jitters, etc...
At the same time, stress tests and acceptance tests have been implemented and ran 
regularly.

The methodologies and the model put in place will be used by BE-BI group in 2016 when 
migrating other big projects to FESA3 (LHC BLM, Tune systems...).

\section{OPERATIONAL USAGE OF THE FEEDBACKS}

\subsection{Interface with operation}
Several properties are settable for operation, either via sequencer 
or via dedicated application (YASP). The main ones, already available during Run~1,
are the On/Off command, loading
of references (both for orbit and tune feedbacks) and setting of optics via the 
BFSU, dynamic change of the references via timing 
events. 

After LS1 re-factoring, new settings, which used 
to be experts settings only, have been exposed to operation. They are now stored in LSA and are trimmable either 
via sequencer or via YASP application, with the advantage of logging of all the changes.
These settings are the gain of the different feedbacks, the number of Eigen values
used for derivation of response matrix and the status of correctors/monitors.


\subsection{Change of reference}
During the ramp and squeeze, the reference orbit is changing because the amplitude and the shape
of the bumps around the IPs are changing. The reference tunes are also changed from injection
settings to collisions settings at flat top. Both changes are played over several seconds/minutes
following the power converters functions and these changes have to be propagated to the
references used by the feedbacks.


For the OFB, the settings (BPM readings/COD settings) of each reference are stored in LSA and
sequencer tasks exist to load the reference and arm the feedbacks.
To be noted that both Tune and Orbit feedbacks
are not really playing a function as the power converters. Feedbacks controller
fit a linear extrapolation between the actual settings and the requested settings over
a time also set by the sequencer tasks and encoded in the timing event
(called {\it timeConstant}). 

The change of reference is triggered by specific timing events as long as the feedbacks stayed
{\it armed} and the same timing event with a payload greater than 99 is used to {\it disarm}. 
All timing events needed to follow the change of references are
generated before launching a given Beam Process in a dynamic timing table which is started at the
same time as the Beam Process itself, Fig.\ref{timing Table}.

\begin{figure}[hbt]
   \centering
   \includegraphics*[width=50mm]{figures/timingTable.eps}
   \caption{Dynamic table table containing timing events to trigger optics and orbit reference changes.}
   \label{timing Table}
\end{figure}

\begin{figure}[htb]
   \centering
   \includegraphics*[width=50mm]{figures/referenceList.eps}
   \caption{List of orbit references used for the squeeze to 80~cm.}
   \label{references}
\end{figure}
Both tune and orbit references are set on 2 different properties of the BFSU
and the change is triggered by a dedicated timing event, introducing a critical
dependence of the BFSU FESA class stability all along the beam cycle: as an example,
11 timing events are used in the squeeze
to 80~cm, Fig.\ref{references}.

\subsection{Change of optics}
The optics are used by the OFC to compute the SVD response matrix of the orbit feedback.
In principle, this matrix should be recomputed for each matched point during the squeeze, but
it is a quite lengthy process which can take between 1 and 2 minutes.

The dynamic change of optics during the squeeze also via timing event is implemented but
was never deployed in nominal operation for different reasons.
During Run~1, the number of optics
loaded and calculated was reduced to avoid stability problem of the OFSU (former version of the BFSU):
the process was not able to compute more that few optics. Moreover, the re-computation of the matrix with the
feedback ON acting on the beam was never tested and then is raised the question of usage of the change of optics
with a matrix re-computation time of the order of a minute comparable with the length of the segements.
The last 2 points need more time with beam to test the stability and evaluate the use.

As a consequence, during 2015 operation, the matrix was recomputed only once before the
start of the squeeze in a discrete mode: with orbit feedback OFF, the matrix is calculated
with an intermediate optics (namely the 2.5~m optics for the squeeze to 80~cm) and the new
references loaded before Feedback is switched ON again, the squeeze being played with the
same response matrix.


\subsection{Feedbacks settings along the cycle}

Thanks to the new experts settings made available to operation during LS1, the usage of the
different feedbacks along the cycle has been extended.
\begin{description}
\item{\bf In preparation without beam}, the optics are fetched for the whole cycle,
a default response matrix is derived ignoring all BPM statuses and default settings of
feedbacks response are loaded ({\it OFB Gain}~=~1, {\it Eigen values}~=~390-420 and
{\it Radial loop Gain}~=~0.5). The devices for the tune feedback are also set
for pilot beam.
\item{\bf During injection} the orbit feedback is manually switched ON to correct the orbit towards the
reference, with pilot beams and/or physics beam.
The tunes are corrected manually towards the references. The Tune devices are switched
to the appropriate ones depending on the injected beam intensity (GATED devices for trains)
as part of the sequence to prepare injection of physics beams.
\item{\bf During prepare ramp} both Tune and Orbit feedbacks are
switched ON via sequencer correcting to the injection references. The change of
orbit reference is also loaded and OFB is armed.
\item{\bf During the ramp} the tune references are kept constant, the orbit reference (crossing and
separation) is changed at 2~TeV. The used settings are {\it OFB Gain}~=~2, {\it Eigen values}~=~390-420 and
{\it Radial loop Gain}~=~0.5. The OFB gain is increased to improve the correction efficiency
during the snapback.
\item{\bf During the change of Tune}, both Orbit and Tune feedbacks are kept ON. The
orbit reference is constant and the tune references (and the fitter settings)
are changed for physics settings linearly over
the QCHANGE Beam Process length.
\item{\bf In preparation of the squeeze}, the response matrix is calculated with the 2.5~m optics and the
11 orbit references are loaded, OFB is armed.
\item{\bf During the squeeze}, the Tune feedback is kept ON with constant references.
The Orbit feedback is also ON, changing orbit references at each matched point with a linear
extrapolation between 2 segments to follow the change of separation and crossing bumps shape
with the optics. The response settings are changed to {\it OFB Gain} = 1, {\it Eigen values} = 390-420 and
{\it Radial loop Gain}= 0.5.
\item{\bf During the preparation of collisions}, both feedbacks are switched OFF, but the
change of orbit references is loaded and played to follow the collapsing of the
separation bumps. Orbit is manually corrected using YASP application at the end of the collisions Beam
Process towards the physics orbit reference.
\item{\bf In collisions}, during th second part of 2015 run, the orbit feedback is ON with
lower gain and Eigen values ({\it OFB Gain}~=~0.2, {\it Eigen values}~=~40-40 and
{\it Radial loop Gain}~=~0.02) to smoothly correct the orbit drift during STABLE BEAMS,
in particular the drifts due to IP8 triplet movements.
After optimization of the collisions (including leveling), the actual orbit is used
as reference for the feedbacks.
\end{description}

The different settings of the feedbacks along the cycle are summarized in Table~\ref{FB gains}.

\begin{table}[htb]
\center
\begin{tabular}{|c|c|c|c|}\hline
  Mode & OFB  & OFB/ Radial & Eigen   \\
   & State & Gain & values   \\\hline
 Preparation & OFF & 1 / 0.5 & 390-420 \\\hline
 INJECTION & {\it ON} &  1/ 0.5 & 390-420 \\\hline
 RAMP & ON & 2 / 0.5& 390-420 \\\hline
 QCHANGE & ON & 1 / 0.5 & 390-420 \\\hline
 SQUEEZE & ON & 1 / 0.5 & 390-420 \\\hline
 COLLISIONS & OFF & 1 / 0.5 & 390-420 \\\hline
 STABLE BEAM & {\it ON}  & 0.2 / 0.02& 40-40 \\\hline
\end{tabular}
\caption{Summary of the Orbit feedback state and settings for the different modes
  of operation in 2015. At injection, the OFB is manually switched on for a short time
to correct towards injection orbit reference. In italic are the manual action.}
\label{FB gains}
\end{table}


\section{MAIN ISSUES DURING 2015 OPERATION}

After LS1, most of the stability problems that
punctuated Run~1 operation have been fixed. We nevertheless had 3 beam dumps due
to Tune Feedbacks in 2015, all due to tune signal quality problems and interference with
the transverse dampers. We also experienced few teething problems with the orbit feedbacks
(wrong reference or timing event not received),
which did not generate any beam dumped.

\subsection{Tune measurement quality}

The performance of the tune feedback is highly depending on the tune signal quality and
more specifically to a good signal to noise ratio. During 2015 operation, we still had 3 dumps
because the tune feedback locked on a wrong tune peak (noise peak) value and the real tunes
were pushed towards resonance leading to beam dumped triggered by high beam losses.

The saturation problem was solved by using different devices for the pilot and nominal bunch
intensity beams. The switch between the devices is in the corresponding sequence and
hardware settings of the devices have been adjusted for 2015 proton intensities.
In addition of the intensity switch, the devices are also switched between GATED and Continuous HS
for single bunch or trains
beam structure. The problem of co-existence between tune measurement and transverse feedback
effect (tune peak damped) was solved by using The GATED tune system: the tune measurement
is gated on a limited number of bunches (the first 400) which also experience a reduced
ADT gain (factor 100 below NOMINAL ADT gains).
The window of the GATED device is now configurable from the tune viewer application, Fig.~\ref{GATED},
as well as the value of the ADT gain (trimmable in LSA). These 2 options allowed an optimization during the
intensity ramp-up so that the TUNE signal measurement improved allowing running with a good quality
of the measurement and then the Tune feedback ON during the nominal physics production.

\begin{figure}[htb]
   \centering
   \includegraphics*[width=50mm]{figures/GATED.eps}
   \caption{Settings of the GATED tune device measurement.}
   \label{GATED}
\end{figure}

Operation with ions was more difficult as the GATED tune device system was not re-calibrated
for ions intensity and then tune measurement was not very clean during ions run.

The co-existence of tune measurement in the vertical plane with the abort gap cleaning
is still a problem and prevented the usage of the abort gap cleaning during the squeeze when
tune feedback ON is needed.

\subsection{Loss of timing event}

At the beginning of the run, it was observed few times problem with the triggering of
change of reference orbit, critical during the ramp as the crossing angles bump amplitude are
changing. This was traced back to a configuration problem of the O/S of the BFSU 
in the processing of interrupts: 2 timing events, one to change the optics and one to change
the orbit reference, were sent too close to each other and one of them was not treated.

Several mitigation were put in place in parallel to avoid critical situation with high
intensity beams. First of all, the configuration of the BFSU machine has been corrected to
 handle properly the interrupts. At the same time, a delay of 500~ms was introduced
in the timing table between the event to change the orbit and the event to change
the optics, the timing for the optics ID change not being critical.
After these changes, no more loss of timing event were observed, but the delay between
the event has never been removed.

\subsection{Wrong orbit references}

Since Run~1, the reference orbits are stored in LSA with the optics tables. The change of
reference is driven by sequencer using the TIMING USER associated to the Beam Process in the active hypercycle.
The automation with the sequencer worked very well for the nominal operation, but during
special run/Machine Development when cloning hypercycle, it happened 2 times that wrong orbit
references were used because of re-use of the Beam process.

\section{ORBIT FEEDBACKS PERFORMANCES}

\subsection{Systematic orbit errors}
The orbit feedback performance was improved during 2015 compared to Run~1 thanks to an
improved orbit measurement quality. The BPM rack cooling has been improved during LS1 to limit
the orbit drift with temperature. The remaining affect after correction is smaller by a
factor 5 to 10 compared to Run~1, Fig.~\ref{orbit error}. 

\begin{figure}[htb]
   \centering
   \includegraphics*[width=70mm]{figures/OrbitErrors.eps}
   \caption{Effect of temperature on the orbit measurement: reference orbit is the orbit
     at the beginning of the STABLE BEAMS.}
   \label{orbit error}
\end{figure}
Some crates
could still be improved around point 3,4 and~5, but the improved quality allowed to
use the orbit feedbacks during physics. 

\subsection{Performance during the squeeze}
Orbit correction during the squeeze is made with the Feedback combined with regular FeedForward
of the corrections. After LS1, orbit corrector functions are forced to follow a PLP segment.
Trim and FeedForward is only possible in the matched points, allowing a smoother orbit in the
squeeze without need to use the Feedback with High gain values.

\begin{figure}[htb]
   \centering
   \includegraphics*[width=40mm]{figures/orbitSqueeze1.eps}
    \includegraphics*[width=40mm]{figures/orbitSqueeze2.eps}  
   \caption{Orbit excursion during the squeeze in horizontal and vertical plane.}
   \label{FF}
\end{figure}
In Fig~\ref{FF} are shown the
orbit excursion in horizontal and vertical plane along the squeeze, in red the best
orbit in 2012 after heavy FeedForward following a special high gain OFB pass through the squeeze
and in blue an example of 2015 with a simple FeedForward in the matched point.

Some horizontal transients remains between the start of the squeeze and the first matched point at
85~s. Operating with a gain 2 times higher could bring down the orbit excursion below 20~$\mu$m,
Fig~\ref{Highgain}, but the source of the perturbation still need to be analyzed.

\begin{figure}[htb]
   \centering
   \includegraphics*[width=80mm]{figures/OrbitGain2.eps}
   \caption{Orbit excursion during the squeeze with nominal gain in blue and with
   2 times nominal gain in red.}
   \label{Highgain}
\end{figure}

\subsection{Orbit Feedback during STABLE BEAMS}

After LS1, orbit drift up to 0.2~mm rms has been observed with a period of 8~hours,
see Fig.\ref{orbit drift}. The drift period is in clear correlation with the IR8 triplet
movement which has been present since the triplet cryostat is filled with Helium,
Fig.\ref{triplet WPS}. 

\begin{figure}[htb]
   \centering
   \includegraphics*[width=60mm]{figures/triplet1.eps}
   \caption{Orbit rms evolution during STABLE BEAMS.}
   \label{orbit drift}
\end{figure}
\begin{figure}[htb]
   \centering
   \includegraphics*[width=60mm]{figures/triplet2.eps}
   \caption{Wire Position System (WPS) measurement of the IR8 triplet over several days
   at the beginning of the run.}
   \label{triplet WPS}
\end{figure}

The cause of the triplet movement is not yet understood despite several tests made along
the year~\cite{LMC}. To compensate the orbit drift, the Orbit Feedback was used during the STABLE
BEAM period with gentle corrections (limited number of Eigen values and reduced gain) and excluding the
interaction region to not interfere with the luminosity leveling.

\subsection{Long-term orbit reproducibility}

The evolution of the rms orbit excluding the areas close to IR1, IR2, IR5 and IR8 is an
indicator of the long term reproducibility of the orbit, Fig.~\ref{reproducibility}.
It includes the BPM errors, ground motion, Orbit Feedbacks corrections, etc...

\begin{figure}[htb]
   \centering
   \includegraphics*[width=80mm]{figures/stability.eps}
   \caption{Arc orbit rms evolution for the 2 beams, 2 planes from fill to fill.
     Each point represent one fill and is taken
     after 5~min in STABLE BEAMS. The rms is calculated over healthy BPMs except IR1,2,5 and 8
   bump region wrt orbit reference in the 25~ns period at day 85.}
   \label{reproducibility}
\end{figure}

The orbit reproducibility is within 50~$\mu$m over 3 months. Long and short term
(from fill to fill) reproducibility are very similar, slightly worse in the horizontal
plane due to IR8 triplet movement.

\section{CHANGES FOR 2016}
No major change is planned to be implemented during the Year End Technical Stop for the
operational feedbacks. Few ideas to improve the usability are nevertheless under study. The first one
is the implementation of function for gains, to allow a more flexible operation. The
Feedbacks gains could be pushed during the first 120~s of the ramp and the first
segments of the squeeze where the largest orbit and
tune perturbations are measured.

For the Tune Feedback, a factor 2 in the efficiency could
be gained by changing the default acquisition settings, moving from 4000 turns at 4~Hz to 2000
turns at 8~Hz.

A re-factoring of the reference orbit construction and storage based on the knob concept is
under development and could help for an easier management of the orbit references.

Several developments are also planned for the TestBed. The first one is to add more tests to
increase the coverage of the use-case and code checks. Adding automatic builds to test at every
change and additional reports should improve the code quality and efficiency of the tests.
A more ambitious plan is to close the loop on the operational scenario tests to explore
dynamic stability limits of the algorithm: capture the controller magnet corrections, recalculate
(simulate) the beam reaction, send back the response to the BFSU.


\section{SUMMARY}

During LS1, the Orbit Feedback Systems went through a major re-factoring due to change of
hardware and migration of the software version. Stability of the systems and operability
have been improved a lot allowing a stable operation of the system during the year.
More experts settings have been made available for standard operation allowing an easier
setting-up and commissioning of the systems.

After few teething problems being solved at the very beginning of the year (software configuration
for the Orbit Feedback and signal quality for the Tune feedback), nominal proton operation
went smoothly with both Feedbacks systems used all along the cycle. We moved from a mode
of operation with Feedbacks disabled during run~1 to a mode of operation with OFB switched ON during
all the STABLE BEAMS period.

\begin{thebibliography}{9}   % Use for  1-9  references
  %\begin{thebibliography}{99} % Use for 10-99 references
  
\bibitem{LMC}
  G. Ferlin, Cryogenic investigation plan for IT IR8 movement, at the LHC Machine Comittee 248$^{th}$meeting.
\end{thebibliography}

\end{document}
