One of ROOT's main ingredients is its interpreter.
It is the primary way of user interaction, be it in C++, Python, or Ruby.
The interpreter also enables key features like the plugin mechanism, signal-slot, and auto-loading of libraries.
Information on known C++ types is the basis e.g. for writing LHC's data, for ROOT's GUI, and for ROOT's documentation system.
Users can develop their code interactively using the interpreter in an explorative process.
This is in fact the preferred way of coding for many physicists when refining their algorithms.
Especially the C++ front-end is useful in that respect, because it allows a smooth transition to compiled code once the algorithm design phase is finished and performance becomes relevant.
Type information for e.g. I/O and the interpreter can be collected by ROOT's interpreter CINT or by a combination of an external program (GCCXML) and a set of python scripts.
Having two options results in a large amount of code duplication and in inconsistencies:
bugs fixed for one option will most probably still exist in the other one;
improvements in one option must be completely rewritten for the other one.
While CINT has a list of known deficiencies that have proved to be difficult to fix, the GCCXML-based approach is (slowly) developed outside our control, depends on the GCC parser and what it provides, and is causing a large build time overhead.
As an example, we are going to hit a dead end with GCCXML soon, because it stopped providing information that is highly relevant for ROOT's I/O system.
This will mean an extra work load for maintaining a special CERN GCCXML version or the lack of support of any recent compiler and the end of support from the the GCCXML developers.
Thus both approaches have major disadvantages, which means none of them can be easily replaced by the other.
By chance, there has been a new development regarding compilers, parsers, and interpreters, that we foresee as a candidate for solving all of these issues:
LLVM (http://llvm.org) has been developed as a completely new compiler.
In fact it is much more:
it is a framework for parsing, compiling, and if needed executing code from several languages, C++ being one of them.
It is written in C++, has been shown to compile code much faster than GCC --- as a matter of fact, this was one of the initial motivations for starting the LLVM project.
It's compilation performance (i.e. the speed of building code) and its optimization performance (i.e. the speed with which compiled code executes) are comparable with GCC's performance,
sometimes LLVM is a bit faster, sometimes a bit slower.
LLVM is open source software.
It has assembled far less crust than GCC and is thus much more maintainable.
It is still in major development, far from being complete;
the project has signaled that it would welcome contributions from our side, based on the expertise we have collected with C++ and interpreters.
The LLVM project has been adopted by Apple; they are the current main sponsors, integrating it fundamentally into their operating system.
Apple gives the impression that it is interested in boosting LLVM's capabilities, in spreading its use, and in making it a solid, well maintained product.
LLVM can currently be built on Linux, Windows, and MacOS; the developers want to add support for more ABIs.
It currently uses the GCC C++ parser by default; a separate, far more performant parser (clang) is being worked on and is expected to reach a usable quality in a few months.
Even with the GCC C++ parser, the LLVM-compiled ROOT is faster than the GCC-compiled version.
The unique feature of LLVM is the clear separation of parser, language dependent code representation, optimization, intermediary (language independent) code representation, compilation, and generation of binaries.
This makes it relatively simple to implement e.g. a C++ interpreter; the LLVM team has provided a proof of concept that they determine to be unusable due to a severe lack of performance, mostly because nobody has taken over the project.
It will also allow us to extract the type information parsed by LLVM and store it for ROOT's use cases.
We see two major fields of work before LLVM can be adopted as a full replacement of CINT, GCCXML, and genreflex (though we would keep Reflex, ROOT's type information database).
Given our experience with CINT and Reflex we expect these projects to be realistic.
CINT and Reflex have more than 350k lines of C-style code whereas LLVM currently has about 130k well structured, object oriented lines of code.
LLVM's complexity should thus be manageable for us.
The interpreter.
We know that it is possible to implement an interpreter based on LLVM.
We expect its performance to be comparable to CINT's.
LLVM's compilation back-end could provide us with a significant improvement:
the interpreter could automatically compile (parts of) code that it has determined as performance critical,
which would bring just-in-time compilation (JIT) to C++.
The majority of work would need to go into interfacing libraries (for which type information exists) with the interpreter, such that compiled functions can be called --- a prerequisite for the use in ROOT.
Once this is implemented, JIT should be enabled to allow for a far superior performance of code interpretation.
Extraction of Reflection Information.
To replace both CINT and Reflex, LLVM must be able to provide information on the types it has encountered while parsing.
This use case of LLVM has not been foreseen by the developers;
we believe, though, that LLVM's clean design, its separation of parsing and compilation, and the vivid development team will allow us to implement this feature.
It would speed up the build time, especially when using LLVM to also compile code, something we will not require.
This part consists of interfacing LLVM with Reflex, making the LLVM-Reflex based type information available to ROOT and ensuring that ROOT's I/O remains as reliable as it is now.