Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Removed BasicGrammar_inl.h and all macros inside it #616

Closed
wants to merge 16 commits into from

Conversation

sponce
Copy link
Contributor

@sponce sponce commented Feb 11, 2020

BEGINRELEASENOTES

  • cleanup of unnecessary (and potentially harmful) macros, mainly around the definition of Grammar
  • macros have been replaced by templated code, which is cleaner, easier to read (much smaller) and reusable (no need to call a macro for each and every new type)
  • couple of other cleanups have been done on the way, e.g. drop of unused config.h, eval_none function and parse_none functions
  • remaining macros have been renamed with prefix DD4HEP as they clash with Gaudi ones
  • note that the change is not fully backward compatible, as no "default" grammars are provided anymore, thus the user may need to instantiate them explicitly in some (rare) cases. See the last commit for an example. These cases are those where the code does not know a priori which objects will come out of e.g. a deserialization from ROOT file.
    ENDRELEASENOTES

@petricm
Copy link

petricm commented Feb 11, 2020

Before we merge, please also verify on the testing mac that it works.

@sponce
Copy link
Contributor Author

sponce commented Feb 11, 2020

I first need to make it work on linux actually ! But yes I'll check on Mac when I'm there.

@gaede gaede changed the title Removed BasicGrammar_inl.h and all macros inside it WIP: Removed BasicGrammar_inl.h and all macros inside it Feb 11, 2020
@MarkusFrankATcernch
Copy link
Contributor

Before this request is merged I suggest to discuss the implications in one of the following dd4hep developer meetings.

@sponce
Copy link
Contributor Author

sponce commented Feb 13, 2020

Second attempt at this. This time all tests pass, but the change is not fully backward compatible (see end of the release notes and last commit). I could make it backward compatible by creating all grammars in libDDCore just in case, as it was before basically, but I find this really an overkill when you need only one of of 50 and when 99% of the time you still do not have anything to do.

@sponce
Copy link
Contributor Author

sponce commented Feb 13, 2020

Not yet fine... it fails again on Mac... Good that I've checked. I'm on it

@sponce sponce force-pushed the sponce_grammacroremoval branch 3 times, most recently from bbf16d4 to 9c080ca Compare February 14, 2020 10:06
@sponce
Copy link
Contributor Author

sponce commented Feb 14, 2020

ok, now also working on Mac :-)

@petricm
Copy link

petricm commented Feb 14, 2020

you still have problems with xerces https://gitlab.cern.ch/mpetric/DD4hep/pipelines/1414233

Copy link
Contributor

@gaede gaede left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the cases where backward compatibility is broken ? Would it affect existing ILC/CLIC/FCC detectors ?

@sponce
Copy link
Contributor Author

sponce commented Feb 14, 2020

What are the cases where backward compatibility is broken ? Would it affect existing ILC/CLIC/FCC detectors ?

As explained in the release notes, no "default" grammars are provided anymore, thus the user may need to instantiate them explicitly in some cases. The commit "Reintroduce missing dictionaries in examples" is about that for the examples we have. As you can see, out of >200 examples, only 3 files were concerned.

These are those where we "load" conditions from an external source, e.g. ROOT file, without knowing a priori what will come out. In such cases, the current code already expects the needed dictionary to be "magically" present : it does not call BasicGrammar::instance<Type> as it does not know the type to instantiate, but directly get on the dictionary of grammars with the key found in the file. So this dictionary needs to be populated by someone.

In case of external types, this is done by calling some macros and it's still unchanged. But for "standard" types (to be defined actually), dictionaries were provided directly, I believe in libDDCore.so. So for double, vector and many others, you did not have to do anything.

This had 2 major drawbacks in my opinion : first that it's a lottery, e.g. you have std::map<std::string, std::vector<std::string>> provided but not std::map<std::string, std::vector<int>>, and second it means that whatever a user is actually using, she/he will pay the price of instantiating tens (if not hundreds) of grammars for nothing, and then look for them in a big map rather than a small one.

So now even "standard" types' grammars have to be explicitly instantiated, like in the mentioned commit. However, in most cases, the explicit usage of a condition of these types triggers the instantiation of the Grammar via the templating machinery. It's only when the type is never mentioned that we need an extra line of code.

I hope this was clearer !

@petricm
Copy link

petricm commented Feb 18, 2020

@sponce This still needs some fixes, it does not work with Xerces-C check here. The Travis tests don't check this, because if would run into a timeout. Please add to the cmake command -DDD4HEP_USE_XERCESC=ON on your build.

@sponce
Copy link
Contributor Author

sponce commented Feb 19, 2020

Problems with Xerces should now be fixed. It's sad that this is not checked by the CI. Thanks for spotting them @petricm

@petricm
Copy link

petricm commented Feb 19, 2020

Thank you for fixing, but on Mac with Xerces it still does not work

1749 FAILED: lib/libDDDBPlugins.dylib 
1750 : && /Library/Developer/CommandLineTools/usr/bin/c++ -ftls-model=global-dynamic -Wheader-hygiene -Wno-c++1z-extensions -Winconsistent-missing-override -fdiagnostics-color=auto -Wdeprecated -Wno-long-long -Wformat-security -Wshadow -pedantic -Wextra -Wall              -pthread -O2 -g -DNDEBUG  -dynamiclib -Wl,-headerpad_max_install_names -pthread -Wl,-undefined,error -o lib/libDDDBPlugins.dylib -install_name @rpath/libDDDBPlugins.dylib DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/CondDB2DDDB.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDB2Objects.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBAlignmentTest.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBConditionsLoader.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBDerivedCondTest.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBDetectorDumps.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBExecutor.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBFileReader.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBLogVolumeDump.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBPlugins.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DDDBvis.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DeVeloServiceTest.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DeVeloTest.cpp.o DDDB/CMakeFiles/DDDBPlugins.dir/src/plugins/DetService.cpp.o  -Wl,-rpath,/Users/gitlab-runner/builds/98de57ac/0/mpetric/DD4hep/examples/build/lib -Wl,-rpath,/Users/gitlab-runner/builds/98de57ac/0/mpetric/DD4hep/lib -Wl,-rpath,/cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib lib/libDDDB.dylib /Users/gitlab-runner/builds/98de57ac/0/mpetric/DD4hep/lib/libDDCond.1.11.dylib /Users/gitlab-runner/builds/98de57ac/0/mpetric/DD4hep/lib/libDDCore.1.11.dylib /Users/gitlab-runner/builds/98de57ac/0/mpetric/DD4hep/lib/libDDParsers.1.11.dylib /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libRint.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libTree.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libNet.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libPhysics.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libMatrix.so /cvmfs/clicdp.cern.ch/software/Xerces-C/3.1.3/x86_64-mac1014-clang100-opt/lib/libxerces-c.dylib /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libGeom.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libRIO.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libGenVector.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libMathCore.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libImt.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libThread.so /cvmfs/clicdp.cern.ch/software/ROOT/6.18.00/x86_64-mac1014-clang100-opt/lib/libCore.so && :
1751 Undefined symbols for architecture x86_64:
1752   "dd4hep::BasicGrammar const& dd4hep::BasicGrammar::instance<dd4hep::AlignmentData>()", referenced from:
1753       dd4hep::AlignmentData& dd4hep::OpaqueDataBlock::bind<dd4hep::AlignmentData>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in DDDBDerivedCondTest.cpp.o
1754 ld: symbol(s) not found for architecture x86_64
1755 clang: error: linker command failed with exit code 1 (use -v to see invocation)
1756 [119/179] Building CXX object DDDigi/CMakeFiles/DDDigiExampleLib.dir/src/DigiTestSignalProcessor.cpp.o

@sponce
Copy link
Contributor Author

sponce commented Feb 20, 2020

Should be fixed now.

@petricm
Copy link

petricm commented Feb 21, 2020

@sponce sadly we need to revert the changes from your previous PRs, there are some unforeseen changes in behavior. Could you please cherry pick those commits (#615 and #611) into this PR and we will deal with the grammar changes in one go once all is resolved.

@sponce
Copy link
Contributor Author

sponce commented Feb 21, 2020

Ok. Can you point me to the changes of behavior so that I have them in mind ?

@sponce
Copy link
Contributor Author

sponce commented Feb 21, 2020

Ah and we should also add a test that checks these changes, so that I cannot break things again.

@petricm
Copy link

petricm commented Feb 21, 2020

It's the CMS change described #628

@petricm petricm closed this Jun 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants