GE: Is the observed linear increase of the writing time caused by the increased number of servers?
PB: The requirement for consistency leads to penalty in writing.
VB: We cannot run 1250 servers, ona at each EPN.
PB: etcd looks quite similar to Zookeeper. I also think we should compare with consul (see https://www.consul.io/).
GE: We also should benchmark the write mode.
PH: What is the use case for etcd? Is it complementary to DDS and parameter manager?
VB: We need consistency between the values at each node for things like run configuration, etc. During Run1/Run2 this functionality is provided by ECS. Zookeeper is more complex and has Java interface. etcd is easy to use and has Go interface.
MA-T: Where do you use these parameters? What is the boarder between the ALFA parameter manager and the etcd?
VB: If consistency is needed, we think etcd is more suitable. In case of big blobs of information (i.e. calibration objects) where we always need the latest value, the ALFA parameter manager is the right choice.
MA-T: RIAK should provide also consistency.
VB: This is not what I understood from the discussion about ALFA parameter manager and RIAK as backend.
PB: Consistency between the few replicas of the conditions DB will be provided by RIAK, but here we need consistency between many thousands of instances. We should distinguish between the startup process and the parameters each process uses. The values of the startup parameters do not change often and they are not write intensive. An example of startup parameter is the location of the ALFA parameter manager. In the ALFA parameter manager we keep the calibration and we always write, no replacement of existing information.
VB: etcd is a kind of state parameter manager similar to what we have now in ECS and DCS. It might play role also in DDS, for example if we want to change the log level without stopping the processes.
PB: We have to use the REST API since there is no Go support. May be it is better to stick to DDS. How much time takes the key-value propagation in DDS?
AL: The DDS key-value code is used only to propagate the parameters needed to start each process. We foresee versioning to cover the updates in the topology.
PB: Lets consider the following use case: we have etcd installed. I update the topology and propagate it using etcd. Then no key-value code development is needed in DDS.
AL: You can use external key-value DB in DDS.
PB: Probably the requirement for standalone key-value in DDS is not needed.
AL: For the moment DDS has only very simple key-value.
Slide 9: one single get in all cases.
Slide 11: CPU spikes during the initial HTTP get. The time is spent to decode the HTTP request and send the answer. The investigation is ongoing, the effect might be caused by the latency.
Segmented access: many single gets.
Tree access: one single recursive get.
VB: I will prepare JIRA ticket (https://alice.its.cern.ch/jira/browse/OCONF-20), please update it with your comments and suggestions. Here is the preliminary list:
Pascal wrote a C++ library that we plan to put in O2.
MA-T: since we use message queue, it is better to prepare an API similar to the ALFA parameter manager.
PB: In order to use message queue you need something to bootstrap the process. Can this be DDS?
PVV: You can use etcd for bootstrap.
GE: How do you discover etcd? Now DDS gives the URL from the topology file.
MA-T: How does the process communicate? It is a HTTP request.
MA-T: Message queue is much better that HTTP request.
PB: Do we want to support multiple protocols? I see the advantage of the message queue approach: scalability, pool of devices, etcd as proxy.
GE: You can guarantee the consistency, but not during the life time => locking is needed. With message queue you can push the new value to all clients. With get request you cannot guarantee the consistency. Locking will kill the performance.
VB: Locking prevents writing during get.
PB: With message queue you do not need locking, but only to make sure the success of the push.
MA-T: We should implement etcd device for the message queue.
Action item: Update the JIRA ticket (https://alice.its.cern.ch/jira/browse/OCONF-20) with all the questions and ideas.
Follow-up from Vacso:
Dear all,
As agreed in today's CWG13 meeting, I've created a JIRA issue for discussing the different needs/use cases in terms of Configuration and the potential overlap of different tools.
Feel free to add any information you find relevant.
I've also created the 2 extra tests to etcd proposed by Predrag:
Benchmark etcd write performance
Identify limiting factor of server performance
The full list of done/foreseen etcd tests is available here.
Consul evaluation is here, but there are not many details for the time being (feel free to propose).
Cheers,
Vasco
PB: Would you give more details about the speed-up? It is calculated with respect to a statically partitioned cluster. In O2 one use case is the synchronous vs asynchronous processing. It also can serve use cases not foreseen now, i.e. production and development at the same cluster, release validation.
PVV: How do you express the sharing? We use the C++ API, instantiate object and accept offer. The frameworks are statically configured. Unlike the traditional schedulers we have micro-service scheduler. You program the framework, it talks to Mesos and accepts the offers.
PVV: How does Mesos decide what are the resources?
PH: We need more "pedagogical" presentation about Mesos. Such presentation together with a demo will be done for CWG10. We will show also some Mesos-based solutions.
PB: Can more than one framework work with DDS? Yes, for example simulation, reconstruction, development, synchronous and asynchronous frameworks can coexist as Mesos frameworks.
GE: The use cases for the moment are cluster with Mesos, Slurm, or no resource management.
MA-T: All these cases are supported. If you need sharing - use Mesos, if not - rely on pure DDS. There is no scheduler in DDS. The DDS plugin submits request to the DDS REST Mesos framework.
PB: Do you need another DDS server if you want to run another topology?
AL: DDS topology update is foreseen. We have to complete the versioning and then we can use it for the topology update.
Action items: