Header - payload relation is maintained by the transport framework.
MA-T: Sending data to two devices is not general case.
PB: Isn't read-only access trivial? No, we have to consider the deallocation at the end of processing. The reading device does not change the data, but the memory management is still difficult.
Persistent shared buffers have to be feature of the framework. We have to consider two important requirements: serialisation and compression.
The buffer should be able to accomodate data with unknown size.
PB: 20 years ago we had the software for the memory management and allocation/deallocation of shared buffers. We called them "memory sections". The sections contained (according to their numbers):
The system worked nicely providing client - server model with separation of data and algorithms. We also had to make a choice between network sockets vs shared memory "camps". The net sockets were selected for the first implementation, but the attempt failed. Finally the successful implementation is based on shared memory. The software still exists.
MK: We have the same in HLT. Our approach combines net and shared memory in similar way to what Predrag described. The message is that in general the approach is good, but the memory management and the shared buffers are difficult to implement.
DR: The interface is important. If you replace shared memory by normal malloc, we do not use the features and still should be able to run. Shared memory is needed only for interprocess communication.
AR: Do you have experience with Boost in addition to SysV and Posix? No, we have no experience with Boost.
Slide 3: The optional sending depends on the header. FairMQ provides different channels and to introduce them at runtime will be difficult. Having multichannel implementation will request for each device to read all the data since the channels are not synchronised. If you do not do event building, we may need additional tricks. For example, if we have to merge data, we need merging device, where merging is equivalent to synchronisation.
Slide 13: Alexey experimented and device 2 is slower than device 3. It gets messages about deallocation. Device 2 knows how many subscribers are active and the buffer is kept. ZeroMQ version > 1.2 has this feature, so the complication is to count the subscribers. The timeout is also difficult to handle if the process crashes.
MA-T: ZeroMQ concept is broken by this custom management.
DR: Not necessary, the memory management is done by an outside package.
TK: The key concept is to push the data an transfer the ownership. Do we need "central memory management"?
MK: We can postpone the technical discussion about the memory management, what is important is to have such possibility in the interface.
TK: The interface is imposing design choices such as reference counting, etc.
MA-T: We have to underline that the meaning of the terms "multipart" and "channel" is different in ZeroMQ. A message is typically multipart message..
PVV: In our TDR the data frames do not have header.
AW: If we have multichannel, do we have to use multipart message? We do not have to distinguish or specify.
DR: What do we do for the automatic serialisation/deserialisation? This is part of the data protocol.
AR: How do we tell the serialisation library where to put the result (i.e.in the shared memory)? Nobody provides this feature. Probably we need our own allocator. We should pay attention to the max. size, multiple channels and merging devices. Concerning the serialisation, we can use example 3.
MA-T: The merging devices are in fact user devices.
AS: I think the performance benefit from the shared memory is largely overestimated. I will make a talk where I show the message passing is an alternative.
DR: The time frames are big and the copy is slow. The interface should impose "no copy".
Action item: Nicolas will make a presentation on the serialisation.
AW: Can we use more than one core for a device? No,
DR: MPI over IP can be compared directly with FairMQ. The numbers are very similar to what we get from IP over IB.
Can we measure EDR? Omni is very similar. The difference is only in some low level latency (Omni vs IB). In the tests with 4Gb messages we get the full bandwidth on a single core.
IntelMPI is used in the tests, do we have results with OpenMPI? I comparison between IP, Omni, EDR was done last Monday, the link will be sent. We plan a test where each node will be both EPN and FLP.
SCIF is transport protocol of Intel over PCE. It provides RDMA capabilities.
AR: Did you work on the socket queue? No.
PB: Where are the modifications you did? In ZeroMQ.
MA-T: Did you write any application to use ZeroMQ or a new protocol? I followed all the steps present in the other extensions.
MK: It would be very welcome to have access to the code, examples and documentation.
Action item: hands on session in April since the code will be ready end of March.