Backpressure

Backpressure for Retrieval

This is the simplest case and has the least serious consequences (i.e. no possibility of data loss).

The Tape Server asks if there is enough free space on the destination disk system, and "reserves" enough space for the files which have been popped from the queue. If there is insufficient space, the tape is dismounted (unless this is a new mount: the mount will not happen) and the retrieve requests are requeued.

Backpressure for Archival

If we are writing to the disk buffer faster than we can archive to tape, the buffer will gradually fill up. Once it is full, there will be a "no free space" error which will be reported to the client (SFO, FTS, etc.) It is up to the client to hold on to the disk files and retry when there is buffer available again.

If everything is working well, we can keep up, e.g. in recent tests writing 5.6 GB/s to 17 tape drives used only about 10 TB of buffer.

The required amount of disk buffer (for long continuous periods of data taking/to survive outages of tape infrastructure) will be set by the Service Level Agreement with the experiment.

Backpressure for Simultaneous Archival and Retrieval

Julien's solution worked out with Andreas:

Have 2 EOS spaces overlaid on the same physical disk, one for archival and one for retrieval:

We can set independent quotas for each space. The quota is set as an absolute number of bytes per space (not a percentage).

For the archival use case, we should set a quota (e.g. 30% of available space). When the quota is reached, the client will receive write failure errors and will back off.

For the retrieval use case, we do not need to set a quota, because when the Tape Server makes its reservation, it uses df to query the available space and then checks if it has enough space to make a reservation. If we can't make the reservation we will never write, so in any case we should never hit a quota. Instead we configure the "free disk space threshold" for the reservation to, e.g. 30% of available space.

Conclusion

This does not solve the case where there are two channels writing (e.g. DAQ + reprocessing) and the risk of DAQ being blocked by another write process filling the disk. In general we expect such a case to be rare and to be solved by tape operators. If we want an automated protection, we could implement priority channels (e.g. by storage class).

Special considerations for ALICE

Post-meeting update: the problem is not solved

The proposed solution depended on the fact that the quota/available disk space for the default and staging spaces would be accounted for separately. However, this is not the case. Whether the disk system is queried for space A or space B, it will return the same number (the space available on the underlying physical SSDs).

This means that if the archival quota is set to 30% and retrievals have filled up 30% of the disk, then both the archival and retrieval backpressure mechanisms will block further writes.