Data Knowledge Catalog Meeting (TPU/NRC KI)

Thursday 18 Jun 2020, 12:00 → 13:20 Europe/Moscow

Other Institutes

Marina Golosova (National Research Centre Kurchatov Institute (RU))

Vidyo room 'DKB'

GitHub: PRs

Trello

DKB@ATLAS instance

Current status: everything works as expected, no new issues.

Integration w/ BigPanDA Mon

1.  Development status: 75% done (0% in production)
   • all requried metadata are added to the `data4es` process;
   • `data4es` process is ready to write progress documents to a separate index;
   • REST API can provide progress summary for a campaign.
2. ToDo:
   • (mgolosova) make `data4es` produce docs with task progress;
   • (viktor(?)) update `data4es`@aiatlas171:
     • (all) review and merge ##176,368,365,369,371,366 (+1) to `master` AND/OR
     • (viktor(?)) merge not-merged-to-master to some `data4es-merge`;
     • (viktor(?)) update ES mapping for `task` docs and load one for the progress data with result mapping;
   • (mgolosova) update REST API server @aiatlas172 (test and merge #360);
   • (mgolosova) update implementation of REST API method campaign/stat (used by BigPanDA).
3. Disucssed:
   • "service" dataset (w/ data formats TXT, LOG) can be ignored in the context of AMI_tags+format-based step names.

DKB installation/development manual

1. Deadline: 30-06-202
2. ToDo:
   • (mgolosova) read, comment, define desired shape, ... (TODAY);
   • (viktor(?), vaulov, mgolosova) write/update sections according to the comments (where possible).

Batch processing

1. Discussed: when/how clear stage output buffer in case of processing errors?
   • "pre-buffer":
     • all output messages, related to a single input messages, are passed to stage.output() all at once;
     • in case of error -- none of them gets to the stage output buffer;
     • messages from the buffer are not removed;
   • flush buffer whenever a single input message is fully processed:
     • at any moment it contains output messages related to a single input message;
     • clean buffer in case of error (just as in case of a single message processing);
     • (+) next stage will get input and can start operating earlier;
     • (Q) does stage generate EOP every time the buffer content is sent to the output?

ex-ICCS paper

1. (mgolosova) Write extended abstract for World Scientific Publisher.
2. (mgolosova) Send abstract and authors list to Alexei Klimentov (THIS WEEK).
3. (mgolosova) See what comments from rea can be addressed soon enough.

12:00 → 12:05 DKB services: status report & news

Current status of the DKB ATLAS instance.

Speaker: Marina Golosova (National Research Centre Kurchatov Institute (RU))

status report

12:10 → 12:30 Other business

• Pull Requests;
• 1-slide reports for current tasks;
• Paperwork;
• ToDo's;
• AOB

Speakers: Anastasiia Kaida (National Research Centre Kurchatov Institute (RU)), Marina Golosova (National Research Centre Kurchatov Institute (RU)), Vasilii Aulov (National Research Centre Kurchatov Institute (RU))

misc