From Alexei.Klimentov@cern.ch Wed May 15 21:29:58 2013 Date: Wed, 15 May 2013 21:29:41 +0200 From: Alexei Klimentov To: Nurcan Ozturk Cc: Alexei Klimentov , Andrej Filipčič , Paul Laycock , Rob Henderson , James Catmore , Kaushik De , Wolfgang Ehrenfeld , Eric Christian Lancon , Maxim Potekhin , Alexandre Vaniachine Subject: Re: ADC TIM in Tokyo. ProdSys2 session (draft slots allocation) Nurcan, Rob,  it is crystal clear now, I added e-mail thread to the agenda, and we will use info during discussion session  Many thanks again Alexei On May 15, 2013, at 6:19 PM, Nurcan Ozturk wrote: Hi all, Thanks for comments. As far as I understand we do not expect much new development for the current system,  that would be OK for us, then the most critical one will be to have the splitJobs option in the current system by  moving it from the development version to the production version. We need it for summer conference productions;  SUSY production will be launched soon, the cache was built yesterday, also all validation productions of NTUP_COMMON  need it. TOP production had greatly benefited from this option in the spring as you know. The other (critical) one will be the front-end tool, Savannah will retire by the end of the year, we won't use Jira to replace  Savannah to receive production requests from groups. If we will use DEFT by the end of the year then the front-end tool  will need to work with DEFT so its development can start with targetting DEFT I think. It would be fine for us to use the  current tool (Rob's tool) till Savannah retires as it scaled well during peak periods. Now onto the Alexei's questions/comments. slide 2: I assume it is about list of tasks. What should be used as input ? This is about making what we call "the task submisison list files", an example at: http://nozturk.web.cern.ch/nozturk/DPDProduction/validationSamples/Top/mc12_8Te V.000100.group_MERGE_list.recon.LIST.p1500 The inputs are parameters used in this list file. Rob's slides explains how he extracts the required parameters from the  DPD Savannah ticket. I just use my own templates located in the above web server. slide 3: I think it isn't so much about monitoring, but tasks/jobs control ? Yes. For instance some buttons/links on task ids/job ids to act on them for finish/abort/kill. Wolfgang was mentioning that  something is already available for analysis users on the dashboard, I have not had a chance to check yet. We may also have a  new functionality on the dashboard; when the tasks are filtered, let's say by a p-tag, to be able to finish/abort all active tasks  selected by the filter in one go. slide 4: My uderstanding that AMI is used to create and populate group containers ? Do you want to do it 'automatically' by ProdSys2 ? Group contacts create their group containers using AMI, this works well. However we can not use the group container itself as input  in our current task definitions, we have to list each dataset in the container individually. Same for period containers, let's say when groups  request to run on AODs from period E for a given stream, we can not use the period container name directly, here is an example: http://nozturk.web.cern.ch/nozturk/DPDProduction/new-open-ended/SUSY/data12_8Te V.000800.group_MERGE_list.recon.LIST.p1328 I have not thought of creating the containers 'automatically' by ProdSys2, I think what we have in AMI is working fine currently.  As far as I remember being able to run on containers directly is already in the design of ProdSys2, will check in the TDR. slide 6 : number of times shows how many times the same input dataset was processed to produce GP output. Is it correct ? The actionn to production is to delete -n outputs automatically by obsoleting tasks, the original issue is related to the SW quality.  If I understood it correctly Yes. How many times the same input was processed to produce those distinct NTUP types. Not the new productions because of the  SW problems (if I understood your comment correctly).  Rob's slides my understanding that we should keep functionality, but probably redesign a bit a tool to have it more automated.  I am not sure about book-keeping with Savannah, but we will discuss it tomorrow We do not want to use Savannah or Jira, see my comment above. We need group contacts to start from a webtool to choose  those required parameters for task definitions. Such a tool was recently requested by the Exotics group in the PC meeting (slide 6): https://indico.cern.ch/getFile.py/access?subContId=0&contribId=3&resId=0&materi alId=slides&confId=250817 I think Rob can help with skipping the Savannah part and put together a page with drop-down menus for groups to use,  to work with the Panda task request interface as it is. However if DEFT will come into production soon, the development  of this front-end tool should start accordingly I think. Cheers, Nurcan. On May 15, 2013, at 10:00 AM, Alexandre Vaniachine wrote: Hi Andrej, I do not think this would be appropriate for the dedicated ProdSys2 session. To avoid mission creep, the requirements update for the current ProdSys are discussed at the ADC Development meetings and finalized at the forthcoming S&C Workshop. Cheers, Sasha On May 15, 2013, at 4:49 PM, Andrej Filipčič wrote: Hi, I think we can mark I and II topics tomorrow quite easily. But if you have some wild wishes which might seem impossible to implement right now, it would be worth to mentioned them. Cheers, Andrej On 15. 05. 13 09:23, Alexandre Vaniachine wrote: Hello Nurcan, Paul, Rob and James, I got confused by lack of separation between the current ProdSys and new ProdSys portions of your slides. Perhaps you may separate the slides in two parts more clearly? Cheers, Sasha