BigPanDA Status Report Oct-Dec 2013
J. Schovancova (jschovan@bnl.gov)
Last update: 2013-12-16

Meeting agenda: 
https://indico.cern.ch/conferenceDisplay.py?confId=284111


Activity since end of September 2013:


### Code maintainability and documentation
* migrated code from personal SVN repository to PanDA repository
	** BigPanDA monitoring: https://svnweb.cern.ch/trac/panda/browser/panda-bigmon
	** PanDA server - MySQL branch: https://svnweb.cern.ch/trac/panda/browser/panda-server/branches/mysql
	** miscellaneous: https://svnweb.cern.ch/trac/panda/browser/bigpanda-misc
	** MySQL clone of the classical monitor: https://svnweb.cern.ch/trac/panda/browser/monitor.branches/branches/mysql
	** MySQL clone of the platform monitor: https://svnweb.cern.ch/trac/panda/browser/pandamon/branches/mysql
* documentation:
	** introduced TWiki pages
		https://twiki.cern.ch/twiki/bin/view/PanDA/BigPanDAmonitoring
		https://twiki.cern.ch/twiki/bin/view/PanDA/BigPandaHTCondorAPI
	** README files
		various files across repositories: panda-bigmon, bigpanda-misc



### EC2 instance maintenance
* pandawms.org instance
	** pandawms.org uses CAcert.org host certificate
	** obtained OSG host certificate, to be installed in January 2014
	** re-iterated with making EC2 machine more secure
* pilots1.pandawms.org instance
	** created in early December to have APF2 instance for BigPanDA



### BigPanDA server
* Tadashi merged MySQL branch into trunk
	** AFAIK this merged version is not yet running on ATLAS instance
		*** please be aware that this merged version cannot be deployed to monitor machines without changes to monitoring code!!!
		*** deployment of stuff to monitor machines should not be done with checkout of HEAD
* successfully tested LSST-like job submission by "an user" to PanDA server, using user's certificate/voms proxy authentication for submission
	** please be aware that the step "PanDA server to compute resources" is still being debugged: we need running APF2 for LSST (work in progress) and pilots on WNs calling the pandawms.org instance to execute the jobs as BigPanDA jobs!
* fetched topology information about BNL LSST resources from AGIS: http://pandawms.org/bigpandamon/schedconfig/
	** prepared infrastructure to update this information regularly. Currently no automatic update is performed.




### BigPanDA monitoring
* drafted list of BigPanDA monitoring pages elements and received comments from Torre
	https://twiki.cern.ch/twiki/bin/view/PanDA/BigPanDAmonitoring
* drafted layout template for BigPanDA monitoring http://pandawms.org/bigpandamon/
	** template blocks documented: https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon/bigpandamon/doc/template_blocks.txt
	** template forks for BigPanDA a ProdSys2 created, both forks inherit from common base template (_base_grid.html)
		*** BigPanDA base: https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon/bigpandamon/templates/_base_bigpandamon.html
		*** ProdSys2 base: https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon/bigpandamon/templates/_base_prodsys2.html
* implemented Django template tag building block to expose data in tabular form
	** using jQuery plugin dataTables for the table functionality on frontend
	** filtering by different columns (exact value, wildcard *, interval of values) implemented
	** filter is reflected in hash tag part of URL, thus filtered page is bookmarkable and enables collaborative multi-user environment



### BigPanDA - HTCondor API
* REST API with X509 authentication to add/update/remove HTCondor jobs from BigPanDA database: https://twiki.cern.ch/twiki/bin/view/PanDA/BigPandaHTCondorAPI
* HTCondor jobs view: http://pandawms.org/bigpandamon/htcondorjobs/#
	** contains modified ATLAS condor job information taken from APF machines on 22nd November 2013
* waiting for feedback from the HTCondor team



### BigPanDA APFmon
* discussed with Torre and Peter Love possible ways to collaborate in areas of monitoring (BigPanDA monitoring, HTCondor jobs monitoring, and APF monitoring), and beyond monitoring.



### BigPanDA APF2 instance
* set up 2nd EC2 machine, pilots1.pandawms.org
* with help of John Hover installed prerequisities for APF2, installed HTCondor
* successfully ran first test Condor-G job on LSST resources at BNL
* APF2 is being configured, to be continued in January 2014.



### Dissemination
* presented BigPanDA talk at HEPiX Fall 2013 Workshop, Ann Arbor, MI, 27th Oct - 1st Nov 2013