Choose timezone
Your profile timezone:
SA2.3 Metrics XML formatting for SA2.4 tools
→
Europe/Zurich
CERN
CERN
Description
A detailed discussion of the XML format required to describe the SA2.3 metrics that will be available in SA2.4.
EVO Phone Bridge
ID: 2297839
Password: 6558
Central European Summer Time (+0200)
Start 2010-09-15 14:30
End 2010-09-15 16:00
EVO Phone Bridge Telephone Numbers:
http://evo.caltech.edu/evoGate/telephone.jsp
The recorded minutes will be destroyed by the end of the week.
Minutes of Meeting
------------------
Participants: Eamonn, Fabio, Owen, Antje, Michele, Jan,
Andres, Andreas, Alberto, Jozef
Eamonn: PriorityBug can be calculated at the granularity of presenting many (x,y) values for open (start time) and closed (end time) for a bug. This then can be used to generate the average time to close an immediate priority bug.
Owen: dCache in the first instance would not be able to produce reports with fine granularity for the PriorityBug metric. The average value could be calculated. There is no tool to do this at the moment.
Owen: We are very cautious to give you out raw data that may be confidential.
Alberto: We do not ask you to export the bugs themselves. The information can be processed first.
Eamonn: looking at the first metric: PriorityBug, one of the more complicated metrics. If we sort this one out, we will sort out most metrics.
Eamonn: first question, what will we have as a common description for all metrics in the XML schema? Will we state in the XML schema the Metric Name and Metric ID?
Alberto: Fine for us.
Eamonn: My point is, what we are going to define in the schema, the metric name and id or both or just one of them?
Alberto: We should just choose. We want to see if the metrics are measurable and extensible.
Owen: We need the schema as soon as possible to determine how to produce the metrics.
Eamonn: We don't need to discuss the metrics again, we've discussed them at length. For the first metrics, we need the start time, end time, an average value or a set of (x,y) pairs to calculate the average (as an alternative). Do people agree with this?
Andres: we need the unit for the beginning and end, hours or days?
Eamonn: The metric is specified in hours. The unit is well defined already in most metrics. So you will have a real number, rational number or integer number with a unit which is usually time or unitless.
Eamonn: only exception to this is a 2d histogram which will be used to describe metrics over the history of the whole project.
Eamonn: Metric ID or Name are very important. The unit is also required, since its common to all metrics.
Alberto: We need the time defined for all metrics in the same units. We can not have days for everything. Some values are small.
Eamonn: Important for PriorityBug but not for all bugs. Some queries must be performed in terms of dates not hours. In this case the resultant value is in hours, but the query is not in hours, but in days.
Eamonn: We need the metric ID, the unit of the result, the start date, the end date and they must specify that they are requesting an immediate priority bug, and then they will produce a result or a set of bugs with the start and end time in the range specified.
Owen: Okay.
Eamonn: How in the schema would you produce dates and times in a separate manner? It seems that the schema does allow for it.
Andres: The date is in dd/mm/yyyy and time in HH:MM:SS.
Owen: So there is a query that is made. We are using a query XML request?
Eamonn: you point to an important point, do we produce an XML query that returns a result, or do we all results to be calculated?
Owen: it might be better to focus on the results.
Eamonn: yes, we can produce results for the moment, but we can define in the metrics document in more detail what is required to perform the query. So we can have two separate cases, what is defined to make a query to obtain the metric vaules and what is the result format to be obtained.
Eamonn: For the second metrics, The BugSeverityDistribution is slightly in error. We actually want the number of bugs still open by a certain date. So the end time is not actually specified in the document.
Owen: We could do the query once a month or once a week?
Eamonn: We could combine the information together per month, which is then combinable to produce a histogram with granularity along the x-axis, defined per month.
Alberto: It would be better if we could produce these values per night.
Eamonn: If we are producing this information per night, then the mechanism must be fully automatable within ETICS, because the overhead to the product teams is too great.
Alberto: Information will be pulled into ETICS.
Owen: How do we do this, if we don't have the mechanism for independent queries a the moment. I agree, its a nice thing to have. We need a common denominator at the moment. We can produce values periodically, monthly or weekly.
Eamonn: Owen, do you produce values by hand in the first instance?
Owen: unfortunately yes.
Alberto: Raw data can change every day, or per month, so the information can still be processed every day.
Owen: We've to minimise this work so we can get on with bug fixing. 1.5FTEs is not a lot of effort.
Eamonn: how useful is it to have information per day. Very important for priority bugs but not so important for the BugSeverityDistribution or BackLog metric.
Eamonn: BugSeverityDistribution per week could be very useful to the EMT meeting per week on Monday afternoon. However, BackLog is not reasonable to produce per day because of inconsistencies due to time lost around holiday times and period of large push through.
Eamonn: We don't want to overtax the developers, however some metrics must be produced on a daily basis.
Eamonn: Weekly or monthly many be all that can be produced for certain groups of metrics. End time date is important for BugSeverityDistribution. We will need to describe them as a discrete set.
Eamonn: BackLog: start time being previous release date, end time being current release date. Number of bugs closed in that time frame and number of bugs opened in that time frame. The result is a real valued, unitless number.
Eamonn: SuccessfulBuilds metrics: what branch will this be calculated on? What branch is to be used to produce EMI-0?
All: (no reply, no feedback)
Eamonn: do we produce the successful builds as part of the metrics section and in the logs.
Alberto: its in the report.
Eamonn: it should be in the metrics section too, defined as a rational number and a precentage. Also, would be useful to produce the number of successes/failures for a subsystem. Example: VOMS building to 100% but runtime dependencies fail, so you obtain a quick overview of what is succeeding for a PT but failed for another PT.
Owen: Will this be produced on the EMT releases only?
Eamonn: would be good to be constantly producing this information.
Owen: With every commit? Can we have our private builds that don't have metrics?
Andres: Just produced on the commited release.
Eamonn: Detection area is not defined in many middlewares to define CertificationTestEffectiveness. What is found by developers and by none developers would be an alternative.
Owen: We could produce something found in a code review.
Eamonn: Bugs found at a code review would be equivalent to bugs found by developers at the time of integration.
Eamonn: We may need an area of the XML schema to define a string that is useful for describing the difference in the interpretation of the metric collection per middleware. We need to be able to distinguish what the difference is.
Andres: We can put a note stating when the metrics will be obtained, and why they might differ.
Eamonn: Next is UpdateDoc. A repository of documents at the time of the release. Do we need a tool that produces the XML description for each of the product teams to calculate the total automatically?
Owen: what are you measuring and whether its valid?
Andres: The documents could be in CVS and then be valid if in there?
Owen: unsure what we are measuring.
Eamonn: we want to take in to account whether the user document and technical documents are defined at the time of release.
Owen: is there a man page for every command argument? It seems rather difficult to measure. Clarity isn't easy to quantify.
Eamonn: The point is, we want to see that the timely release of the documentation is provided and what is the delay. The review of the documentation is a completely separate issue that requires manual reviews and dedicated manpower. Over time you will see whether documentation is late in arriving from different partners.
Eamonn: DelayOnRelease we can skip for the moment. Static Code Analysers are much more important to look at. The majority of result values will be real valued. Some values will be normalised or percentages.
Eamonn: Owen, you would have experience of this with using so much Java in dCache.
Owen: problems with Emma, producing values. Fine if using junit to produce TestCoverage on its own. Only partial information available.
Eamonn: Have people considered combining unit tests exclusively of other static code analysers? So, you would define FindBugs with its own separate report, and the TestCoverage could be defined based on unit testing only.
Owen: we tried combining functional test suite with unit testing in Emma, if I understand correctly? Emma didn't know about functional tests, but does know about junit tests.
Alberto: We can use Emma in ETICS.
Eamonn: Test Coverage in the case of UNICORE will perform very well. Some groups will improve over time where they will score badly at the beginning.
Alberto: should TestCoverage actually be defined as UnitTestCoverage. Functional tests are also important.
Eamonn: My understanding is that this is primarily based on unit testing.
Owen: The TestCoverage is very ambitious including functional tests.
Eamonn: Need to clarify this with Gianni and e-mail everyone with a clarification on this point. Need to be sure that we can't easily go beyond this.
Owen: auto deployment of 5 services using ETICS will be very difficult.
Eamonn: you will see big discrepancies in Test Coverage per language because Eclipse is built in such a way that you generate the framework for each unit test as you generate each class function. C will probably score low, and C++ will score a little higher because of cppunit.
Owen: That's a good point, I don't any good framework for C. There is good support for Python.
Eamonn: SupportedPlatforms will skip.
Eamonn: Cyclomatic complexity will have an integer value generated automatically in ETICS. There is a python and Java calculation. Will there be a C++ one?
Alberto: C++ is available.
Eamonn: Static code analysers, Perl::Critic is not in ETICS at the moment, but there is a very large overlap between what we have in the metrics report. Has anyone used Sonar before. No, okay.
Eamonn: The large variety already available in ETICS may be sufficient. I'll do an analysis of Sonar and Perl::Critic to ensure that they are correct.
Eamonn: The result values should be real values, not integer as specified in the document.
Alberto: We start with the metrics already in ETICS and then best to leave you to investigate Perl and Sonar. The list is summaried in the documents.
Eamonn: I'll modify the list on page 29 and make sure there is a better match.
Eamonn: Between myself and Andres we can describe the XML structure. There will be a bit of liaising back and forth.
Andres: The metric schema definition is sufficient to describe to 100% all the metrics that are defined by SA2.3. So we will use the current schema in ETICS. If we need to add something, we can add it later.
Eamonn: do you have something akin to producing a barchart of histogram 2-d plot? Does the schema accommodate this.
Alberto: we can produce histograms.
Eamonn: I think we need a few examples of output data that each metric can produce. We will produce XML that will then be pulled into ETICS. That will then serve as an example for other middlewares.
Alberto: Okay I agree.
Eamonn: okay what will be the next step? I think that Gianni, Andres, Lorenzo and myself could produce a few examples for the other middlewares. We should reduce the number of people in the discussion for a while.
There are minutes attached to this event.
Show them.