Standardizing ways of understanding which bugs development teams are fixing and which ones they aren't

20120330 KS:
http://status.qa.ubuntu.com/reports/kernel-bugs/reports/rls-p-tracking-bugs.html
is where bugs that the development team is committing to fix are tracked. bugs go onto this report, and are separated out into teams by the addition of the "rls-p-tracking" tag being added to the bug.

Since Precise is an LTS, and there will be new hardware features, etc. being added to it, as well as support concerns, keeping this list active in q will be needed.
Proposal is for Q development to have "rls-q-tracking" used to indicate development team commitment to fixing bugs.

When a team has reviewed a high or critical bug, and its clear the development team isn't going to bee able to commit to fixing it, would like to have a convention to convey this information that we can all agree on. Something like 'rls-q-nottracking' (which is
to be used sort of like the 'bot-stop-nagging' tag ;) ), so the stakeholders know the development team has reviewed the bug, and will not be committing to fixing it.
This way the priority and importance don't get mitigated, but what's being done and not done by a development team is clear. Tag will also be useful for bugs with multiple tasks, when the critical tasks for one team are done (while it has the rls-q-tracking tag on), but the non critical ones against another team won't be, after the ones that the development team commits to are done, the tag can be changed to rls-q-nottracking) so it keeps the report from getting cluttered up with bugs that aren't really being worked on any more, but were by some teams historically.

The current convention to get a team to review and assess a bug is to assign the team to it. Given the wide range of bugs that the development teams have to look at each day though, being able to add a filter on which bugs the release and support teams wants them consider to prioritize for fixing would be useful as well (effectively a proposed or incoming queue) on the team assigned bugs. Agreeing on a common way of maintaining an incoming or proposed filtered set of bugs for the teams, will help with the triaging of which are the most important bugs to review and make decisions on whether it will get fixed or not by that development team (by adding the rls-q-tracking or rls-q-nottracking) tag.

20120427 cjwatson: For me, the tracking tag system worked very poorly. I didn't even fully understand the intended semantics until February or March or so. I felt that problems with the prior system of release-criticality had been attempted to be solved by applying another layer of abstraction, which was not consistently explained to leads ("your team has committed to fix this bug" isn't the same as "this bug is release-critical" in my mind) never mind developers, and whose main effect for me was that I found I was having to keep track of one extra list - there were many bugs I wanted to keep track of in one way or another which I felt were important for the release but which I wasn't prepared to absolutely commit my team to fixing. The psychological implications of the previous system were better for this.

In talking with Kate, I understand that she was trying to solve at least the following problems:

 1) Many build failure bugs were targeted/milestoned as release-critical, making the previous list unwieldy.
 2) Launchpad did not offer a way to break down the targeted/milestoned list by team (since this requires heuristic knowledge Launchpad doesn't have).

However, solving 1) by introducing a new list is clearly not the answer. After all, if people had actually understood how the tracking tags worked, they could just as easily have tagged all those build failure bugs, creating exactly the same problem! All this did was move the goalposts. The real solution here should have been to agree which build failure bugs are release-critical. Based on the experience with eliminating out-of-date binaries at the end of the 12.04 cycle, I propose that we regard build failures that cause there to be out-of-date binaries in the archive (i.e. the package built before on this architecture and now it doesn't) as release-critical, and that removing the outdated binaries if appropriate is one valid way to downgrade the bug to not be RC any more (this means that in most cases it doesn't require a significant commitment from Canonical in the case of build failures in universe). This is in line with Debian's battle-tested practice, and it should work excellently in the 12.10 cycle because we're starting out with zero out-of-date binaries and the +1 maintenance team should be able to keep this figure low.

Problem 2) has been solved by way of the team-mappings spreadsheet that roughly assigns bugs to teams based on their source package. Now, this isn't perfect, and I've expressed quibbles with the choice of implementation in the past, but it has the major virtue that it is essentially working (modulo the problems laid out in https://blueprints.launchpad.net/ubuntu/+spec/other-q-arsenal-report-enhancements). This team assignment is entirely orthogonal to the way in which release-critical bugs are identified, so there is no need to consider it further for this purpose.

Another problem with tags is that they have no access control. Now, we don't need to go overboard in worrying about this; there are many cases where they're useful anyway, either because nobody's actually interested in abusing them or because it's easy to spot and revert abuse. However, "this bug needs to be fixed for the next Ubuntu release" is a special case. Many users aggrieved by inaction on a particular bug are liable to jump on any such mechanism for identifying a bug if they know what it is, and I would conjecture that the only reason that we didn't see widespread abuse of the rls-p-tracking tag is that not even all leads understood it very well. The ability to target a bug to a release is restricted in Launchpad precisely because we use it for this kind of tracking.

I propose that we return to the combination of targetting, milestoning, and importance documented in https://wiki.ubuntu.com/RCBugTargetting, or small refinements of it as necessary, for identifying release-critical bugs. This has the advantage that development teams already understand it, it had been thrashed out over several previous UDSes, it has largely appropriate access control (and is easy enough to improve if there are problems), and, with the possible exception of build failure bugs which I addressed above, it was generally being used consistently. If people think it would be helpful, it would be fine to add a tag such as "rls-q-nottracking" to indicate that we've explicitly reviewed an RC bug and determined that it shouldn't be RC; I don't think such a tag is likely to be abused by non-developers, who don't generally have much interest in us *not* fixing bugs. Arsenal can be changed to take the results of an appropriate search as input, and it can continue to apply team mappings to that. This should solve the previous problems without introducing so many new ones.

[vorlon 20120509] I think there are several requirements here for a good workflow that eliminates the current uncertainty:
 - a consistent, cross-team way of identifying bugs that need to be fixed for the release
 - a consistent, cross-team way of identifying bugs that stakeholders want to *propose* be fixed for the release
 - a consistent, cross-team way of marking proposed bugs as triaged (turfed) as ones we won't commit to fixing
 - a way for stakeholders to nominate bugs that doesn't require them to know which team is responsible for fixing the package

I think we also need to clarify the distinction between "the team is committing to fixing this bug" and "this bug is a release blocker", and precisely which it is that we care about tracking team-wide.

From 2012-05-09 UDS session:

Requirements:
- assigning and unassigning to team is not sufficient.
- acceptance should be access controlled.
- release targetting, and nominations are access controlled.
- acceptance and reject part needs to be access controlled and audit trailed.
- ubuntu bug control can nominate bugs to a release
- ubuntu release nominators can approve nominations for a release

Incoming state - unprocess - via tag
Process accepted - must be targetted to series.
Process rejected
Need a system that will work for the flavors as well.

If a core package, it should show up under Ubuntu Engineering.
Nomination - not tied to ACL. (will use a tag) rls-q-incoming
Acceptance/Reject - things that are targetted to series are things that are going to be fixed.
FTBFS - only those that result in out of date binaries should be accepted.
Should not accept on part of other teams.
Statement we are fixing for release.

Normal process for turfing.... add a tag (rls-q-notfixing)
 - if there is a series task set it to Won't Fix
 - if its targetted to -updates or a point release milestone e.g. 12.04.1
 Bugs we plan to fix in a given milestone: targetted and milestoned
 Bugs we won't fix in the cycle: rls-q-notfixing
 Bugs we plan to fix in the release: release targetted only - is not closed state, not tagged with rls-q-notfixing.

Things targetting for SRUs. need to be made consistent. If we know we won't be able to fix by the release, target to -updates so it's a list of potential SRUs.

Work items:
[ursinha] consider rls-q-* tags when calculating gravity (+ for incoming, - for notfixing): DONE
[vorlon] write up proposal based on discussion: DONE
[vorlon] check to make sure meets jason's requirements and get manager signoff.: DONE
[brian-murray] will create reports once process signed up.: DONE
[kate.stewart] need a plan for flavors, orthogonal mechanism. - possible tag.: DONE
[kate.stewart] process pages need clean up - updated RC -bug target needs to be replaced.: DONE