Improvements to the ISO tracker and some build related scripts

Registered by Stéphane Graber

Discussion of changes that would help:
  - release team with managing and publishing images, understanding
    the quality/shipability
  - qa teams (ubuntu and flavors) for understanding quality of images and issues for
    developers
  - testers ease of using system, logging results, automated logging, knowing bugs

Blueprint information

Status:
Started
Approver:
Steve Langasek
Priority:
Medium
Drafter:
Stéphane Graber
Direction:
Needs approval
Assignee:
Stéphane Graber
Definition:
Drafting
Series goal:
Accepted for quantal
Implementation:
Started
Milestone target:
milestone icon ubuntu-12.10-beta-1
Started by
Steve Langasek

Related branches

Sprints

Whiteboard

Pre-session notes:
KES 2012/03/15 - ponder points:
   - automated updating of paths on iso tracker to reflect what's coming off the builders without manual entry ( cloud-images, netboot, wubi ) based on manifest
   - aggregation of bug reports and results from iso.qa.ubuntu.com and localized-iso.qa.ubuntu.com
   - training material on how to add localized images to localized-iso.qa.ubuntu.com
   - clarification of criteria for localized image to be eligible to be added.

2012/03/23 - discussion with QA on standardizing on result input format with LAVA
   - look into adding release specific tests as ability
   - accepting in LAVA's JSON format for importing results from Checkbox, Linaro, etc.
   - dynamically adding tests into tracker, mechanisms.
   - when testcase repository/database comes on line, we need to have mechanism to work with unique ids from it. (currently testcase id is assigned by tracker)

RS/KES 2012/04/25 - look into providing better visual clues as to the classification of bug states on the iso tracker (untriaged bugs, blocking bugs, unblocking bugs) ** (note this needs to tie into mechanism agreed to in discussion Steve's leading in:
https://blueprints.launchpad.net/ubuntu/+spec/other-q-release-bug-list-workflows

Random items taken from other blueprints and feedback:
 - Need to get the subscription page back online
 - Add a Ready status to the build record, that still lets new results be added but prevent new builds from being pushed automatically and can be used to generate the list of images to release.
 - Generate some kind of overview for hardware profiles (once the API integration with LP/ubuntu-friendly is done)
 - Improve subscription report to better deal with long lists of subscribers
 - Support time range in top testers report

Pad: http://summit.ubuntu.com/uds-q/meeting/20234/foundations-q-iso-tracker/
Session notes:
Testcase management parts moved to another session.
 http://summit.ubuntu.com/uds-q/meeting/20462/qa-q-isotracker-testcases/

 Feature Requests:
  - good data for daily testing (getting data over a range of dates)
  - Bugs -> filtering bugs by new, triaged, release critical. Showing it up on tracker.
    Possibly 2 columns, one for untriaged, one for release critical/non critical.
  - Add a checkbox - Ready for Release.
 - Filter on a hardware profile? reports based on hardware profile?
   Use case: during beta 2 has someone with an ATI graphics card.
   Ubuntu Friendly will be data source of hardware profiles.

 - Do we want to put a per user view? what they've done, etc. Want aggregates. --> WISHLIST (code contributions welcome).
   - aggregation of bug reports and results from iso.qa.ubuntu.com and localized-iso.qa.ubuntu.com
   - clarification of criteria for localized image to be eligible to be added. - TBD - chinese, italian, ...
     --> discussion of images.

2012/03/23 - discussion with QA on standardizing on result input format with LAVA
   - look into adding release specific tests as ability
   - accepting in LAVA's JSON format for importing results from Checkbox, Linaro, etc.
   - dynamically adding tests into tracker, mechanisms.
   - when testcase repository/database comes on line, we need to have mechanism to work with unique ids from it. (currently testcase id is assigned by tracker)
Random items taken from other blueprints and feedback:
 - Need to get the subscription page back online
 - Generate some kind of overview for hardware profiles (once the API integration with LP/ubuntu-friendly is done)
 - Improve subscription report to better deal with long lists of subscribers
 - Support time range in top testers report

Multiple bug columns?
 e.g.http://iso.qa.ubuntu.com/qatracker/milestones/204/builds
Fold this into test case scrub:
  - Automated test cases, installer testing, --> add clarification?
  - Mark test case as manual or automated.
  - Rename to be more meaningful: upgrade --> Upgrade, no user interaction.

 Wishlist: input a package name, and all affected images automatically marked for rebuild.

[vorlon, 2012-06-01] please find specific people, not teams, to be responsible for the work items; team WIs don't show up anywhere useful on status.u.c

(?)

Work Items

Work items:
[stgraber] bug columns - extract info from launchpad, show untriaged in one columns, and release critical/not in another: POSTPONED
[stgraber] add release ready status to build record: DONE
[stgraber] figure out how to add hardware profile to Ubuntu Friendly, and finishing hardware integation (waiting to know what tool will be used): POSTPONED
[darkmuggle-deactivatedaccount] incorporate script so that cloud images automatically publish: POSTPONED
[ubuntu-release] what are criteria for image being added to localized tester: POSTPONED
[gema] provide input examples for LAVA format, for stgraber to know what the tracker should be reading: POSTPONED
[registry] testcases need to be denoted as manual or automated, and naming needs to be cleaned up: DONE
[ubuntu-cdimage] automatically post a diff between builds to notice boards on the tracker, linking to log of what changed for history. Superseded images have differences: POSTPONED
[ubuntu-release] decide on what we put out, between builds, day, since milestone, between milestone: POSTPONED
[stgraber] netboot changelogs put out to notice boards as example: DONE
[stgraber] Allow for filtering by time range for /history and the top testers report (possibly others report too): DONE
[timrchavez] separate instance for tracking Professional service. Look into build infrastructure automatically feeding in: POSTPONED
[stgraber] Implement subscriptions page: DONE
[stgraber] Improve subscription report to better deal with long lists of subscribers: DONE

Dependency tree

* Blueprints in grey have been implemented.