ISO/QA testing tracker improvements for Precise
The QA tracker at http://
It's in desperate need of some small changes to better work with the amount of testing Ubuntu requires nowadays.
This session is meant to discuss what are the most important changes we need to make the tracker work better for the LTS.
These changes should ideally be implementable very quickly as the tracker starts being used for the first alpha.
Blueprint information
- Status:
- Started
- Approver:
- Steve Langasek
- Priority:
- High
- Drafter:
- Stéphane Graber
- Direction:
- Approved
- Assignee:
- Stéphane Graber
- Definition:
- Approved
- Series goal:
- Accepted for precise
- Implementation:
-
Beta Available
- Milestone target:
- None
- Started by
- Colin Watson
- Completed by
Whiteboard
== Notes from the session ==
Fixing https:/
Background:
- rework the http://
- will be made available in 12.04 cycle for tracking.
New Features
- Drupal 7
- Database accesses dynamiclly (build set)
- Launchpad Authentication (Admin: QA and release team, Launchpad: testing)
- Daily build notifcation
- Jenkins can pull list of testing from there, and push back to ISO tracker now.
- Archive view of previous ISO testing
- Easier administration interface
- Access to historical testing and access to prior releases
- Every string in new version is suitable for internationaliz
(should be possible to script doing this, useful for laptop community)
Wishlist of Reports: (For consideration to add as time and contributions permit).
Would it be possible to have view of results per user?
- Possible, need to know how sorted.
Would be really nice to have a way to pull up previous test results after image respin. Or ultimately move bug reports forward when a respin doesn't apply to them.
Linking multiple results to hardware testing? new table? link to ubuntu friendly? ???
- Should be fairly straightforward; let people indicate what hardware working on via systemID.
Testcase coverage needs to be reworked to be clearer.
- Depend on number of products shipping the test case (number of folks who've run the test cases is significant indicator of confidence in results).
Notifications being reworked.
Do people want to know when first build?
Fail for build?
Build is fully tested?
Subscribe to dailies and know when one failed?
Would like an opt in subscription per user interface?
Difference between dailies and milestone testing.
Note: Color scheme is good - Charlie able to see. :)
API:
- adding builds and adding results.
- do we want to remove? if computer crashes - how handle this.
- allow users to remove own results for active only.
- easier to customize flags - iso-testing getting added by cron jobs.
May need to make sure not flooding if multiple.
Improve Launchpad integration so more useful information added per bug. (which test were people running when they found this bug? which image were they using?)
Do we still want to use clarifying serious bugs:
https:/
armel vs. x86... discrepancy - serious for image vs. launchpad seriousness.
(how indicate not serious if wrong - admins can do this).
Bugs that don't affect x86 should not be critical was historical. Not accurate.
(scope of impact should be governing it).
Test instance is available at: http://
== Actions ==
Work items for precise-alpha-1:
[kate.stewart] verify with IS that drupal7 can be deployed: DONE
[primes2h] provide list of specialization done on prior code base, and review stgraber's port.
[stgraber] Add cross linking to ubuntu friendly machines for test runs: DONE
[stgraber] Make sure the Drupal 7 version offers the same features as the current one: DONE
[stgraber] Add comment (changelog of package) to notification: DONE
[stgraber] Implement XML-RPC API to add builds: DONE
[stgraber] Read-only access to archived/released milestones: DONE
[stgraber] Think about the right way to offer download links for ISO products in a generic way (needs to support multiple files, multiple locations and checksum): DONE
Work items for precise-alpha-2:
[stgraber] Fine grained notification system (new, respin, removed, pass result, fail result): POSTPONED
[stgraber] arrange code review by IS for the Drupal 7 version of the tracker: DONE
[stgraber] Rework the result page, allow people to push multiple results and edit/remove their own: DONE
[stgraber] Put a link to the Launchpad profile for each result: DONE
[stgraber] Document and clarify what it means when serious is clicked. serious -> blocker.: DONE
[stgraber] Document XML-RPC API and provide python examples: DONE
[stgraber] Implement XML-RPC API to add results and query milestones, products, builds, testcases and results: DONE
[stgraber] Allow filtering of the build list: DONE
Work items:
[jibel] get hardware being used for testing added to ubuntu friendly : POSTPONED
[xdatap1] discuss with primes2h a way of making it more social; come up with proposals.: POSTPONED
[brian-murray] improve launchpad integration of the iso qa tracker website (make sure options are in qatracker_
[stgraber] Get an API to query hardware from Ubuntu Friendly (LP: #899361) : POSTPONED
[stgraber] Log all admin actions in Drupal's event log: DONE
[stgraber] Report for results per user: DONE
[stgraber] Re-implement testcase coverage report: DONE
[stgraber] Re-implement subscription overview report: DONE
Work Items
Dependency tree
* Blueprints in grey have been implemented.
