Automated ubiquity testing with autopilot
Our goal for this cycle is to have daily images tested with the ubiquity autopilot test suite for all desktop manual test entries for each flavor. Let's talk about the current progress and what needs finished.
https:/
Can we blocked merges to proposed based on test results?
Current tests: https:/
- Push server automated tests to qatracker
- Migrate manual tests to 'optional' once automated by ubiquity testing
- Explore ideas for submitting 'generic' manual test results (ie smoketests) to tracker to be counted
- Create Long term plan for manual testcases that are no longer needed -- how to maintain or not?
Blueprint information
- Status:
- Not started
- Approver:
- Jono Bacon
- Priority:
- Undefined
- Drafter:
- Nicholas Skaggs
- Direction:
- Approved
- Assignee:
- Nicholas Skaggs
- Definition:
- New
- Series goal:
- Accepted for trusty
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
Whiteboard
current status:
4 tests running everyday
testing 6 different flavors of ubuntu
need to have long period of stable results before we plug into release process
could block on move from pending/current
could report to qatracker
tests status
4 basic tests
all good
custom install will see some additional tweaks
test migration plans
run all tests in same enviroment
only have CI pull the "vetted" tests, so we can add new tests as we write them
coordinate with CI team on best practicies for migration
~2-3 weeks vetting before migration
perhaps seek to push new tests with each milestone
new tests
run test-suite from preseed (smoke tests)
run after every testsuite
do we still need manual tests? Assumption that auto testing works with other flavours.
certain things only a user will see
UI artifacts are common in the installer
yes, and perhaps migrate them to run-once.. We would still want user confirmation for tests (though we'd feel better about only having one response of it working)
working tests:
Install (entire disk)
Install (manual partitioning)
Install (entire disk with lvm and encryption)
Install (entire disk with lvm)
english and non-english(random language)
add for alpha 1:
Install (auto-resize) - Moved to alpha 2
add for alpha 2:
Install (auto-resize)
Install (erase and re-install)
expanded manual partitioning
post installation tests added
add later:
boottime options
oem install
free software only install
live session
Auto-Resize Notes:
* requires an 80G disk image size anything less than that ubiquity doesn't provide the option to install alongside.
Work Items
Work items for ubuntu-14.04:
Review results of automated tests and ensure they are stable no false positive and no false negative and fix or drop flaky tests Plan to do this review on or around 13 Dec in time for alpha 1. : TODO
[jibel] After the review period, hand over project to Ubuntu CI team : INPROGRESS
[nskaggs] Coordinate with cjwatson and plars to integrate this testing with image publication from pending -> current : INPROGRESS
[nskaggs] Coordinate with CI team on best practicies for migrating new tests as we write them : TODO
[jibel] Integrate publication of results from jenkins jobs with ISO Tracker : TODO
[jibel] Add support for reboot after installation and run post-installation tests : TODO
[jibel] Add support to the runner for multi disks and pre-installed disks : TODO
[nskaggs] add tests timeline to the ubiquity wiki page : TODO
[nskaggs] add bugs for needed tests, filed against ubiquity, tag with needs-autopilot
[dpniel] make sure doc/video/tutorial is added so folks can contribute new ubiquity tests successfully : INPROGRESS
[dpniel] finish working tests : DONE
[dpniel] plan what's required to reach target for alpha 2 and add to the wiki page for others to get involved with. : TODO
Dependency tree
* Blueprints in grey have been implemented.
