Cloud Image Testing and Automation
[RATIONALE]:
Currently the Cloud Images are built and released based on upstream tests. The idea is that if the upstream tests work, that the images will just work. However, due to bugs hit this past cycle, and with the proliferation of new clouds, the need to validate that the cloud images work where they should and characterisiticly are what we say they are, is becoming apparent.
Also, with the proliferation of new cloud vendors, there is a need to automate the release process. Some in the community have asked for a more frequent release cadance for the cloud images.
As a result, for the R-cycle, we will implement testing and automation that will be aimed at ensuring enhanced quality of the images.
[GOAL]:
1) Implement deeper testing for the Ubuntu Cloud Images
2) Improve the quality of the Ubuntu Cloud Images by pro-actively looking for regressions and stress testing.
3) Implement Automatic releases to increase the release cadance for the Cloud Images
The following Tests will be implemented:
* Image Characteristic Tests:
- Define attributes of a Cloud Image, i.e. prescense of certain files, file formats, and package sets. For example:
- precences of /etc/cloud and files under it
- check boot loader configruation for grub and grub2
- deeper cloud-init testing
- check SSH configuration
* Bug regression tests: look over the Cloud Image bugs related to the bulding of the images. For example:
- check that /var/log/
- In order for a cloud image bug to be "fixed released" a test should be written to prevent future regressions.
* Image boot tests
- Test that QCow images boot on KVM and OpenStack
- Test that outputed files are valid
- Validate boot loader and console configurations
* Image proactive bug searching in Cloud instances
- Fully excersize disk, I/O and network looking for potential bugs
- Increase Amazon instance sizes tested
- Excercise -proposed updates to check for issues which may cause problems
- Use of Juju for testing Juju/Cloud Integration
- Mock workload tests
* Image performance testing
- Measure I/O performance to identify problems with kernels or userland configurations caused by updates or build process
* Explore implementation of UTAH testing framework
* Integration and publication of results to Jenkins Instance
- Add the daily images to iso.qa.ubuntu.com
- Automatically update test results to iso.qa.ubuntu.com
For Automation:
* Integrate testing into image build process.
- Daily images for the development release must pass characteristic and regression tests.
- Daily images for the stable releases must pass all tests to be published
- Milestones for the development image are subjected to all tests suites
- Introduce automatic promotion of QA'd daily images when the daily has a new kernel or boot-criticial (i.e. reboot required to use) package.
- Automate email announcement generation
- Implement Twitter Announcements
- RSS Feeds
Feedback Requests (via the two major Ubuntu Cloud Lists)
https:/
https:/
Blueprint information
- Status:
- Started
- Approver:
- Antonio Rosales
- Priority:
- High
- Drafter:
- Ubuntu Server
- Direction:
- Approved
- Assignee:
- None
- Definition:
- Discussion
- Series goal:
- Accepted for raring
- Implementation:
-
Started
- Milestone target:
-
ubuntu-13.04-beta-1
- Started by
- Antonio Rosales
- Completed by
Whiteboard
[TEST PLAN]
The core of this blue print is adding tests. Therefore the implementation of this blue print is a test itself.
[ASSUMPTIONS]
It is assumed that this will be a multi-cycle blue print that will undergo refinement. Implementation of some pieces may happen in later cycles.
[RISKS]
The biggest risk for implementation is cost. For some cloud vendors, the ability to implement all tests would represent a significant financal burden. This risk will be mitigated by stepping tests such that only relavent tests will be done as approprate.
[RELEASE NOTE]
Starting with "r", the Ubuntu Cloud Images are now on a release cadance that follows the kernel SRU process. In general, new Cloud Images for all supported stable releases and long-term releases will follow a few days after availability of kernel, gcc and other boot-critical packages. Further, all released daily builds undergo exhaustive testing to ensure quality and to check regressions, performance and integration into Cloud environments. Users who are interested in tracking the testing can find the test results on http://
[FEEDBACK from RFC]:
Suggested for Release Notes and Notification:
1/ release notes for each releases are published and kept on
cloud-images.
from the individual /<suite>/<version>/ directory? As you remove
versions you still keep the release notes.
2/ Set up a notification method via twitter, or RSS Feed.
3/ Keep sending emails for the time being
Work Items
Work items:
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
[darkmuggle-
Dependency tree
* Blueprints in grey have been implemented.
