UEC testing
UEC is difficult to test, and at the same time would benefit from thorough testing. It requires lots of hardware and, with more topologies being supported in 10.04 it will only get more complex. This session will explore how to organize testing for UEC during the 10.04 development cycle, to ensure it reaches the QA level appropriate for an LTS release.
Blueprint information
- Status:
- Complete
- Approver:
- Jos Boumans
- Priority:
- High
- Drafter:
- None
- Direction:
- Approved
- Assignee:
- Mathias Gug
- Definition:
- Superseded
- Series goal:
- Accepted for lucid
- Implementation:
-
Beta Available
- Milestone target:
-
ubuntu-10.04
- Started by
- Dustin Kirkland
- Completed by
- Thierry Carrez
Whiteboard
Secondary engineer: kirkland
Feedback mdz 2009-11-25:
I suspect that some of these work items may be too large, e.g. integrating the test suite into checkbox seems like more than 1-2 days of work, given that I don't think checkbox has a concept of testing across multiple machines yet
jib review (2009-12-2): As discussed on IRC, the biggest uncertainty is in "* Enable Eucalyptus upstream test suite". Please clarify the state of the test suite and what is needed from our side to make this works. Please also double check that all work items are no more than 1-2 days work each.
jib review (2009-12-3): Marked "Eucalyptus test suite integration" as an uncertainty factor and approved for Feature Definition Freeze.
ttx review (2009-12-4) -- The work items now reflect the complexity of the work to be done. Some technical uncertainty around the "Identify hardware requirement to be able to run all the topologies" since the various network arrangements require some magic programmable networking if we are to automate everything. Setting it up could take time, so I think this should also be listed as an Uncertainty.
=======
Status:
Systems fail to reboot after installation (bug 556833). Faliure seems to be isolated to multi-network. For lucid we'll stick with preseed to install base systems with proper networking configuration. UEC will be installed from packages.
Need to decide what to do with uec-testing scripts: deferred to Maverick.
Uncertainty:
[mathiaz] * State of Eucalyptus upstream tests unclear, work items will have to be refined as we go along
[mathiaz] * Identifying & setting up hardware topologies
Work items for lucid-alpha-2:
[mathiaz] * Write functional tests plan: DONE
[mathiaz] * Write Power management tests plan: DONE
[mathiaz] * Write stress/stability tests plan: DONE
[mathiaz] * Write upgrade test plan: DONE
[mathiaz] * Define a comprehensive, finite list of topologies to be tested: DONE
[mathiaz] * Identify hardware requirement to be able to run all the topologies: DONE
[mathiaz] * Automate UEC installation of CLC+Walrus+CC+SC / NCs: DONE
[mathiaz] * Automate UEC Installation of CLC / Walrus / CC / SC / NCs: DONE
[mathiaz] * Automate UEC installation of CLC+Walrus / CC1+SC1 / NC1 / CC2+SC2 / NC2 on single network: DONE
[mathiaz] * Identify a Canonical QA resource to assist with testing: DONE
[mathiaz] * Obtain the Eucalyptus test suite: DONE
[mathiaz] * Look at enabling Eucalyptus upstream test suite: DONE
[mathiaz] * Identify Eucalyptus upstream testing bzr branch(es): DONE
[mathiaz] * Run Eucalyptus upstream functional ebs suite into our uec-testing: DONE
[mathiaz] * Automate most of the functional tests: DONE
[mathiaz] * Automate stress/stability tests: DONE
[mathiaz] * Exercise all installation/
[mathiaz] * Exercise all functional tests on A1: DONE
[mathiaz] * Exercise all stress/stability tests on A1: DONE
Work items for lucid-alpha-3:
[mathiaz] * Get a UEC installed on the hardware from the UEC Network Integration hardware: DONE
[mathiaz] * Get automated installation running with preseed on UEC Network Integration hardware (BLOCKED by bug 512632): DONE
[mathiaz] * Get installation/
[mathiaz] * Get functional tests to run on infrastructure: DONE
[kirkland] * Get stress/stability tests to run on infrastructure: DONE
[mathiaz] * Automate UEC upgrade of existing 9.10 CLC+Walrus+CC+SC / NCs: DONE
[mathiaz] * Run Eucalyptus upstream functional walrus suite into our uec-testing: DONE
[mathiaz] * Test and debug installation from an iso: DONE
[kirkland] * Perform all installation tests for A2: DONE
[kirkland] * Perform all functional tests at A2: DONE
[kirkland] * Perform all stress/stability tests at A2 (do we have any of these?): DONE
[kirkland] * Perform tests on A3 candidate: DONE
[mathiaz] * Automate UEC installation of CLC+Walrus / CC1+SC1 / NC1 / CC2+SC2 / NC2 on mutli-network: POSTPONED
Work items for ubuntu-
[mathiaz] * Follow up with IS on mamoncillo state (replaced with marula): DONE
[mathiaz] * Investigate UEC installation of CLC+Walrus / CC1+SC1 / NC1 / CC2+SC2 / NC2 on mutli-network: DONE
[mathiaz] * Include 8021q kernel modules on the netboot installation media (bug 530459): DONE
[mathiaz] * Create vlan-udeb package (#wait for FFE - bug 530468): DONE
[mathiaz] * Include vconfig command line on the netboot installation media: POSTPONED
[mathiaz] * Write preseed files for CLC+Walrus / CC+SC installs: DONE
[mathiaz] * Write preseed file for NC install: DONE
[mathiaz] * Conduct complete multi network installation: POSTPONED
[mathiaz] * Hand over testing rig to hggdh2: POSTPONED
[kirkland] * Perform all installation tests for A3-topo1 (succeeded, must work around Bug #524147): DONE
[kirkland] * Perform all functional tests at A3-topo1 (succeeded): DONE
[kirkland] * Perform all stress/stability tests at A3-topo1: DONE
[kirkland] * Perform all installation tests for A3-topo2 (succeeded, must work around Bug #527648): DONE
[kirkland] * Perform all functional tests at A3-topo2 (succeeded): DONE
[kirkland] * Perform all stress/stability tests at A3-topo2 (succeeded, 20 instances): DONE
[kirkland] * Perform all installation tests for A3-topo3 (succeeded, must work around Bug #527648): DONE
[kirkland] * Perform all functional tests at A3-topo3 (succeeded): DONE
[kirkland] * Perform all stress/stability tests at A3-topo3 (succeeded, 40 instances): DONE
[hggdh2] * Perform tests on B1 candidate: DONE
[ttx] * Sign off on B1 candidate tests: DONE
Work items for ubuntu-
[mathiaz] * Hand over testing rig to hggdh2: DONE
[ttx] * create new ISO-based UEC topologies tests: DONE
[mathiaz] * Write CLC+Walrus preseed installation in vlan environment: DONE
[mathiaz] * Write CC1+SC1 preseed installation in vlan environment: DONE
[mathiaz] * Write NC1 preseed installation in vlan environment: DONE
[mathiaz] * Test multi-network preseed installation in vlan environment: DONE
[mathiaz] * Write CC2+SC2, NC2 preseeds for multi-cluster installation in vlan environment: DONE
[mathiaz] * Test multi-cluster installation in vlan environment: POSTPONED
[hggdh2] * Perform all installation/
[hggdh2] * Perform all functional tests at B1 (pending bug 542075, POWERSAVE policy not tested): DONE
[hggdh2] * Perform all stress/stability tests at B1 (bug 546293) : DONE
[hggdh2] * Perform tests on B2 candidate (bug 556833, bug 559230): DONE
[kirkland] * Sign off on B2 candidate tests (bug 556833, bug 55923) : DONE
Work items for ubuntu-10.04:
[mathiaz] * Test multi-cluster installation in vlan environment (bug 556833): POSTPONED
[hggdh2] * Perform all installation/
[hggdh2] * Perform all functional tests at B2 (bug 556833): DONE
[hggdh2] * Perform all stress/stability tests at B2 (bug 556833): DONE
[hggdh2] * Perform tests on RC candidate: DONE
[kirkland] * Sign off on RC candidate tests: DONE
[hggdh2] * Perform all installation/
[hggdh2] * Perform all functional tests at RC: DONE
[hggdh2] * Perform all stress/stability tests at RC: DONE
[hggdh2] * Perform all installation/
[hggdh2] * Perform all functional tests at FINAL: DONE
[hggdh2] * Perform all stress/stability tests at FINAL: DONE
[kirkland] * Sign off on release tests: DONE
