12.10 certification testing

During the Q cycle, the certification team will have a number of routine duties to perform

- Weekly testing to check that tools are working correctly (while also aiming to discover some hardware related bugs in Ubuntu)
- Milestone testing, which should be more thorough than weekly testing, in order to catch more bugs and raise them with the appropriate parties as soon as possible.
- Certification testing, at the end of the cycle to determine whether systems can be certified or not.

The purpose of this blueprint is to think about what changes can be made to these processes, in order to do better certification.

= Functional coverage =

Some new hardware is becoming commonplace and we should be including it in our certification coverage, at least to the extent that we test it and if it fails we report that about the system:

Functional coverage usually takes a good hour, so this will be discussed in a separate session.

- Accelerometers should be included as a greylist item
- USB 3.0 should be included as a greylist item (meaning that USB 3.0 devices need to be tested against the USB 3.0 port)
- 3G/WWAN modems should be included as a greylist item
- Touchscreens multitouch should be considered for inclusion as a greylist item
- HDMI audio should be included as a greylist item
- Considering adding Super-key testing (i.e. launcher shortcut) to spec and backport to 12.04.1

= Process =

== 12.10 kernel in 12.04 LTS testing ==

From 12.10 onwards, changes to the kernel and some other closely linked packages such as X will be tested to work on the LTS (12.04) during the development cycle. This means that systems scheduled to be certified with 12.10 will be also tested on 12.04 LTS with the 12.10 backport, to ensure that nothing breaks.

Also, this means that we will need to make changes to the certification site (both public and internal) to be able to distinguish a system certified with 12.04 kernel, or with 12.04 with a backported kernel.

Strictly speaking this would double the testing effort required by the certification team, since each system would have to be tested with each kernel in combination with the 12.04 userspace and 12.10 userspace.

One method of doing this would be to change the test infrastructure so that a 12.04 test run is done immediately after the 12.10 one is completed.

== Weekly testing ==

At the moment weekly testing is performed on a subset of about 25 systems available in the certification labs. A tool is used to try and get the best hardware coverage possible on this small set of systems (by not including multiple systems with the same GPU or Wireless card for example). The main purpose is to identify when the testing tools break, but issues with Ubuntu itself may be uncovered as well. For this reason an automated run is done (no manual testing), which means that some issues may not be detected. This mostly works well, but improvements may be made to better detect regressions in the tools (there were one or two cases in the last cycle where problems went unnoticed for some time). For example, the status of some tests changed from passing, to being skipped due to changes in the internal working of Checkbox. Since they weren't appearing as failures this wasn't picked up.

This could be achieved by having a method that compares two subsequent week results, and reports on things that might have changed.

Testing of the 12.10 kernel in 12.04 will also have to be included in the weekly testing process. It might be necessary to test them bi-weekly if a good method for integrating them with the normal test run cannot be found.

== Milestone testing ==

Testing at release milestones is performed on the same systems as weekly testing, but instead of only doing a fully automated run, manual testing is performed as well. This means that any issues which would block certification will be detected, assuming they exist on the systems in the test pool. Last cycle it was found that some systems which weren't included in the test pool had very critical failures caused by a driver for a component not covered by certification (IR reciever). We would not like to have to test on every system at every milestone, but at the same time we'd like to be able to detect issues like this earlier, which can only really be achieved by a full test run. Some sort of comprimise may be possible.

Because systems will need to be tested manually at milestones, the technique described above for handling the 12.10 kernel in 12.04 LTS described above will probably not be suitable.

== ARM Certification ==

It is already known that the Certification testing tools (mainly Checkbox) are mostly not working on ARM devices. Since some ARM certification testing may be necessary during the Q cycle we need to achieve a baseline of usability so that Checkbox can be adapted to testing a specific ARM based product. This will be defined as the ability for Checkbox to submit hardware information and a basic set of test results.