Improvements to the release process of Checkbox

Checkbox is a tool that is used by a number of parties in their day to day work, and so they depend upon it to be somewhat reliable and not break often. For this to be the case, Checkbox requires a release processs (to indicate when and how it is being released) and a test plan which indicates which functionality is going to be tested to try and ensure that it's working. At the moment it has neither.

== Release Cadence ==

One of the most important aspects of a release process is to have a regular, predictable schedule. This allows users of the software to plan for forthcoming changes and for arrangements to be made to have resources available for proper testing. Also, for testing to be effective there must be a period of stabalisation where critical bugs which were found during testing can be fixed. The following is the proposal for the initial cadence of two weeks:

Days 1-8: All merges may be accepted into trunk, no restrictions
Days 9-12: Only bug fixes may be accepted into trunk and manual testing begins at this point
Days 13-14: Only fixes for critical bugs identified by testing may be accepted into trunk. Manual testing is rerun to confirm bug fixes are effective and that no further regressions occur.

In addition on day 13, there will be a release meeting involving the release co-ordinator and a representative from each stakeholder to go over the major issues found so that everyone can be aware of them. Checkbox will be released on the 14th day of the cadence.

In order for work to continue on developing new features and fixing bugs during this freeze period, a scheme will be worked out to branch the code from which we will release Checkbox at a particular point. This will prevent potentially important work from being delayed while maintaining stability.

Going forward into the next cycle, the plan is to shorten the cadence down to as little as one week. This can only be achieved by increasing automated testing and making sure that there is good confidence in the effectiveness of that testing. In this way 'freeze' periods (which slow development) can be kept to a minimum and improvements brought to the users faster.

The ultimate goal would be for everyone involved to have strong enough confidence in the automated testing that they are willing to accept this as assurance that a version of Checkbox is sound enough to use.

== Routine Automated Regression Testing ==

In order to provide a solid foundation on which to perform more thorough testing prior to release, it is important to have a strong automated test suite in place that can make sure the fundamental functions and features of Checkbox are working properly. Since Checkbox is a fairly complex piece of software and also heavily geared around user interaction in most cases (although it does support running 'headless') it will take some effort to fully automate everything. More to the point, in terms of time resources 'routine automated regression testing' must be kept distinct from automated testing in general. These tests are intended to run pre-merge, on every merge. There must therefore be a limit on the amount of time they take to run. More than a few minutes and it may be necessary to start splitting the tests out.

At the moment Checkbox does have something in the way of a test suite that fits this description, but it is still in its infancy and requires more effort. At the moment the strategy for expanding it is to add tests on a case-by-case basis whenever there is a 'test escape' which looks easily automatable. This is a decent strategy, but going forward some measure of coverage and a more structured approach would be desirable. Checkbox being a (mainly) Python based application, there are several tools which will aid in measuring coverage (http://pypi.python.org/pypi/coverage/)

== Pre-release Manual Testing/Extensive Automated Testing ==

The final line of defence before release needs to be thorough and cover as many of the most important use-cases as possible.Since Checkbox is based to a large extend around manual testing and is primarily used through its graphical user interface, most of the more extensive testing will have to be geared around real or mock 'manual' testing - i.e. manipulation of the graphical user interface. The most important thing to get right here is to properly prioritise the tests that will be run so that the test cases which, if they were to fail it would be considered critical, are introduced and run first. A list of potential use-cases can be seen here:

https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AhbvF3mVZ2BadG9GRXcwdmdZVTFiOW9JT21wbEY1S1E

Initially this testing will have to be manual, but there are tools available which allow for automation of UI interaction (http://xpresser.com/). Still this type of testing would need to be kept distinct from automated regression testing, as it would potentially still take a long time to run.

A further point to be considered is; where to keep the test case definitions? Even if the test cases are automated with a tool such as Xpresser, it is important to store some kind of formal definition for maintenance purposes so that if an automated test breaks then its intent can be kept. Some liason should be made with the Platform QA team to see what they recommend, but it would be possible to fall back on a simple spreadsheet.

During this cycle the goal will be to implement all of the use cases mentioned in the spreadsheet above (except where a use case is dropped by consensus), initially as manual test cases and at least half of the test cases should be automated.

== Versioning/Branch management ==

Not directly related to quality, but still and important aspect of releasing software, is versioning.