Simplifying the Checkbox project

Registered by Daniel Manrique

"One checkbox to rule them all"

The current checkbox test harness infrastructure has a somewhat convoluted structure: there are at least four versions of checkbox (vanilla or community, compatibility, certification and oem). Each includes extensions to base checkbox, as well as a set of appropriate tests.

Historically this has led to dependency problems, confusion as to which version of checkbox contains which test, test conflicts and overlapping, and a coupling of tests with the harness which complicates use, maintenance of tests and support of checkbox and its derivatives.

The goal here is:

1- To simplify the tool and repository structure. Ideally there should only be one checkbox tool, and one test repository. These should be separated so that the test repository can be upgraded without needing to upgrade the tool too, *unless* a test requires a specific version of the tool (which can be nicely expressed with apt dependencies).
2- To manage the different kinds of tests that are done (oem, qa, certification, sru, bootchart, etc) via a whitelist mechanism. The existing whitelisting facilities should be revised to make this easier than it currrently is (i.e. auto-whitelisting of dependencies, possibly a nice GUI to select sets of tests, etc).
3- To clearly document test development and tool usage, enabling both Canonical and the community to more easily contribute tests to the repository.

As a bonus, some work could be done on checkbox to address usability and ease-of-use concerns. This will improve the life of test engineers (who have to spend a lot of their time using checkbox) as well as entice community members to do some more testing (via an easier, nicer, more pleasant-to-use tool).

Benefits: Ideally, checkbox users (QA, Hardware Certification and OEM) will have an easier time writing and running tests, and a consolidated test repository will make it easier to see which tests already exist, helping reduce duplication of work currently caused by isolation between versions of checkbox. A simpler structure will also ease debugging of test-related problems. Finally, a better-structured, friendlier checkbox infrastructure will hopefully entice community to contribute in both creating and running tests.

Blueprint information

Jeff Lane 
Daniel Manrique
Needs approval
Series goal:
Milestone target:
milestone icon november2011
Started by
Victor Tuson Palau

Related branches



== Definition of Done ==

- We have a minimum ("normalized") set of checkbox tool packages and checkbox test packages (ideally, just one of each). - done
- Existing test suites or sets are expressed as whitelists on the tests included in checkbox-data. -done
- It is easy to invoke testing using a whitelist for the type of testing to be done (i.e. certification, sru, unity, and so on). No need for different checkboxes, just different whitelists. -done
- It is easy to write new whitelists (i.e. automatic whitelisting of dependencies and no unintuitiveness or mumbo-jumbo). (not implemented)
- We have clear, up-to-date documentation on how to use checkbox, write tests, define and use whitelists for it.
- *bonus* Checkbox UI is less quirky, easier and quicker to use.

== Observations ==
 * oem-qa-checkbox is heavely modified, but they don't maintain it.
* oem and qa have contributed some of the functionality we need to checkbox but we haven't merged them - need to be more proactive in incorporating community contributions. Our bug review/resolution process also needs to be more proactive.

== Proposal ==

== For Oneiric ==
 * Only one checkbox and several test packages (Adv: install dependencies with tests, but if one fail to install the whole test suite won't be set up)
 * Get rid of checkbox-certification
 * Split core/tests
 * Merge the cores
 * Getting checkbox-editor into universe
 * Ordering of tests (there is already a merge request)
 * Having the tests ordered as the OEM QA does (power_management/supend, power_management/hibernate)

1) Merge versions into core (
2) Documentation
3) Separate out tests
4) Add features (time permitting)


Work Items