Automated Testing Framework

[asac Jun 10]: this spec is now a dependency of arm-m-ui-and-test-heads; important spec for user-platforms; please prioritize milestones accordingly.
[asac 22 Jun 10]: after discussion with scottb it seems that the work item to write all the testcases currently in the -beta block might require this whole spec to be targetted at final; may I suggest to separate the framework spec (this one) from the test case development? I think for this spec ensuring that a few examples are available would be enough and it would allow us to target beta or even alpha-3.
[asac 22 Jun 10]: massaging spec a bit; setting to ubuntu-10.10 milestone as previously discussed. Please adjust it if you decide to follow my suggestion about splitting out the test case development.
[asac 22 Jun 10]: setting to alpha-3 after discussion with plars; we can bump that to beta if we consider the two items targetted for that milestone part of the core implementation of this spec.
[pwlars 08 Jul 10]: Abrek is hosted on launchpad (http://launchpad.net/abrek) and is usable in its current state for installing and running tests. One test, stream, has been provided as an example of how to write tests. Results parsing support should come soon, as well as some additional commands.
[pwlars 26 Jul 10]: Added support for parsing results, now at a point where it makes sense to start merging more test definitions so that more tests can be installed and run.
[pwlars]: Format for exchanging data has been agreed upon with dashboard, but waiting on dashboard code for receiving and handling this data to land. Implemented sw profile gathering code on client side, working on hw profile gathering code and have a prototype for how communication to the server might work, but need the other side to firm up to test it properly.
[pwlars 06 Aug 10]: hardware profile gathering code has been implemented, but still needs to be utilized in the test runner. Packaging has been worked on, but will not commit the branch until license approval is official.
[pwlars 25 Aug 10]: Packaging done and checked into separate branch. Work in progress on a results module to add additional operations on results stored locally, and on a dashboard plugin to allow for communication with the dashboard.
[pwlars 9 Sep 10]: Modified one of the work items slightly because it was way too general. Other tests to be added will be listed on the testsuites and profilers BP as they emerge.

Status:
[pwlars 8 Oct 10]: Abrek can now upload results directly to the dashboard, or optionally dump the raw json bundles.

Work Items (maverick-alpha-2):
Setup project in launchpad for validation tools: DONE
Create skeleton client framework: DONE
Write code needed for defining a test installer: DONE
Support installation of dependencies: DONE
Write install command: DONE
Support removal of installed tests: DONE
Write code needed for defining a test runner: DONE
Make client capable of running tests: DONE
Write sample tests: DONE
Document how to write tests: DONE

Work Items (maverick-alpha-3):
Write code needed for defining a test parser: DONE
Support listing tests that are known or installed: DONE
Support listing results from previous runs: DONE
Write code to gather software profile: DONE
Write code to gather hardware profile: DONE
Define format for pushing results to the server: DONE
Write code to allow test parsing and conversion to neutral format: DONE

Work items (ubuntu-10.10-beta):
Package the test client for installation: DONE
Add a results module to provide additional operations for locally stored results: DONE

Work items (ubuntu-10.10):
Support registration and configuration for communicating with server: DONE
Add support for pushing results to a server: DONE
Merge test definitions from User Platforms team: INPROGRESS
Allow for profilers to be run while testing: POSTPONED