UTAH Deployment and further usage

Need to discuss:
- Go over WI's left over from last cycle and discuss whether they are still required: DONE
- Current feature set
  * 'run_as' and permissions. i.e. root runs the testsuite setup and cleanup and testcase setup and cleanup commands and creates the testsuite directory but other users might run tests and cannot create files in that directory (discuss adding a mini-schema for commands so that all of them have run_as option)
  * add support for skipping cleanup on failure (possibly by propagating --no-destroy/clean from the provisioner into the client)
- Missing features/needed improvements
- Provisioning methods
  * ARM: INPROGRESS
  * MaaS: BLOCKED
  * LXC: TODO(?)
  * VM snapshots:TODO
  * Reusing already provisioned systems: TODO(?)

- Standardize return codes for test failure, error, timeout, missing parameters, bad environment setup, etc: TODO
- How to pass parameters to a test case and override them in the master runlist: TODO(?)

- Discuss the process of releasing a new stable version of UTAH (i.e., deploy to this place, pass these specific tests, increment version number): TODO
- Discuss versioning (are we ready for 1.0? if not, what do we need for that?)
We will move to version 1.0 when snapshotting and autolanding of submissions are in place

- Discuss packaging approach (i.e. physical machine provisioning is not needed for many installs, should we make it a separate package?)

- Discuss test case skeletons in different programming languages and integrate into phoenix: TODO(?)
- Discuss shortcut methods for test case development (i.e., instead of specifying a runlist, just use dev method on current directory):
- Discuss unittest, nose, in relation to best practices: DELAYED
Maybe we can discover the test cases in a nose test suite to be able to plug easily into utah.
discuss in next team hangout.

- Discuss how should utah be used in a lab to avoid using ssh keys for a given user to be able to bzr branch a test suite
discuss later

- Discuss defaults update in server (precise, i386, desktop)
discuss later

- Discuss creation of new packages:
  - documentation with all the html files, (to create a different project)
  - testing library in python (before common.py gets copied from the example over and over).
  - bootspeed
  - automatic download/update of ISOs, VMs, containers, etc.
- Discuss having just one binary (for the server at least)

Descoped (we do not expect to get to these during this cycle, but we might early next):

[rbasak] Provide MAAS scripts to integrate provisioning method: TODO
[nuclearbob] Work with rbasak to integrate MAAS provisioning: TODO
[nuclearbob] Integrate rbasak's panda scripts into UTAH: BLOCKED
[larry-e-works] Deploying MAAS in the lab and make it stable for production: TODO
[nuclearbob] Start using MAAS from UTAH to provision machines: TODO
[rbasak] Provisioning: Investigate ARM server/netboot testing via QEMU-libvirt on x86: TODO
[nuclearbob] expose virtual watchdog via libvirt: TODO
[nuclearbob] libvirt ARM support: TODO
[joetalbott] Enable the client to run the same test suite more than once: TODO
[joetalbott] Add support for versioned runlists and control flow (while/for and if), i.e. version, timeout, master runlist level: TODO
[javier.collado] Add support for XUnit output: POSTPONED
[joetalbott] Add proper support for testcase types as we currently only support userland: TODO
[nuclearbob] Move QRT Testing to UTAH: TODO

Note 20121217: yaml file generation:
- yaml files need to be generated whether the client runs or not. In the event of failure or iso validation, the yaml file still needs to be generated according to the schema.
  * This requires the class result to be engineered in a way that it can be reused. We don't want to be generating yaml files in 3 different places and want to maintain only one.

Work items for ubuntu-13.04-month-1:
[nuclearbob] Complete the config file so that we can read all the options from there: DONE

Work items for ubuntu-13.04-month-3:
[joetalbott] Look into summary data mismatch on reboot testcases: DONE
[joetalbott] Add 'tc_id' as 'fetch_location/testcase' to result output: POSTPONED
[pwlars] Clarify what the next task is about: DONE
[joetalbott] add support for 'run_as' for build, setup, and cleanup commands at the testsuite and testcase level.: POSTPONED
[nuclearbob] Split the utah package in the agreed fashion: DONE
[javier.collado] Make UTAH server return a default yaml file when things haven't run and there are no logs from the client. This will help with the reporting and make counting pass rates a more exact science: DONE
[cjohnston] Come up with the licensing text to be added to the files: DONE
[cjohnston] Add the licensing information to all the files in the utah project: DONE
[joetalbott] Change phoenix to create files with the appropriate licensing: POSTPONED
[javier.collado] Handle unicode encoding/decoding in shell command output: DONE
[nuclearbob] Fully automating ARM Kernel testing (panda boards): POSTPONED

Work items for ubuntu-13.04-month-4:
[nuclearbob] Remove workaround for lp:1061242 after it's fixed: DONE
[nuclearbob] Provisioning: LXC: POSTPONED
[javier.collado] Only allow 'timeout' as an override in tslist.run: POSTPONED
[nuclearbob] Add a configuration option for keeping the VM around and not run cleanup on failure of a test case for development: DONE
[joetalbott] Add support for quick development methods (i.e. use master.run or dev method on current directory if not provided) : POSTPONED
[joetalbott] Make the runlist a positional argument: POSTPONED
[javier.collado] Create another readthedocs project for dev and add the doc to the utah package: DONE
[javier.collado] Add jenkins job to ensure that ISO static validation is working fine: POSTPONED
[javier.collado] Standardize return codes for test failure, error, timeout, missing parameters, bad environment setup: DONE

Work items for ubuntu-13.04-month-6:
[javier.collado] Add a common.py class to UTAH: POSTPONED
[javier.collado] Reengineer the yaml generation class so that it can be used not just by the client but also by iso static validation and for error situations (see Note 20121217 on the whiteboard): POSTPONED
[javier.collado] Add 'utah' user account to the sudoers list in the utah provisioned client machine : POSTPONED