BIOS and firmware upgrades

MAAS bios and firmware upgrades
===============================
questions:
 - when?
 - confirmation step? elmo says yes
   - Uploading the binary blob to MAAS is implicit confirmation that it's good.
 - how to detect upgrade needed?
 - how to take allocated machines out of allocation before upgrading? (think about juju using 1000 units, how do you identify which need upgrading?)
 - how to deal with vendor-specific binaries
  - user will upload *something* that runs as a commissioning script (shell archive, even)

Requirements:
 - confirmation of each vendor binary before applying it
  - implicit in uploading to MAAS
 - Require knowledge of whether a particular update was applied or not (or ran and had an error)
 - Make use of existing commissioning script functionality
 - upgrade history

ideas:
 - use a commissioning script (can be a binary)
   - where do the vendor upgrades come from?
 - extend the API to let external scripts enforce policies like "upgrade firmware after a node has been used for X days".
 - let the user select which commissioning scripts will be run during the commissioning phase.
 - Can upgrade in ephemeral environment without touching the hard disk. Turn netboot on, commission, netboot off and boot back to where it was.
 - BIOS upgrade scripts should be able to check the state of the BIOS on the machine; an up-to-date BIOS is a no-op.
 - MAAS could promise to put credentials on the machine such that the upgrade scripts can modify tags on the node, to indicate bios revisions for example. We could even go one step further and provide utilities (available to commissioning scripts) to list the tags attached to a node or attache a new tag to a node.
 - Mark commissioning scripts as destructive or non-destructive, to allow us to do a destructive or non-destructive commissioning cycle.
   - Default to destructive for existing user scripts to be on safe side.
   - Destructive/non-destructive is perhaps not the right division. Commissioning and Refresh might be better, where Commissioning is potentially long, and Refresh is something that can happen much more frequently.
 - separate set of scripts for a "refresh cycle" that would be also run during initial commissioning.
 - After BIOS upgrade we need to recapture lshw information.

NOTE FROM ELMO:
 - Many (HP) BIOS upgrade scripts are interactive - they may have a --yes / --force option but [Elmo] hasn't investigated.
 - The scripts may require a full terminal - `ssh -t` rather than just ssh-and-run-the-script.
 - Piping `yes` to the scripts _might_ work, but [Elmo] hasn't checked.
 - There may be multiple blobs - BIOS, firmware for various odds-and-ends and so on.
 - "Would it be suprising to you if the 'Refresh' commisioning cycle ran on every reboot?"
   - Yes; BIOS updates can take a while, and that could piss me off.
   - Would rather be able to choose when to run BIOS updates.
   - A way to say "for this node, update BIOS on next reboot."
 - In an ideal super-world, this would be a managed rolling process - "Sounds very Landscape to me." ~Elmo.