Firmware Test Suite Updates for 12.04

= Initial Ideas =

== FWTS ==
* Upstream ownership? Managing new patches + testing.
* good way to collect BIOS related bugs in LP as a reference to implement a test in fwts? public tag like hwe-firmware?
* Continued work on the fwts results parser
* PCIe ASPM BIOS checks
     - see http://smackerelofopinion.blogspot.com/2011/03/making-sense-of-pcie-aspm.html
* Focus on integration with apport (not implemented during Oneiric cycle)
* The Intel PCC interface (http://acpica.org/download/Processor-Clocking-Control-v1p0.pdf) could do with some sanity checking (LP#863175)
       Looked at this, evaluation of PCC Methods generally do SMIs which are impossible to evaluate from userspace. See notes in LP#863175
* The method test could give references to ACPI specifcation when it detects non-compliant AML (LP#859568)
* The method test currently covers most methods/objects, but should be expanded to cover ALL methods/objects over time. We should aim for this over this cycle.
* The battery test should sanity check _BTP via /proc/acpi/battery/*/alarm (LP#853875)
     Scanned 3336 kernel bugs, 462 contained _BTP
* Catching IRQ mis-routing issues.
* Automated kernel error message extraction to help keep klog test in sync with kernel.
* Dump/validate EFI_SYSTEM_TABLE (signature 0x5453595320494249)
     - hard to find this header - it's passed over at boot time and the _init data is lost. Not found a good way
     to search for header in memory (where is it kept? inconsistent across firmware?)
* Continue expanding regression checking of fwts with http://kernel.ubuntu.com/git?p=cking/fwts-test.git
* Establish a plan for a pilot program for increasing usage with O{E,D}Ms and BIOS vendors with the goal of getting feedback
* Leverage fwts-live as an easy entry point for gathering this data

== FWTS-Live Image ==
* a clear version to identify the kernel and fwts version contained in the image. should rebuild when there is a new Ubuntu kernel release?
* slimming down image size
* can we leverage Ubuntu Core?
* make it bootable on UEFI systems
* re-basing on Oneiric once released
* bringing up Pango experimental builds
* automated (or easier) submission to lp.net/fwts-live
* integrating results parser
* automate opening up nautilus/explorer to appropriate fwts directory on usb media when mounted

== UDS Notes ==
=== Future maintenance ===
* cking transferring to kernel team
* new owner needed for fwts
 * development, packaging, building, etc

plan:
 * start a mailing list for fwts development
  * using exsiting list as announcement list
 * new patches to go to new mailing list for review
 * maintainer to apply acked patches
 * continue with normal builds
  * transfer this process to another HWE Team member
 * implement k-t style "2 acks required" process
  * with a new list, keeping ~firmware-testing-team list free for announcements

=== Automated build/test cycle ===
 * cking has been talking to pgraner about automating fwts (fwtsts?)

=== Apport integration ===
 * not a high priority for cking
  * migrate this effort to live image

=== Intel PCC interface ===
 * new test for fwts
  * low priority
  * any systems using this?
  * is it in the kernel? if not, no need to test
 * requires SMIs, so not possible from userspace

=== ACPI method tests ===
 * would like to reference the spec more
  * eg, when finding errors, explain non-compliance
 * currently covers ~80 of exposed methods
  * would like more coverage (about ~30 more)

=== Battery (_BTP) tests ===
 * low priority, not much usage in the wild [ actually, ~13.4% of firmware has _BTP, so still valid ]

=== IRQ misrouting tests ===
 * can we automate this?
  * cking: "I have no idea [how]"

=== procfs ===
 * need to port tests from using /proc to /sys
  * 80% done presently

=== kernel error log parsing ===
 * can vary between kernel versions
 * currently a manual fixup process
 * takes "a morning" per kernel release
 * how much effort to automate? or warn about potential changes?

=== FWTSTS ===
 * git repo online
 * 30% coverage so far
  * for each test case written, another test case to ensure regressions are found is implemented

=== FWTSTSTSTSTSTS ===
 * coming soon?

=== Non-x86 build deps ===
 * cking has done a bit of work in this area, and the hope is that we should be able to build for armel now.

== = New Test Requests? ===
  * Nothing new in room :)

 === FWTS Live ===
 * Can we include kernel version and fwts version in iso name
 * Can we continue to update SRU kernels as they're released?
  * Decided to use the latest stable userspace (Oneiric), latest kernel in Development (Precise), and latest fwts version available in fwts-devel PPA
 * Image size: see if others have ideas on how we can further slim down image size (action)
 * Always focus on current stable release (with SRU kernel updates)
  * Using latest Precise kernel builds as they're released, copied to fwts-mainline PPA and rebuild against Oneiric
  * This fixes up a few UEFI related bugs
 * Put together dev build when nearing new release
 * Does this boot on UEFI based platforms with Legacy CSM disabled?
  * It does now :) (vanhoof)

=== FWTS Live Data submission ===
 * Can we leverage Ubuntu Friendly to gather fwts results.log for submission? (checkbox can attach arbitrary files (e.g. fwts logs) to a submission that goes to launchpad. A job description file would have to be written and added/pushed to the checkbox package. Other considerations: privacy implications, depending on how much personal data the logs contain, as I believe all hwdb submissions are publically available. Don't hesitate to ask the checkbox/ certification team, they're happy to help with this).
 * Could we write a apport/checkbox symptoms file to submit results.log rather than an apport hook?
  * If so, could we modify fwts_wrapper to ask the user if they would like to submit the data to Launchpad

=== FWTS Results Parser ===
* Give ayan a ping to see where things are at

=== FWTS Misc ===
* Look into automating how nautilus / win explorer can be opened up automagically when mounted
* Look into file formatting issues Win vs Unix (unix2dos)

Work items for ubuntu-12.04:
[colin-king] Check UEFI CSM: DONE
[colin-king] SMBIOS test should fully sanity check the header and preferably dump the header in information fields: DONE
[colin-king] FACS alignment (LP#869575): DONE
[colin-king] Sort out dependencies for non-x86 builds (LP#861927): DONE
[colin-king] Better DMI decoding scanning. (LP#874373): DONE
[colin-king] Handle deprecated /proc and new (replacement) /sys interfaces correctly for Lucid..Precise support: DONE
[jk-ozlabs] setup patchwork for fwts: DONE
[vanhoof] register mail list for patch traffic, subscribe <email address hidden>: DONE
[colin-king] scan though known tables for _BTP usage: DONE
[canonical-hwe-team] assess time required to automate kernel log message changes?: POSTPONED
[vanhoof] chat with bjf on how fwts-live build is composed for a similar project: DONE
[vanhoof] send out request for folks to see what low hanging fruit exists to slim down fwts-live image size: DONE
[manjo] give build a try on UEFI machines: DONE
[vanhoof] sort out booting fwts-live on UEFI with Legacy CSM disabled: DONE
[canonical-hwe-team] Investigate symptoms file based submission of results.log to launchpad (if feasible, update fwts_wrapper to ask first): POSTPONED
[ayan] check in on nautilus automagically opening right path on usb drive on mount: POSTPONED
[ayan] check into file formatting issues: POSTPONED
[colin-king] to put acpi core engine download / apply script to fwts git repo: DONE
[vanhoof] formally kick off OEM/ODM Pilot with fwts-live: DONE
[alexhung] Detect aspm in FADT table - show warning if it is set: DONE
[alexhung] Detect L0s/L1 aspm errors in PCI configuration space - show warning and/or errors: DONE
[vanhoof] always use the latest kernel from the Ubuntu release in development (Precise): DONE
[vanhoof] investigate how to include the kernel version in the name of the .img (will be supported with live-build v3: DONE