ARM Server Benchmarking and Performance

From Etherpad:

Workload tests:
 - bzr branch lp:server-workload-testing
 - Currently done and maintained by Canonical
 - Linaro could help getting it to work across the ARM boards available there as well as with LAVA
openssl: openssl-test? gnu-tls?
ARM Workload:
 - Linaro is looking at extracting the power consumed when stressing the board support
 - It'd be useful to benchmark not only the test results, but the power that was consumed during the test run
Areas to investigate while checking for performance:
 - Not that clear at the moment
 - Generally looking at the usual test cases/benchmarks for a start
 - One way would be to get feedback from the hw vendors to understand what would be the priority
ACTION:
[rsalveti] Make sure we have a session at Linaro Connect to cover power usage at the ARM boards during test case executions
IO/Storage: Ceph, Swift, Gluster?
Areas of focus on feature/functional side:
 - KVM
 - LXC
 - OpenMPI
Calxeda is interested in OpenMPI performance/testing.
From P:
As actual ARM server hardware begins to enter the marketplace, a standardized way to test and benchmark performance must be developed and run on all active platforms we support for ARM server.
http://www.phoronix-test-suite.com/
http://openbenchmarking.org/
Upcoming PTS tests: MySQL, TPCC MySQL, various SPEC tests (might not be public), Unixbench, plus others (can also easily add in any others, just ask)
https://wiki.ubuntu.com/ARM/QA/Server
https://blueprints.launchpad.net/ubuntu/+spec/server-o-arm-image-qa
https://blueprints.launchpad.net/ubuntu/+spec/server-o-arm-image-validation
- membase?
- Ensure that multi-system testing setups are standardized
- weekly meeting to reveiw testing on #ubuntu-meeting (Thursdays 15:00 UTC)
ACTIONS:
- Automate testing [explore PTS documentation about batch-run options]
- Find an IPV6 testing tool
- Add a test for Membase
- Add a test for Hadoop and other workloads
- Explore stress/load testing (memtester, stress, kernel build test, netperf, unix bench)
- Test on real hardware when available
- Compare P test results against O.
- AHCI testing
- Investigate PTS logging sensor data [Run any test(s) with "MONITOR=all" environmental variable to see all sensors... or see included HTML/PDF documentation... or run "phoronix-test-suite system-sensors" to see all supported/unsupported sensors. Try latest PTS release for latest sensor coverage.]

------
=User Stories=
 - George wants to setup an openmpi cluster on his new ARM server nodes and would like a common set of tools to measure performance of openMPI on his ARM cluster.
 - Rebekah is setting up her new ARM cluster and would like a set of common tools to measure performance of virtualization technologies such as KVM and LXC.
 - Jacob has his new ARM server nodes up and running with Ubuntu and would like use a set of common tools to test performance of workloads such as Hadoop in his environment.

=Assumptions=
 -All of the mentioned pieces of software already work on ARM platforms.
 -The work here is to create a common set of scripts for folks to use to tune and analyze performance on their ARM servers. This goal is not to produce performance results.

=Test Plans=
 - Identify appropriate benchmark for each workload
 - Run each script on all available ARM server hardware and some x86 hardware by first set
 - Automate running of scripts for each workload.
* Script will be complete for this blueprint when it accurately exercises the workloads, provides benchmarking data for performance evaluation, and is able to executed in automated process.

=Release Note=
 - Common set of performance set of testing scripts available for ARM servers
  *Once work is complete need to details what the scripts are and where they can be obtained.

Work items:
[server-workload-testing-team] Add a test for KVM: POSTPONED
[craig.magina] Add a test for LXC: TODO
[craig.magina] Add a test for OpenMPI: INPROGRESS
[server-workload-testing-team] Add a test for Membase: POSTPONED
[craig.magina] Add a test for Hadoop and other workloads: DONE
[server-workload-testing-team] Add a test for AHCI: POSTPONED
[server-workload-testing-team] Ensure that multi-system testing setups are standardized: POSTPONED
[server-workload-testing-team] Setup weekly meetings to reveiw testing on #ubuntu-meeting (Thursdays 15:00 UTC): TODO
[server-workload-testing-team] Automate testing (explore PTS documentation about batch-run options): DONE
[server-workload-testing-team] Find an IPV6 testing tool: POSTPONED
[server-workload-testing-team] Explore stress/load testing (memtester, stress, kernel build test, netperf, unix bench): POSTPONED
[server-workload-testing-team] Test on real hardware when available: DONE
[server-workload-testing-team] Compare P test results against O: POSTPONED
[server-workload-testing-team] Investigate PTS logging sensor data: POSTPONED
[rsalveti] Make sure we have a session at Linaro Connect to cover power usage at the ARM boards during test case executions: TODO