Track the performance of Ubuntu on different architectures
Explore the idea of tracking the performance of Ubuntu on different architecture types using our Hardware Certification systems. This could be posted daily during development cycles. Investigate running the tests across all the machines applicable, aggregate the data for each architecture and post results publicly. This should allow us to catch any system performance regressions that come from a particular package update or kernel change.
Blueprint information
- Status:
- Not started
- Approver:
- Marjo F. Mercado
- Priority:
- Undefined
- Drafter:
- None
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- Discussion
- Series goal:
- None
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
Whiteboard
=== Background ===
Right now all of my systems running the trackers are dedicated solely to that task and that's all they do constantly. In regards to the community, it would be a bit harder to verify, but when uploading the test data it also uploads the hardware information and installed packages, etc. i.e. - http://
During Lucid, Phoronix were doing kernel builds and testing on a regular basis.
They want us to contribute to the performance testing efforts.
How to contribute? What sort of things we can do in multiple archs? i.e. ARM
Beyond kernel performance, what else can we measure?
Right now what they are tracking, for Maverick packages, can we seen at
http://
It a three or four step process to setup a new test node. After that it fully autoamted.
Some questions:
1. Is the process documented?
1. What packages can be updated?
The kernel trackery systems right now at kernel-
kernel from the daily PPA. At the Ubuntu Tracker it updates all the packages.
1. Is the entire process automated?
Yes.
1. What kind of infrastructure do you need for each node that is not the node itself?
The process of setting up a new test node is basically installing the phoronix-test-suite
package, running a test installation command, then running a phoronic-test-suite command
that "syncs" your phoronix-test-suite client with the specific Phoromatic account, and then
inserting any cron jobs that do the actual apt-get upgrade.
1. Can we assure consistancy between runs?
<michaellarabel> Right now all of my systems running the trackers are dedicated solely to that task and that's all they do constantly. In regards to the community, it would be a bit harder to verify, but when uploading the test data it also uploads the hardware information and installed packages, etc. i.e. - http://
Marc has one concern, Phoronix takes a long time to run the suite. Suggestion: a subset of the tests.
Downloading from a cache is fully supported (editing just one XML conf file)
Is it interesting from the Kernel team point of view?
Performance is not a priority for the kernel team at the moment
Having our own instance or send results to the Phoronix server
=======
Phoronix Test Suite - Tests
=======
aio-stress - AIO-Stress [VERIFIED, FREE]
apache - Apache Benchmark [VERIFIED, FREE]
battery-power-usage - Battery Power Usage [VERIFIED, FREE]
blogbench - BlogBench [VERIFIED, FREE]
bork - Bork File Encrypter [VERIFIED, FREE]
build-apache - Timed Apache Compilation [VERIFIED, FREE]
build-imagemagick - Timed ImageMagick Compilation [VERIFIED, FREE]
build-linux-kernel - Timed Linux Kernel Compilation [VERIFIED, FREE]
build-mplayer - Timed MPlayer Compilation [VERIFIED, FREE]
build-mysql - Timed MySQL Compilation [VERIFIED, FREE]
build-php - Timed PHP Compilation [VERIFIED, FREE]
bullet - Bullet Physics Engine [VERIFIED, FREE]
bwfirt - Bwfirt [VERIFIED, FREE]
byte - BYTE Unix Benchmark [VERIFIED, FREE]
c-ray - C-Ray [VERIFIED, FREE]
cachebench - CacheBench [VERIFIED, FREE]
compilebench - Compile Bench [VERIFIED, FREE]
compliance-acpi - ACPI [VERIFIED, FREE]
compliance-sensors - LM Sensors [VERIFIED, FREE]
compress-7zip - 7-Zip Compression [VERIFIED, FREE]
compress-gzip - Gzip Compression [VERIFIED, FREE]
compress-lzma - LZMA Compression [VERIFIED, FREE]
compress-pbzip2 - Parallel BZIP2 Compression [VERIFIED, FREE]
crafty - Crafty [VERIFIED, FREE]
dbench - Dbench [VERIFIED, FREE]
dcraw - dcraw [VERIFIED, FREE]
doom3 - Doom 3 [VERIFIED, RETAIL]
encode-ape - Monkey Audio Encoding [VERIFIED, FREE]
encode-flac - FLAC Audio Encoding [VERIFIED, FREE]
encode-mp3 - LAME MP3 Encoding [VERIFIED, FREE]
encode-ogg - Ogg Encoding [VERIFIED, FREE]
encode-wavpack - WavPack Audio Encoding [VERIFIED, FREE]
espeak - eSpeak Speech Engine [VERIFIED, FREE]
et - Enemy Territory [VERIFIED, FRE
etqw-demo-iqc - ET: Quake Wars Image Quality [VERIFIED, NON-FREE]
etqw-demo - ET: Quake Wars Demo [VERIFIED, NON-FREE]
etqw - ET: Quake Wars [VERIFIED, RETAIL]
ffmpeg - FFmpeg [VERIFIED, FREE]
fgl-glxgears - ATI fgl_glxgears [VERIFIED, RESTRICTED]
fhourstones - Fhourstones [VERIFIED, FREE]
fio - Flexible IO Tester [VERIFIED, FREE]
fs-mark - FS-Mark [VERIFIED, FREE]
gcrypt - Gcrypt Library [VERIFIED, FREE]
geekbench - Geekbench [VERIFIED, NON-FREE]
gmpbench - GMPbench [VERIFIED, FREE]
gnupg - GnuPG [VERIFIED, FREE]
graphics-magick - GraphicsMagick [VERIFIED, FREE]
gtkperf - GtkPerf [VERIFIED, FREE]
hdparm-read - hdparm Timed Disk Reads [VERIFIED, FREE]
himeno - Himeno Benchmark [VERIFIED, FREE]
hmmer - Timed HMMer Search [VERIFIED, FREE]
hpcc - HPC Challenge [VERIFIED, FREE]
idle-power-usage - Idle Power Usage [VERIFIED, FREE]
idle - Timed Idle [VERIFIED, FREE]
iozone - IOzone [VERIFIED, FREE]
j2dbench - Java 2D Microbenchmark [VERIFIED, FREE]
java-scimark2 - Java SciMark [VERIFIED, FREE]
jgfxbat - Java Graphics Basic Acceptance Test [VERIFIED, FREE]
john-the-ripper - John The Ripper [VERIFIED, FREE]
jxrendermark - JXRenderMark [VERIFIED, FREE]
lightsmark - Lightsmark [VERIFIED, NON-FREE]
mafft - Timed MAFFT Alignment [VERIFIED, FREE]
md-gromacs - GROMACS MD Benchmark [VERIFIED, FREE]
mencoder - Mencoder [VERIFIED, FREE]
minion - Minion [VERIFIED, FREE]
mrbayes - Timed MrBayes Analysis [VERIFIED, FREE]
n-queens - N-Queens [VERIFIED, FREE]
nero2d - Open FMM Nero2D [VERIFIED, FREE]
network-loopback - Loopback TCP Network Performance [VERIFIED, FREE]
nexuiz-iqc - Nexuiz Image Quality [VERIFIED, FREE]
nexuiz - Nexuiz [VERIFIED, FREE]
npb - NAS Parallel Benchmarks [VERIFIED, FREE]
openarena - OpenArena [VERIFIED, FREE]
opencl-ati - ATI Stream SDK - OpenCL Tests [VERIFIED, FREE]
openssl - OpenSSL [VERIFIED, FREE]
opstone-svd - Opstone Singular Value Decomposition [VERIFIED, NON-FREE]
opstone-svsp - Opstone Sparse-Vector Scalar Product [VERIFIED, NON-FREE]
opstone-vsp - Opstone Vector Scalar Product [VERIFIED, NON-FREE]
padman - World of Padman [VERIFIED, FREE]
pgbench - PostgreSQL pgbench [VERIFIED, FREE]
phpbench - PHPBench [VERIFIED, FREE]
postmark - PostMark [VERIFIED, FREE]
povray - POV-Ray [VERIFIED, FREE]
prey - Prey [VERIFIED, RETAIL]
pybench - PyBench [VERIFIED, FREE]
pyopencl - PyOpenCL [VERIFIED, FREE]
qgears2 - QGears2 [VERIFIED, FREE]
quake4 - Quake 4 [VERIFIED, RETAIL]
ramspeed - RAMspeed [VERIFIED, FREE]
render-bench - Render Bench [VERIFIED, FREE]
scimark2 - SciMark [VERIFIED, FREE]
smokin-guns - Smokin Guns [VERIFIED, FREE]
specviewperf10 - SPECViewPerf 10 [VERIFIED, FREE]
specviewperf9 - SPECViewPerf 9 [VERIFIED, FREE]
sqlite - SQLite [VERIFIED, FREE]
stream - Stream [VERIFIED, FREE]
stresscpu2 - StressCPU2 Stress-Test [VERIFIED, FREE]
sudokut - Sudokut [VERIFIED, FREE]
sunflow - Sunflow Rendering System [VERIFIED, FREE]
super-pi - Super PI [VERIFIED, NON-FREE]
supertuxkart - SuperTuxKart [VERIFIED, FREE]
tachyon - Tachyon [VERIFIED, FREE]
tiobench - Threaded I/O Tester [VERIFIED, FREE]
tremulous - Tremulous [VERIFIED, FREE]
trislam - Triangle Slammer [VERIFIED, FREE]
tscp - TSCP [VERIFIED, FREE]
unigine-heaven - Unigine Heaven [VERIFIED, NON-FREE]
unigine-sanctuary - Unigine Sanctuary [VERIFIED, NON-FREE]
unigine-tropics - Unigine Tropics [VERIFIED, NON-FREE]
unpack-linux - Unpacking The Linux Kernel [VERIFIED, FREE]
urbanterror - Urban Terror [VERIFIED, FREE]
ut2004-demo - Unreal Tournament 2004 Demo [VERIFIED, NON-FREE]
vdrift-fps-monitor - VDrift [VERIFIED, FREE]
vdrift - VDrift [VERIFIED, FREE]
video-cpu-usage - 1080p H.264 Video Playback [VERIFIED, FREE]
video-extensions - MPlayer Video Playback Tests [VERIFIED, FREE]
warsow - Warsow [VERIFIED, FREE]
wine-cloth - Humus Cloth [VERIFIED, FREE]
wine-domino - Humus Domino [VERIFIED, FREE]
wine-fire2 - Humus Fire2 [VERIFIED, FREE]
wine-hdr - Humus HDR [VERIFIED, FREE]
wine-metaballs - Humus Metaballs [VERIFIED, FREE]
wine-vf2 - Humus Volumetric Fogging 2 [VERIFIED, FREE]
wine-water - Humus Water [VERIFIED, FREE]
x11perf - x11perf [VERIFIED, FREE]
x264 - x264 [VERIFIED, FREE]
xplane9-iqc - X-Plane Image Quality [VERIFIED, FREE]
xplane9 - X-Plane [VERIFIED, FREE]
yafray - YafRay [VERIFIED, FREE]
List of kernel tests:
* Networking
* disks
* CPU
How Ubuntu benefits from this?
* Performance regressions
* aio-stress results have changed with ext4 changes
* The important thing is looking at the results
* Loading of applications is critical in ARM
* Top-down tests are necessary; create some tests for things that matter to us
* Use a current set of tests
* GUI Testing
* Consistent load & environment before running the test
== ACTIONS ==
* Look at technology for application GUI testing
== SUGGESTIONS ==
* Consider setting the cpu frequency governor to max before running performance tests.
* Also consider echoing a value from proc(5) into /proc/sys/
