Linaro Android

Integrate ARMs benchmarks into LAVA

Registered by vishal

Why?
We would want to run a set of benchmarking applications with a least variance possible in LAVA to track the improvements in our builds with optimizations and toolchain imrpovements. In this effort we have automated execution of a set of Application recomended by the benchmarking team in ARM using monkeyrunner:
* Linpack
* Quadrant
* Geekbench 2 ( from google play)
* Vellamo
* Andebench
* Antutu Benchmark-59
* Caffeine mark
* Nbench (from google play)
* GLBenchmark 2.1.4-10

The apks and scripts are privately hosted at :
https://linaro-private.git.linaro.org/gitweb?p=people/bhoj/benchmarks.git;a=summary

We have basic support for monkeyrunner in LAVA but we need support for wifi on the devices in LAVA since few of the benchmark applications have prerequisites of Wifi.

Context?
This is part of the larger ARM benchmarking effort.

What gets produced?
The scripts can be called from LAVA.

Where will the work get put?
LAVA

Blueprint information

Status:
Complete
Approver:
Zach Pfeffer
Priority:
Low
Drafter:
vishal
Direction:
Approved
Assignee:
vishal
Definition:
Approved
Series goal:
Accepted for 2012q2
Implementation:
Implemented
Milestone target:
milestone icon 12.07
Started by
vishal
Completed by
vishal

Related branches

Sprints

Whiteboard

Notes:
[2012/6/25 vishalbhoj] The blueprint would track the WI to make these benchmarks run in an automated manner in LAVA.
[2012/7/9 vishalbhoj] Automated benchmarks have been deployed for snowball. Vellamo,GLBenchmark,Antutu,Quadrant currently fail due to missing wifi connectivity. currently Geekbench,AndEbench,Linpack are running with expected results. Here is an example results for snowball
http://validation.linaro.org/lava-server/dashboard/streams/private/team/linaro/android-daily/bundles/fbf290547e8e7f9b4aefe6d7a10af3d847740865/
[2012/7/16 vishalbhoj] The monkeyrunner scripts for benchmarks are deployed to run on
https://android-build.linaro.org/builds/~linaro-android/panda-ics-gcc47-tilt-stable-blob/#
https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc47-igloo-stable-blob/
[2012/7/23 vishalbhoj] We have deployed monkeyrunner scripts for origen: https://android-build.linaro.org/builds/~linaro-android/origen-ics-gcc47-samsunglt-stable-blob/#build=93 . Currently we have hosted a mirror for benchmarks git into LAVA server till the private git access in LAVA is resolved.

Meta:
Roadmap id: ANDROID2012-BENCHMARKING-PLATFORM
Headline: The set of Android benchmarks that were run by ARM have been integrated into LAVA
Acceptance: The set of Android benchmarks that were run by ARM have been integrated into LAVA

(?)

Work Items

Work items:
[doanac] Add support for Wifi in LAVA: DONE
Fix Monkeyrunner scripts failing due to DDMS timeout: DONE
[doanac] Provide access to private gits from LAVA : DONE
Automate these applications in LAVA for Panda: DONE
Disable Auto-rotation for snowball : DONE
Automate these applications in LAVA for Snowball: DONE
Automate these applications in LAVA for Origen: DONE

Nearby

This blueprint contains Public information 
Everyone can see this information.