Browser Benchmarking Investigation and Setup
As browsers are now critical and an essential part of the user experience when using both a normal desktop and smaller devices (phone, tablet, etc), having a continuous effort on benchmarking the main browsers available at the market is desired.
The benchmark results will be then used by Linaro engineers and the main Linaro partners, so a proper work plan can be in place for further optimization.
Plan:
* Investigate the current benchmark solutions already used by the main browser vendors
* Integrate the defined benchmarks at LAVA, so it can be daily executed over Linaro images
Blueprint information
- Status:
- Complete
- Approver:
- Ricardo Salveti
- Priority:
- Essential
- Drafter:
- Ricardo Salveti
- Direction:
- Approved
- Assignee:
- John Rigby
- Definition:
- Approved
- Series goal:
- Accepted for trunk
- Implementation:
-
Implemented
- Milestone target:
-
12.11
- Started by
- Ricardo Salveti
- Completed by
- John Rigby
Related branches
Sprints
Whiteboard
[rsalveti, Oct 9, 2012] John, would you mind updating the blueprint with a few WIs based on your current investigation?
[jcrigby, Oct 17, 2012] Looked at mozilla speedtests but could not get it to work on my host so skipped it. Realized that the ideal thing would be to be able to control a browser from an external script and be able to scrape results since many of these tests only display results on a page. selenium.webdriver fits the bill and works well for Firefox out of the box. Chromium works also but needs server called chromedriver that translates the webdriver protocol to the chromium proto. Binaries for chromedriver are available for download for i386 and amd64 but not arm. I uploaded a tweeked chromium-browser package to the linaro kernel ppa to build an arm one.
[rsalveti, Oct 24, 2012] John, please correct me if I'm wrong, but this blueprint will probably need to be finished by the next cycle.
[jcrigby, Oct 25, 2012] Yes, need to finish next cycle. Existing work items include issuing the tests and reporting pass/fail. To get actual results one needs to examine the log. We to decide how to gather the numeric results. They could be gleaned from logs but better solution seems to set up a server side solution to which we can post results. Would like to discuss this with lava and infra teams at connect.
[fboudra, 2012-11-28] John, any ETA?
[jcrigby Nov 28, 2012] working on panda but some tests that were working before are now failing so still debugging.
[jcrigby Nov 29, 2012] after more testing it seems that selenium.webdriver for firefox is broken since last month. Will work around this by only doing chrome-browser for now.
[jcrigby Nov 30, 2012] CI job has been created, it finds the latest panda-x11 prebuilt image and starts a browser-benchmark test on lava using that image.
Meta:
Headline: Daily browser benchmark tests now running on LAVA.
Acceptance: CI jobs trigger daily benchmark tests for Firefox and Chromium on Panda and Origen.
Roadmap id: CARD-129
Work Items
Work items:
Investigate what auto browser benchmarking is available: DONE
Choose one or start with one: DONE
Submit bug/patch to add chromedriver to chromium-browser packaging: DONE
Get it running on host then with lava on host: DONE
Tweek chromium-browser packaging to build chromedriver: DONE
Get it running on a target like panda: DONE
Get test merged with lava trunk: DONE
Work around webdriver hang issue: DONE
Create and debug CI jobs to drive tests to lava: DONE
