Running LibreOffice testsuites for Ubuntu q-series

http://<email address hidden>/msg03771.html
Notes:

    If using ppa, would require 12+ hours to build
    if daily, would monopolize a machine everyday
    Developer build
        quick
    Multiple Test Suites
    During build, do inbuild tests
        can't be used (against installation)
    Test suite that runs over the UNO-API
    could be used against installation
    Ubuntu ships crippled (incomplete) libreoffice installation (e.g. without libreoffice-base)
    how to run full test suite against crippled installation?
        blacklist tests?
    install full installation and test anyways?
    Are we targetting packaging or upstream bugs?
        both
    perhaps we run suite twice; once for upstream, once for ubuntu?
    Build was redone for precise cycle (getting away from go-oo build/libreoffice-build repo)
        atypical
            caused some one-off bugs
    Nothing in the QA lab for libreoffice yet
    Simplest way to test:
        Get package, make source build, install libreoffice, run test suite
    Proposed repo is used to help test before pushing to release archive
        autopkgtest
            uses debian control file
                has test
    pulls source from archive
        build,upgrade, downgrade, run test
    check for dependency breakage
    does it make sense to repull and recompile libreoffice when dependency changes?
        rebuild once a week and ignore dependencies
            would cause too many rebuilds, too many dependencies
    Use Jenkins
        weekly build, normal jenkins interactions
    Do we run against development version that we are targetting upon stable
        makes sense to track versions we intend to land when stable?
        focus on packaging bugs, ignore the bugs from upstream?
    git binary diffs to bissect commits for bug
        can narrow to a range of 64 commits

Decisions:
    Regular rebuilds happening (once a week)
    Run the test suite twice
        once against ubuntu default install (i.e. without libreoffice-base package)
        once against full install

Results available from public jenkins instance:
-release: https://jenkins.qa.ubuntu.com/job/quantal-pkg-libreoffice/
-prerelease: https://jenkins.qa.ubuntu.com/job/quantal-pkg-libreoffice_prerelease/
upstream git: https://jenkins.qa.ubuntu.com/job/quantal-pkg-libreoffice_git/

Work items:
[jibel] Set up on the lab regular rebuilds of the package (with bjoern-michaelsen): DONE
[jibel] Automate creation of the test environment: DONE
[jibel] Write a script that automates the build of libreoffice in the test env and run the build time tests: DONE
[jibel] Integrate test suite with Jenkins and publish results to jenkins.qa.ubuntu.com (https://jenkins.qa.ubuntu.com/job/quantal-pkg-libreoffice/): DONE

Work items for ubuntu-12.10-beta-2:
[bjoern-michaelsen] Running the tests with the default install and with a full install (with jibel): DONE
[bjoern-michaelsen] Check results weekly and report any problems: DONE
[jibel] Create a local git mirror of http://anongit.freedesktop.org/git/libreoffice in the lab and sync over http to workaround lab limitations: DONE
[jibel] Patch rules file on-the-fly to use the local mirror: DONE
[jibel] Add jenkins jobs for mirror sync and upstream build/test : DONE
[jibel] Test build from upstream release tag/ Revert to using an unmodified rules file (RT 55402): BLOCKED