Stable test suite

The current test suite, which was inherited from Maatkit, needs to be overhauled because over the years it has become unstable under its own size and weight, so to speak. Whereas we tested Maatkit and my vboxes or Baron's, we can no longer do that. We must use http://jenkins.percona.com/ for true continuous integration. Furthermore, we must test on multiple platforms and three versions of MySQL (5.0, 5.1, and 5.5).

At the moment, all the tests work, more or less. prove -r lib/ does not work (unless you're really lucky), but if you run the sub-test suites individually, eventually they will all work. It is precisely this instability we must fix. prove -rs lib/ should work 100% of the time. That way, when a test breaks, we know it's because the code we just merged is causing a regression.

The root of the instabilities is usually related to time. My laptop is fast (dual core, SSD HDD, etc.), but the Jenkins boxes can be *very* slow. Many tests load something and either don't wait or wait a second and then presume the test is ready to proceed. This causes non-determistic failures which are a true pita to track down.

Other causes of instability are being revealed by Jenkins and all its platforms. For example, I used to think 32-bit vs. 64-bit didn't matter because we're using Perl, but now it does matter because we're also using sh and system commands. On 32-bit platforms, for example, awk uses 32-bit integers which is problematic.

Another serious problem that cripples the test suite is that some tests break replication or leave the sandboxes servers not in their original state. MySQL 5.5 seems to cause this pretty easily.

Another problem is PERCONA_TOOLKIT_BRANCH (formerly MAATKIT_WORKING_COPY). This makes it possible to run a test from any directory, but this is no longer manageable because, unlike svn branches which barely work, bzr branches work well so we'll probably end up having many of them.

Yet another problem: this isn't serious but it's highly desirable because it's highly indicative: some tests may leave temp/junk files or directories after running. Or, the tests aren't idempotent and they should be. Every single .t file should be very well-bahaved, tidy, etc.

Finally, the sandbox/ scripts aren't ideal. I wrote them when my Bash wasn't very strong. They all need to be updated and made smarter, more reliable, easier to use, etc.

We need to make a very large investment to make the test suite truly stable. Percona Toolkit seems have to have gain a much wider user base compared to Maatkit, no doubt due to the Percona name brand. So here's what we need to do: (TBD in Las Vegas)

###
### 1.