Community exploratory testing
For the 14.04 cycle we want to encourage as much exploratory testing as possible. On the phone, tablet and desktop -- everywhere. Let's discuss ideas, plans, needs, etc to ensure this happens.
Review current efforts:
https:/
https:/
Blueprint information
- Status:
- Not started
- Approver:
- Jono Bacon
- Priority:
- Undefined
- Drafter:
- Nicholas Skaggs
- Direction:
- Approved
- Assignee:
- Nicholas Skaggs
- Definition:
- New
- Series goal:
- Accepted for trusty
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
Whiteboard
[amjjawad] If I may share what I have in mind, I think we can agree eventually on some points and set some plans. While it is no secret that Ubuntu and Canonical is focusing their efforts of the Phones and Tables and from the mailing list, I see that very clear, specially the last plan/decision about Ubuntu 14.04 to go for GNOME 3.8. Anyway, I feel there are some important points we should not forget:
1- As much as the phone and table projects are important, let's not forget that Ubuntu is also for PCs and Laptops
2- I strongly suggest to have dedicated group of people for QA/Testing, Bugs, Devs, etc for the i386 and amd64 ISOs. The focus should not be on some aspects and ignore or forget about the others
3- We need a clear plan/roadmap for this cycle (14.04) and we need Plan B so if Plan A will fail, we go automatically to Plan B. I know this is eventually what should happen but I have seen lots of mistakes the previous cycle so hope we can avoid that.
By having dedicated groups for each product, we can ensure successful results in less time as everyone knows what he/she will work on and what to expect etc.
Also, we need to recruit as much new testers as possible. Those who have no experience with testing are the best candidate but then the challenge is who will mentor them or help them to learn? it is a challenge that we need to face and deal with. You need more testers? you should be ready to teach them. Wiki Pages won't be helpful for a newcomer. Videos? yes, that would be helpful a lot.
There are lots of things we can discuss to encourage any kind of test.
---
Exploratory testing
Testing a development release, enourage the culture of exploratory testing, report a bugs.
https:/
Dogfooding the first phone release:
- flash the device
- look for bugs
- (potentially) write autopilot tests
https:/
Prioritizing between ISO/package testing and exploratory testing (came up on yesterday's meeting)
Install development release, go wild, try to break things, look for bugs and report them.
No specific tests or guidelines, focus on exploratory testing.
Communicate to the community.
Explain what's exploratory testing:
- anything that isn't part of the testcases !
- definition: "simultaneous learning, test design and test execution"
Tips and tricks.
Metrics? Reporting?
bug reports aren't necessarily the best metric
still a metric
number of people who tested and what and when
how do collect that info?
James Bach (awful), James Whittaker
Whittaker: taking a tour of an application
video: https:/
book: http://
exploratory testing is easier for new folks, keeps things fresh, excellent for new contributors
What are the outputs?
Bug reports, testing session log.
Mailing list & IRC channel activity --> ultimately should lead to bug reports?
If you don't take notes, you might not be able to reproduce a bug.
How notes will look like?
Are they useful for other people?
Screencapturing?
Can we use the packages tracker for reporting results (create an "exploratory testing" product, and testcases as needed)?
sure, the testcase could simply be a description
should we share what we tested? Do we need more feedback on what's being tested besides bugs?
Define a tag for the bugs reported during exploratory testing. If you use the tracker, you automatically get a tag.
Make a summary with the bugs reported.
Implementation Ideas:
use qatracker
needs a SSO account, requires login
can link to specific page where the reporting form is -> users do not necessarily need to navigate
we can modify the tracker to create a simpler page (just bribe stgraber with milk+cookies)
use a spreadsheet
where? google docs? (requires a google account)
use wiki
the ubuntu wiki is slow, on some days really slow, so it's not really a good alternative
use pad
the ubuntu etherpad needs a SSO account as well and a membership on the -etherpad team
use +1 page
record bugs, notes, time spent
sounds like the tracker, but maybe a bit simpler
needs creating
why reinvent the wheel?
cgoldberg and knome offer to help with tracker :-)
Work Items
Work items:
[nskaggs] communicate wiki pages, definitions, goals, etc: TODO
[nskaggs] include methodology on wiki pages: TODO
[nskaggs] make a video: TODO
[elopio] make a summary of the exploratory testing chapter on the Agile Testing book.: TODO
[nskaggs] mockup how it could work on the tracker and share: TODO
Dependency tree
* Blueprints in grey have been implemented.
