Chaos drivers to test corner cases
It would be very neat to implement a set of 'chaos' drivers that could be implemented for each pluggable 'driver-like' backend that can be provided in nova (for example the servicegroup driver, or the vm driver..). The concept would be that a single chaos driver would wrap a working driver and the chaos driver would randomly (likely via a specified seed and/or rate) raise different types of exceptions that the driver interface allows to be thrown. For the cases where it does not throw it would pass on the request to the underlying driver, thus it would act as a driver that sporadically fails at a much higher rate (depending on the randomness) than the underlying wrapped driver. This would be neat as a way to test the corner cases of the code using said driver. It also provides a repeatable (due to the set seed) & unique view into what state the system is left in after said exceptions occur. The results of such types of 'chaos' drivers could provide unique insights into how the overall system recovers from failures and could lead to new code to ensure the system is left in a consistent state (if it is not already).
Blueprint information
- Status:
- Not started
- Approver:
- None
- Priority:
- Undefined
- Drafter:
- Joshua Harlow
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- Drafting
- Series goal:
- None
- Implementation:
- Unknown
- Milestone target:
- None
- Started by
- Completed by
Related branches
Related bugs
Sprints
Whiteboard
JH: It might also be useful to have this in cinder, heat, quantum ... to test there ability to handle 'chaos'
JH: Similarily one could create chaos clients that wrap the existing nova/glance... clients but cause spurious http errors (or similar)JH: It might also be useful to have this in cinder, heat, quantum ... to test there ability to handle 'chaos'
JH: Similarily one could create chaos clients that wrap the existing nova/glance... clients but cause spurious http errors (or similar)
Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)