Add Reporting Facility in Rally
Rally is a great tool to get perf/scale numbers; However when we have a lot of testcases, its hard to read through the entire logs to get a summary of view of what ran and what failed; The proposal here is to put a command line option post rally execution (similar to the json print option) to summarize the testcases ran, and the pass/fail stats; Also on failure the enhancement should list out the failure reasons against each testcase. If this is added, one can even run functionality test from rally and get a summary of the results. This includes 3 work items:
1) Add "rally task show" command that actually prints a summary of the details
2) Make FAIL/SUCCESS based on SLA checks
3) Add special argument to "rally task show" --print-errors. That will print traces
Blueprint information
- Status:
- Not started
- Approver:
- Boris Pavlovic
- Priority:
- Medium
- Drafter:
- Chandra Ganguly
- Direction:
- Approved
- Assignee:
- Chandra Ganguly
- Definition:
- Approved
- Series goal:
- None
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
Related branches
Related bugs
Sprints
Whiteboard
The idea is to use Rally for system sanity every night and we chain together scenarios in one file. So we need something that pretty prints a sequence of scenarios. An example below shows each scenario and overall status without having to go through all the json output etc. This is intended not for 1 scenario but when rally is used for a chaining together scenarios and using it for PASS/FAIL.
Results Summary
---------------
Total : 4
Passed: 2 [50%]
Failed: 2 [50%]
+-----+
| # | Scenario Name | RunTime | Result |
+-----+
| 1.1 | Authenticate.
| 1.2 | Authenticate.
| 2.1 | Authenticate.
| 2.2 | Authenticate.
+-----+
Note: **: Throughput in bps is listed instead of RunTime
Also, Pass/Fail for throughput test is based on 5% tolerance if
no exception found
FAILED TestCases
+-----+
| # | Failed Test Name | Failed Logs |
+-----+
| 2.1 | Authenticate.
| | | -+- "global name 'LOG' is not defined" |
| | | -+- 'Traceback (most recent call last):\n |
| | | File "/usr/local/
| | | packages/
| | | line 84, in _run_scenario_
| | | method_
| | | scenario_output\n File |
| | | "/usr/local/
| | | lly/benchmark/
| | | nticate.py", line 75, in validate_cinder\n |
| | | LOG.error(
| | | global name \'LOG\' is not defined\n'] |
| | | |
| 2.2 | Authenticate.
| | | -+- "global name 'LOG' is not defined" |
| | | -+- 'Traceback (most recent call last):\n |
| | | File "/usr/local/
| | | packages/
| | | line 84, in _run_scenario_
| | | method_
| | | scenario_output\n File |
| | | "/usr/local/
| | | lly/benchmark/
| | | nticate.py", line 75, in validate_cinder\n |
| | | LOG.error(
| | | global name \'LOG\' is not defined\n'] |
| | | |
+-----+
