Support API retry function with Idempotency in creating/updating a stack

Registered by Takashi Natsume

Currently Heat doesn't have a retry function in creating/updating a stack.
In case of API request failure, Heat would change stack status to "XXX_FAILURE" .
(Or start rollback process if the rollback flag was true.)

However, I think there are some circumstances/scenarios where retry is appropriate.
(i.e. 503 response or timeout due to server failover)

On the other hand, however, a retry function may result in creating duplicate resources when Heat couldn't get response from nova and other APIs.
In order to avoid this, I believe leveraging idempotency that is discussed in nova could be a solution.

https://blueprints.launchpad.net/nova/+spec/idempotentcy-client-token

Our idea for implementing a retry function to Heat is:
Heat to retry POST requests with a ClientToken header to avoid creating duplicate resources, until it reaches a retry limit.
Heat to retry PUT/GET requests until it gets 2xx response or it reaches a retry limit. ClientTokens are not used in this case.
Heat to retry DELETE requests until it gets 2xx response or 404 response or it reaches a retry limit. ClientTokens are not used in this case.

Note:
Some of pythonclients have retry functions, but they cannot be utilized because Heat has to judge the necessity of retry by itself.

Blueprint information

Status:
Not started
Approver:
Steve Baker
Priority:
Undefined
Drafter:
None
Direction:
Approved
Assignee:
None
Definition:
Drafting
Series goal:
None
Implementation:
Not started
Milestone target:
None

Related branches

Sprints

Whiteboard

Hm removing icehouse since we have not "accepted" this proposal just yet.

For more details, See the following site.
https://etherpad.openstack.org/p/kgpc00uuQr

(stevebaker) Removing from icehouse until there is an assignee who starts working on it.

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.