Dynamically obtain addresses and statuses from workers for worker-based engine
Worker-based engine should be able to dynamically obtain information about workers, inluding:
- service discovery: where the workers are available and what each of them can do
- status: what is each worker doing (which tasks are running on it right now)
This should be done in a way that does not affect, or better improves scalability and resilience of WBE.
It is suggested that this functionality should be implemented in pluggable manner, allowing engine to employ one of provided backends through common interface.
Blueprint information
- Status:
- Started
- Approver:
- Joshua Harlow
- Priority:
- High
- Drafter:
- Ivan Melnikov
- Direction:
- Approved
- Assignee:
- Joshua Harlow
- Definition:
- Approved
- Series goal:
- None
- Implementation:
-
Slow progress
- Milestone target:
- None
- Started by
- Joshua Harlow
- Completed by
Related branches
Related bugs
Sprints
Whiteboard
The possible backends for this may use:
- same transport that is used for communication for transports that support broadcasts/fanouts;
- fixed configuration (e.g. have one backend that works the way engine works now: gets list of topics from config instead of querying it somewhere);
- zookeeper;
- filesystem
- ...
Gerrit topic: https:/
Addressed by: https:/
Working on wbe-worker-info (WIP)
Addressed by: https:/
Begin abstracting out the topic requesting from WBE engine
Addressed by: https:/
Use monotonic time when/if available
Addressed by: https:/
WIP for having the ability to have tooz find workers+topics
Addressed by: https:/
Add a WBE worker finder base class and use it
Addressed by: https:/
Use a notifier instead of a direct property assignment
Work Items
Dependency tree
* Blueprints in grey have been implemented.
