Job worker/node awareness
Registered by
ukd1
For several use cases, such as working with large files, it would be useful to be able to specify the machine / node that a particular job is to be run on in order to save transferring of data inter-node or node-server-node.
Blueprint information
- Status:
- Not started
- Approver:
- None
- Priority:
- Undefined
- Drafter:
- None
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- New
- Series goal:
- None
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
Related branches
Related bugs
Sprints
Whiteboard
I have a similar need for cache coherency and large dataset availability. Currently I have to solve it by creating special per-host suffixes on my targeted function names, so that both client and worker know which host the function will run on.
This doesn't scale well, and means I had to write my own consistent hashing library to handle job queueing in front of gearmand.
(?)
