[EDP] Job Manager Components
This component is responsible for Job management lifecycles: creating, scheduling, monitoring etc.
We assume to support following job types:
* Pig
* Hive
* Jar file
* Oozie workflow (not in version 0.3)
* YARN application (not in version 0.3)
* Streaming api scripts (not in version 0.3)
Object model for this component has two items: job and job execution.
Job Object
* Name
* Description
* Job type
* Mains (location of main job binary)
* Libs (locations of supporting job binaries)
Job Execution
* Job Id
* Input Data Source Id
* Output Data Source Id
* Start time
* End time
* Cluster Id
* Info (job status from Oozie)
* Progress
* Oozie Job Id
* Return code
* Job Configs
* Logs (Optional, possible not will be in version 0.3)
This component should implement following methods:
* Create job
* List jobs
* Delete job
* Execute job. This method retrieves the job binaries listed in the Job object referenced by Job Id, applies user specified configuration for the job, and starts the job on an already running cluster.
* Terminate job instance
* List job instances
* Get job instance status
Blueprint information
- Status:
- Complete
- Approver:
- Sergey Lukjanov
- Priority:
- High
- Drafter:
- Trevor McKay
- Direction:
- Approved
- Assignee:
- Nadya Privalova
- Definition:
- Approved
- Series goal:
- Accepted for 0.3
- Implementation:
- Implemented
- Milestone target:
- 0.3
- Started by
- Nadya Privalova
- Completed by
- Sergey Lukjanov
Related branches
Related bugs
Sprints
Whiteboard
https:/
Gerrit topic: https:/
Addressed by: https:/
Added REST API for job and data source with simple validation
Addressed by: https:/
First version of job manager
Work Items
Dependency tree
* Blueprints in grey have been implemented.