[EDP] Add a new job-types endpoint

Registered by Trevor McKay

The current /jobs/config-hints/<job_type> endpoint is not adequate for providing configuration hints for job types because it does not take a cluster or plugin argument. This has been a problem for a long time, since the config hints need to be specific to a particular plugin type and even the version (often synonymous with the hadoop version). Additionally, there is not a way currently for a user to query a Sahara instance to find out what kinds of jobs its supports.

To solve both of these issues, we can add a new job-types endpoint that will report the following information to the user:

All of the job types that can be run on that Sahara instance
   For each job type, the plugins that support it
     For each plugin, the versions of that plugin that support it with (optionally) configuration hints for that version

With this endpoint, a user can know exactly which types of jobs can be run and what the choices are for a cluster deployment to run those jobs (what image is needed, what templates do I need to use or create, etc)

Proposed JSON output from the endpoint would look something like this (for a cluster with only the hdp plugin enabled and without config hints)

{
    "job_types": {
        "Hive": [
            {
                "description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
                "name": "hdp",
                "title": "Hortonworks Data Platform",
                "versions": {
                    "1.3.2": {},
                    "2.0.6": {}
                }
            }
        ],
        "Java": [
            {
                "description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
                "name": "hdp",
                "title": "Hortonworks Data Platform",
                "versions": {
                    "1.3.2": {},
                    "2.0.6": {}
                }
            }
        ],
        ... and more job types here ...
    }
}

Blueprint information

Status:
Complete
Approver:
Sergey Lukjanov
Priority:
High
Drafter:
Trevor McKay
Direction:
Approved
Assignee:
Trevor McKay
Definition:
Approved
Series goal:
Accepted for kilo
Implementation:
Implemented
Milestone target:
milestone icon 2015.1.0
Started by
Sergey Lukjanov
Completed by
Sergey Lukjanov

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/edp-job-types-endpoint,n,z

Addressed by: https://review.openstack.org/157563
    [EDP] Add a new job-types endpoint

Addressed by: https://review.openstack.org/161250
    Add job-types endpoint

Addressed by: https://review.openstack.org/161263
    Implement job-types endpoint support methods for HDP plugin

Addressed by: https://review.openstack.org/165220
    Implement job-types endpoint support methods for Vanilla plugin

Addressed by: https://review.openstack.org/165231
    Implement job-types endpoint support methods for Spark plugin

Addressed by: https://review.openstack.org/165236
    Implement job-types endpoint support methods for CDH plugin

Addressed by: https://review.openstack.org/165494
    Implement job-types endpoint support methods for MapR plugin

Addressed by: https://review.openstack.org/165524
    Implement job-types endpoint support methods for Fake plugin

FFE

Addressed by: https://review.openstack.org/166494
    Adding HDP config hints implementation

Addressed by: https://review.openstack.org/167420
    Adding config hints for vanilla plugin

Addressed by: https://review.openstack.org/169466
    Adding config hints for CDH plugin

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.