Orphaned ES Aliases

Orhpaned ES Aliases

Summary
=======
Searchlight will create orphaned aliases if a resource group name has changed.

Problem Statement
===============
A Searchlight user has the ability to change the name of a resource group. When this happens, the aliases associated with the old resource group name will still exist in ES, but not be used. This will have an impact on the new indexing libraries. See the blueprint:

https://blueprints.launchpad.net/searchlight/+spec/swift-direct-indexing-library

as an example.

Description
=========
Searchlight does not keep track as to which aliases/indexes are valid. Valid meaning being actively used by Searchlight. This turns out to be a problem when we setup and use a new alias.

Here is an example of new aliases being created.

1. User modifies searchlight.conf and sets "resource_group_name" for all Resource Types to "searchlight". The user starts Searchlight.

2. Searchlight creates the aliases "searchlight-search" and "searchlight-listener"

3. The Swift indexing library starts. It queries Searchlight for the aliases to use and receives "searchlight-search" and "searchlight-listener".

4. The user stops Searchlight. The user modifies serachlight.conf and sets "resource_group_name" for all Resource Types to "my-sl". The user starts Seachlight.

5. Searchlight creates the aliases "mysl-search" and "mysl-listener". The previous aliases "searchlight-search" and "searchlight-listener" still exist, but are unused by Searchlight. The Swift indexing library is still using "searchlight-search" and "searchlight-listener".

6. Badness occurs when an end-user tries to query Swift information.

We will need to do two things to fix these issues.

Issue #1:

We need to remove the orphaned aliases as soon as they become orphaned. This implies some Searchlight keeping some state to know what aliases have been previously created by Searchlight. We cannot just delete all aliases that are not being used by Searchlight. In the example above that would mean deleting all aliases that are not "mysl-search" or "mysl-listener". This won't work because the ElasticSearch instance being used by Searchlight may also be used by other non-Searchlight services/processes. We do not know what other non-Searchlight aliases are in existence.

Instead Searchlight needs to tack which aliases were created by Searchlight and are no longer being used. A simple way of doing this is to have a Searchlight management index and store documents under it. No extra infrastructure is needed.

Issue #2:

The indexing libraries need to be aware of when the aliases has changed. Assuming we implement the fix for issue #1 as described above. In this case, the indexing libraries can catch the missing alias exception when reading/writing form ElasticSearch. Upon receipt of this exception, the indexing library can re-query the plugin_info from Searchlight and get the new alias names.