User-defined index prioritization

[sharpau]

As a JanusGraph user, I would like to be able to tell JanusGraph to prioritize certain indices higher than others, because I know things about the distribution of the keys in those indices that makes one index particularly slow.

Full background:

We've run into an issue with JanusGraph (preexisting from Titan) with index prioritization (referring to GraphCentricQueryBuilder#constructQueryWithoutProfile). We have 2 indices: a mixed index in ElasticSearch, and an exact-match composite index on two properties. The mixed index is a pretty normal name index. However, the composite index is extremely dense - we basically switched to using this composite index because we kept ending up with supernodes in the graph that made certain vertices impractical to work with, so the composite index may contain millions of entries on only a single-digit number of keys.

Each of these indices work fine on their own, but when you have a query that both indices apply to it gets problematic. In short: we'd like to always hit the mixed index first, because it narrows down the number of results significantly more quickly than the composite index. But the index scoring in query construction automatically gives composite indices a weight of 2000 just for being composite indices. So this means we use the super dense index first and don't really narrow down the number of results any, and it's much slower (whereas if we remove the clauses that match the composite index, the mixed index alone performs fine).

So far the best idea we have to fix this is to only query the mixed index and do the composite index filtering in memory. This is just a) a hassle and b) very error-prone, because when all your queries have a limit, doing any filtering outside of the graph traversal means that you don't know what limit to give to the traversal, and traversals without limits can have problems with the Cassandra backend (i.e. exceeding max frame size).

In my ideal world, I would have some way of telling JanusGraph "hey, when you are considering these two indices, always go to this mixed index first!"

[davidclement90]

I have the same kind of issue and I find two ways to deal with it thanks to PR #233.

1. If I want an exact match on my property, I configure my composite index to match equal operator and my mixed index to match textContains operator. Then I set ES mapping with an exact match.

2. If not, I create a fake property. The composite index still match equal operator. The mixed index is composed by the property and the fake property. I use copy_to on the ES mapping to copy the property to the fake property. So I query the fake property to use mixed index.

[myitroad]

I come across the same problem.
In detail, when I use both Mixed index and Composited index. In one query witch to match two vertex properties, one by textContains match using Mixed index, other by extract match using Composited index. The query result return as an abnormal order.

I have try only by ES query (using es-head plugin), the result order (score) seems great. Below is my es query body:

{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "rdfs:label": {
              "query": "this is my query",
              "operator": "or"
            }
          }
        }
      ],
      "should": [
        {
          "match_phrase": {
            "rdfs:label": {
              "query": "this is my query",
              "slop":"1"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "from": 0,
  "size": 10,
  "sort": [],
  "aggs": {}
}

While I use both textContains match and extract match, the result order seems abnormal. Below is gremlin query body:

g.V().has("rdfs:label",Text.textContains("this is my query")).has("rdfs:category","business").valueMap(true)

Additional, i have change some source code to fit our need, such as use es operator or instead of and:

//Java org.janusgraph.diskstorage.es.compat.AbstractESCompat
    public Map<String,Object> match(String key, Object value, String fuzziness) {
        final ImmutableMap.Builder builder = ImmutableMap.builder().put("query", value);
        builder.put("operator", "or");
        if (fuzziness != null) builder.put("fuzziness", fuzziness);
        return ImmutableMap.of("match", ImmutableMap.of(key, builder.build()));
    }

We focus on the result order, and haven't find effective ways to resolve this. Could any one give me a hint?