Skip to content

This feature allows to index points, circles and boxes in PREFIX_TREE mapping for ES. #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amcp merged 1 commit into from
Jun 7, 2017
Merged

This feature allows to index points, circles and boxes in PREFIX_TREE mapping for ES. #294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/elasticsearch.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Elasticsearch is a distributed, RESTful search and analytics engine capable of s
JanusGraph supports https://www.elastic.co/[Elasticsearch] as an index backend. Here are some of the Elasticsearch features supported by JanusGraph:

* *Full-Text*: Supports all `Text` predicates to search for text properties that matches a given word, prefix or regular expression.
* *Geo*: Supports all `Geo` predicates to search for geo properties that are intersecting, within, disjoint to or contained in a given query geometry. Supports points, lines and polygons for indexing. Supports circles, boxes and polygons for querying point properties and all shapes for querying non-point properties. Note that JTS is required when using line and polygon shapes (see <<search-predicates#geoshape,Geoshape documentation>> for more information).
* *Geo*: Supports all `Geo` predicates to search for geo properties that are intersecting, within, disjoint to or contained in a given query geometry. Supports points, circles, boxes, lines and polygons for indexing. Supports circles, boxes and polygons for querying point properties and all shapes for querying non-point properties. Note that JTS is required when using line and polygon shapes (see <<search-predicates#geoshape,Geoshape documentation>> for more information).
* *Numeric Range*: Supports all numeric comparisons in `Compare`.
* *Flexible Configuration*: Supports embedded or remote operation, custom transport and discovery, and open-ended settings customization.
* *TTL*: Supports automatically expiring indexed elements.
Expand Down
2 changes: 1 addition & 1 deletion docs/searchpredicates.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ Additional data types will be supported in the future.

[[geoshape]]
=== Geoshape Data Type
The Geoshape data type supports representing a point, circle, box, line, polygon, multi-point, multi-line and multi-polygon. Index backends currently support indexing points, lines and polygons. Indexing multi-point, multi-line and multi-polygon properties has not been tested.
The Geoshape data type supports representing a point, circle, box, line, polygon, multi-point, multi-line and multi-polygon. Index backends currently support indexing points, circles, boxes, lines and polygons. Indexing multi-point, multi-line and multi-polygon properties has not been tested.
Copy link
Contributor

@sjudeng sjudeng Jun 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add language to make clear that indexing circles and boxes is only supported under Elasticsearch?

Copy link
Contributor Author

@davidclement90 davidclement90 Jun 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only on Elasticsearch ?
Solr is also tested. And I think that the code allow it. https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L361

Copy link
Contributor

@sjudeng sjudeng Jun 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, perfect! That code was added in #79 I just hadn't tested circle/box indexing. Thanks!

Geospatial index lookups are only supported via mixed indexes.

.Note:
Expand Down
124 changes: 70 additions & 54 deletions janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchIndex.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import com.google.common.collect.Iterators;
import com.google.common.collect.LinkedListMultimap;
import com.google.common.collect.Multimap;
import com.spatial4j.core.shape.Rectangle;
import org.apache.commons.lang.StringUtils;
import org.elasticsearch.Version;
import org.elasticsearch.common.geo.ShapeRelation;
Expand Down Expand Up @@ -317,7 +318,7 @@ public void register(String store, String key, KeyInformation information, BaseT
if (textAnalyzer != null) {
mapping.field(ANALYZER, textAnalyzer);
}
break;
break;
case TEXTSTRING:
if (textAnalyzer != null) {
mapping.field(ANALYZER, textAnalyzer);
Expand Down Expand Up @@ -422,11 +423,11 @@ public Map<String, Object> getNewDocument(final List<IndexEntry> additions, KeyI
Object value = null;
switch (keyInformation.getCardinality()) {
case SINGLE:
value = convertToEsType(Iterators.getLast(add.getValue().iterator()).value);
value = convertToEsType(Iterators.getLast(add.getValue().iterator()).value, Mapping.getMapping(keyInformation));
break;
case SET:
case LIST:
value = add.getValue().stream().map(v -> convertToEsType(v.value))
value = add.getValue().stream().map(v -> convertToEsType(v.value, Mapping.getMapping(keyInformation)))
.filter(v -> {
Preconditions.checkArgument(!(v instanceof byte[]), "Collections not supported for " + add.getKey());
return true;
Expand All @@ -446,7 +447,7 @@ public Map<String, Object> getNewDocument(final List<IndexEntry> additions, KeyI
return doc;
}

private static Object convertToEsType(Object value) {
private static Object convertToEsType(Object value, Mapping mapping) {
if (value instanceof Number) {
if (AttributeUtil.isWholeNumber((Number) value)) {
return ((Number) value).longValue();
Expand All @@ -456,7 +457,7 @@ private static Object convertToEsType(Object value) {
} else if (AttributeUtil.isString(value)) {
return value;
} else if (value instanceof Geoshape) {
return convertgeo((Geoshape) value);
return convertgeo((Geoshape) value, mapping);
} else if (value instanceof Date) {
return value;
} else if (value instanceof Instant) {
Expand All @@ -468,18 +469,31 @@ private static Object convertToEsType(Object value) {
} else throw new IllegalArgumentException("Unsupported type: " + value.getClass() + " (value: " + value + ")");
}

private static Object convertgeo(Geoshape geoshape) {
if (geoshape.getType() == Geoshape.Type.POINT) {
@SuppressWarnings("unchecked")
private static Object convertgeo(Geoshape geoshape, Mapping mapping) {
if (geoshape.getType() == Geoshape.Type.POINT && Mapping.PREFIX_TREE != mapping) {
Geoshape.Point p = geoshape.getPoint();
return new double[]{p.getLongitude(), p.getLatitude()};
} else if (geoshape.getType() != Geoshape.Type.BOX && geoshape.getType() != Geoshape.Type.CIRCLE) {
} else if (geoshape.getType() == Geoshape.Type.BOX) {
Copy link

@amcp amcp Jun 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it OK not to catch IOException for BOX but required for CIRCLE?

Copy link

@amcp amcp Jun 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump

Copy link
Contributor Author

@davidclement90 davidclement90 Jun 6, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it's OK.
IOException is throwed by geoshape.toMap()

For Box, the map is constructed without using geoshape.toMap().
For Circle, the map is constructed with geoshape.toMap().

Rectangle box = geoshape.getShape().getBoundingBox();
Map<String,Object> map = new HashMap<>();
map.put("type", "envelope");
map.put("coordinates", new double[][] {{box.getMinX(),box.getMaxY()},{box.getMaxX(),box.getMinY()}});
return map;
} else if (geoshape.getType() == Geoshape.Type.CIRCLE) {
try {
return geoshape.toMap();
Map<String,Object> map = geoshape.toMap();
map.put("radius", map.get("radius") + ((Map<String, String>) map.remove("properties")).get("radius_units"));
return map;
} catch (IOException e) {
throw new IllegalArgumentException("Invalid geoshape: " + geoshape, e);
}
} else {
throw new IllegalArgumentException("Unsupported or invalid shape type for indexing: " + geoshape.getType());
try {
return geoshape.toMap();
} catch (IOException e) {
throw new IllegalArgumentException("Invalid geoshape: " + geoshape, e);
}
}
}

Expand Down Expand Up @@ -557,17 +571,17 @@ private String getDeletionScript(KeyInformation.IndexRetriever informations, Str

switch (keyInformation.getCardinality()) {
case SINGLE:
script.append("ctx._source.remove(\"" + deletion.field + "\");");
script.append("ctx._source.remove(\"").append(deletion.field).append("\");");
if (hasDualStringMapping(informations.get(storename, deletion.field))) {
script.append("ctx._source.remove(\"" + getDualMappingName(deletion.field) + "\");");
script.append("ctx._source.remove(\"").append(getDualMappingName(deletion.field)).append("\");");
}
break;
case SET:
case LIST:
String jsValue = convertToJsType(deletion.value, scriptLang);
script.append("def index = ctx._source[\"" + deletion.field + "\"].indexOf(" + jsValue + "); ctx._source[\"" + deletion.field + "\"].remove(index);");
String jsValue = convertToJsType(deletion.value, scriptLang, Mapping.getMapping(keyInformation));
script.append("def index = ctx._source[\"").append(deletion.field).append("\"].indexOf(").append(jsValue).append("); ctx._source[\"").append(deletion.field).append("\"].remove(index);");
if (hasDualStringMapping(informations.get(storename, deletion.field))) {
script.append("def index = ctx._source[\"" + getDualMappingName(deletion.field) + "\"].indexOf(" + jsValue + "); ctx._source[\"" + getDualMappingName(deletion.field) + "\"].remove(index);");
script.append("def index = ctx._source[\"").append(getDualMappingName(deletion.field)).append("\"].indexOf(").append(jsValue).append("); ctx._source[\"").append(getDualMappingName(deletion.field)).append("\"].remove(index);");
}
break;

Expand All @@ -583,11 +597,13 @@ private String getAdditionScript(KeyInformation.IndexRetriever informations, Str
switch (keyInformation.getCardinality()) {
case SET:
case LIST:
script.append("ctx._source[\"" + e.field + "\"].add(" + convertToJsType(e.value, scriptLang) + ");");
script.append("ctx._source[\"").append(e.field).append("\"].add(").append(convertToJsType(e.value, scriptLang, Mapping.getMapping(keyInformation))).append(");");
if (hasDualStringMapping(keyInformation)) {
script.append("ctx._source[\"" + getDualMappingName(e.field) + "\"].add(" + convertToJsType(e.value, scriptLang) + ");");
script.append("ctx._source[\"").append(getDualMappingName(e.field)).append("\"].add(").append(convertToJsType(e.value, scriptLang, Mapping.getMapping(keyInformation))).append(");");
}
break;
default:
break;

}

Expand All @@ -600,21 +616,21 @@ private Map<String,Object> getAdditionDoc(KeyInformation.IndexRetriever informat
for (IndexEntry e : mutation.getAdditions()) {
KeyInformation keyInformation = informations.get(storename).get(e.field);
if (keyInformation.getCardinality() == Cardinality.SINGLE) {
doc.put(e.field, convertToEsType(e.value));
doc.put(e.field, convertToEsType(e.value, Mapping.getMapping(keyInformation)));
if (hasDualStringMapping(keyInformation)) {
doc.put(getDualMappingName(e.field), convertToEsType(e.value));
doc.put(getDualMappingName(e.field), convertToEsType(e.value, Mapping.getMapping(keyInformation)));
}
}
}

return doc;
}

private static String convertToJsType(Object value, String scriptLang) throws PermanentBackendException {
private static String convertToJsType(Object value, String scriptLang, Mapping mapping) throws PermanentBackendException {
try {
XContentBuilder builder = XContentFactory.jsonBuilder().startObject();

Object esValue = convertToEsType(value);
Object esValue = convertToEsType(value, mapping);
if (esValue instanceof byte[]) {
builder.rawField("value", new ByteArrayInputStream((byte[]) esValue));
} else {
Expand Down Expand Up @@ -755,34 +771,34 @@ public QueryBuilder getFilter(Condition<?> condition, KeyInformation.StoreRetrie
Geoshape shape = (Geoshape) value;
final ShapeBuilder sb;
switch (shape.getType()) {
case CIRCLE:
Geoshape.Point center = shape.getPoint();
sb = ShapeBuilder.newCircleBuilder().center(center.getLongitude(), center.getLatitude()).radius(shape.getRadius(), DistanceUnit.KILOMETERS);
break;
case BOX:
Geoshape.Point southwest = shape.getPoint(0);
Geoshape.Point northeast = shape.getPoint(1);
sb = ShapeBuilder.newEnvelope().bottomRight(northeast.getLongitude(),southwest.getLatitude()).topLeft(southwest.getLongitude(),northeast.getLatitude());
break;
case LINE:
sb = ShapeBuilder.newLineString();
IntStream.range(0, shape.size()).forEach(i -> {
Geoshape.Point point = shape.getPoint(i);
((LineStringBuilder) sb).point(point.getLongitude(), point.getLatitude());
});
break;
case POLYGON:
sb = ShapeBuilder.newPolygon();
IntStream.range(0, shape.size()).forEach(i -> {
Geoshape.Point point = shape.getPoint(i);
((PolygonBuilder) sb).point(point.getLongitude(), point.getLatitude());
});
break;
case POINT:
sb = ShapeBuilder.newPoint(shape.getPoint().getLongitude(),shape.getPoint().getLatitude());
break;
default:
throw new IllegalArgumentException("Unsupported or invalid search shape type: " + shape.getType());
case CIRCLE:
Geoshape.Point center = shape.getPoint();
sb = ShapeBuilder.newCircleBuilder().center(center.getLongitude(), center.getLatitude()).radius(shape.getRadius(), DistanceUnit.KILOMETERS);
break;
case BOX:
Geoshape.Point southwest = shape.getPoint(0);
Geoshape.Point northeast = shape.getPoint(1);
sb = ShapeBuilder.newEnvelope().bottomRight(northeast.getLongitude(),southwest.getLatitude()).topLeft(southwest.getLongitude(),northeast.getLatitude());
break;
case LINE:
sb = ShapeBuilder.newLineString();
IntStream.range(0, shape.size()).forEach(i -> {
Geoshape.Point point = shape.getPoint(i);
((LineStringBuilder) sb).point(point.getLongitude(), point.getLatitude());
});
break;
case POLYGON:
sb = ShapeBuilder.newPolygon();
IntStream.range(0, shape.size()).forEach(i -> {
Geoshape.Point point = shape.getPoint(i);
((PolygonBuilder) sb).point(point.getLongitude(), point.getLatitude());
});
break;
case POINT:
sb = ShapeBuilder.newPoint(shape.getPoint().getLongitude(),shape.getPoint().getLatitude());
break;
default:
throw new IllegalArgumentException("Unsupported or invalid search shape type: " + shape.getType());
}

return QueryBuilders.geoShapeQuery(key, sb, SPATIAL_PREDICATES.get((Geo) janusgraphPredicate));
Expand Down Expand Up @@ -940,11 +956,11 @@ public boolean supports(KeyInformation information, JanusGraphPredicate janusgra
if (janusgraphPredicate instanceof Cmp) return true;
} else if (dataType == Geoshape.class) {
switch(mapping) {
case DEFAULT:
return janusgraphPredicate instanceof Geo && janusgraphPredicate != Geo.CONTAINS;
case PREFIX_TREE:
return janusgraphPredicate instanceof Geo;
}
case DEFAULT:
return janusgraphPredicate instanceof Geo && janusgraphPredicate != Geo.CONTAINS;
case PREFIX_TREE:
return janusgraphPredicate instanceof Geo;
}
} else if (AttributeUtil.isString(dataType)) {
switch(mapping) {
case DEFAULT:
Expand Down
Loading