CQL storage backend #166

pkendall64 · 2017-03-15T19:50:31Z

Adds a CQL stotage backend that is compatible with the table structure of the Astynanx code.
This PR addresses issue #35.

A new library (javaslang) is introduced which provides very nice functional library which makes the code a lot more concise and understandable.

janusgraph-bot · 2017-03-15T19:51:03Z

Committer of one or more commits is not listed as a CLA signer, either individual or as a member of an organization.

mbrukman

Thank you for the PR! This is a massive contribution. Please also add yourselves to CONTRIBUTORS.txt and your company to AUTHORS.txt in the root of this repo.

I am far from a Cassandra/CQL expert (so I'm hoping @fppt can weigh in on this, and anyone else who's proficient with Cassandra), but I'd like to ask a few things:

this repo uses a 4-space indent, this PR uses tabs exclusively; would it be possible to switch to 4-space indent everywhere so we don't diverge from a single standard?
can you please add the module to .travis.yml at the root of the repo and mark it optional so that it gets automatically tested? Then we can also see what Travis has to say about this change (I'm guessing it wasn't tested with the PR, as we specifically address each module in a separate build).

pkendall64 · 2017-03-16T07:41:51Z

We'll get onto those changes and update the PR, thanks for the feedback.

mbrukman · 2017-03-16T20:28:18Z

@pkendall64 – why did you delete the branch and close the PR? You can just update the code on your side and git push -f to update the branch and we'll continue the review. No need to open a new PR each time we have a round of reviews.

You can probably re-open this PR (since you created it), recreate the branch and force-push to it via git push -f and we can continue where we left off.

pkendall64 · 2017-03-16T21:19:47Z

Yeah, thats what I did and it just disappeared!

mbrukman · 2017-03-16T21:44:10Z

@pkendall64 – you pushed "delete branch" button on GitHub, so it deleted the branch in orionhealth:feature/cql-backend but presumably you have a local copy of it on your workstation? In which case, you can just push to the same repo / branch to continue.

You can also fetch PR on the CLI using git.

In the worst case scenario, if you've deleted the branch in the orionhealth repo as well as locally, you can still attach .patch or .diff to the PR URL to get the full contents — click on either of those links to see what I mean.

pkendall64 · 2017-03-16T21:56:57Z

Sorry about that, all fixed now. I had to checkout the old hash and force push that, then force push the new stuff over the top and now it's ok again.
Don't really know what happened the first time!

mbrukman · 2017-03-16T22:07:48Z

Great to hear your PR is back, @pkendall64! :-)

amcp

good contribution. I think you need to add some more tests, there is some hairy logic in store manager regarding computing supported feature set. perhaps find a way to test enabled/disabled of features that depend on configuration.

amcp · 2017-03-18T02:23:13Z

janusgraph-cql/.classpath

@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="UTF-8"?>


is this an eclipse specific file? should we be committing this to the repo, or should people be able to determine their own IDE configurations?

Sure, I'll remove the eclipse files

amcp · 2017-03-18T02:23:23Z

janusgraph-cql/.project

@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>


same as above, eclipse specific

amcp · 2017-03-18T02:24:15Z

janusgraph-cql/pom.xml

+            </plugins>
+        </pluginManagement>
+    </build>
+</project>


add newline to the end of file

amcp · 2017-03-18T02:27:52Z

janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/CQLKeyColumnValueStore.java

+
+package org.janusgraph.diskstorage.cql;
+
+import static com.datastax.driver.core.querybuilder.QueryBuilder.*;


please import specific things and not wildcard

amcp · 2017-03-18T02:28:52Z

janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/CQLKeyColumnValueStore.java

+        this.tableName = tableName;
+        this.getter = new Getter(storeManager.getMetaDataSchema(this.tableName));
+
+        initialiseTable(this.session, this.storeManager.getKeyspaceName(), tableName, configuration);


spelled initialized elsewhere in this project so please match existing spelling

amcp · 2017-03-18T02:40:35Z

janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/CQLStoreManager.java

+    }
+
+    private Cluster initialiseCluster() throws PermanentBackendException {
+        // XXX: Deal with retry policy and potentially others


amcp · 2017-03-18T02:41:23Z

janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/CQLStoreManager.java

+                fb.keyOrdered(false).orderedScan(false).unorderedScan(true);
+                break;
+            }
+            case "org.apache.cassandra.dht.ByteOrderedPartitioner": {


create string constants that get the canonical name from the actual classes. that way if version update changes the package of ByteOrderedPartitioner, you can catch it as compile time error here

I don't think you want to have compile time dependencies on these classes as they are internals to cassandra and we really only want this depend on the driver jar and not the cassandra internals.
Perhaps it would be better to get the simplename of the class and switch on that accordingly?

yes switch on simplename please

amcp · 2017-03-18T02:42:06Z

janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/CQLStoreManager.java

+        return builder.build();
+    }
+
+    private Session initialiseSession(@SuppressWarnings("hiding") final Cluster cluster, final String keyspaceName) {


why is hiding the member cluster ok?

This is just our eclispe settings. I'll remove the annotation.

amcp · 2017-03-18T02:42:51Z

janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/CQLStoreManager.java

+                        .flatMap(addTime -> Iterator.ofAll(keyMutations.getAdditions()).map(addition -> columnValueStore.insertColumn(key, addition, addTime)));
+
+                return Iterator.concat(deletions, additions)
+                        .grouped(20) // XXX Pull this out as a configuration option


please pull out the batch size into configuration

amcp · 2017-03-18T02:43:13Z

janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/CQLTransaction.java

+import com.datastax.driver.core.ConsistencyLevel;
+import com.google.common.base.Preconditions;
+
+public class CQLTransaction extends AbstractStoreTransaction {


add classdoc

Added classdoc.

boliza · 2017-03-22T11:25:48Z

should the CQL storage Backend just a part of the module janusgraph-cassandra, not a single module?

janusgraph-bot · 2017-03-22T18:48:05Z

Please verify the committer name, email, and GitHub username association are all correct and match CLA records.

pkendall64 · 2017-03-22T19:53:52Z

We have seperated the CQL backend because we don't want to bring in cassandra-all, we only want the driver.

We are also working on a PR that creates a janusgraph-cassandra parent that has embedded, thrift, astyanax and CQL as child projects along with a core project which will contain common configuration and other cassandra helper and abstract classes. There will also be a common test project which will contain common test code.

This should make the while cassandra section a lot cleaner. Our ultimate goal is that you should be able to have maven dependencies on the backends you require (e.g. cassandra-cql and elasticsearch) and have a minimal set of transitive dependencies pulled in to create a runnning instance if JanusGraph.

janusgraph-bot · 2017-03-22T21:57:03Z

Committer of one or more commits is not listed as a CLA signer, either individual or as a member of an organization.

sjudeng

Thanks for the hard work on this! At some point before merge please squash commits where possible.

pkendall64 · 2017-05-07T01:42:06Z

Ok, thanks for the thorough review! We'll look into squashing where appropriate to minimise it to the actual functional commits.

…

On Sun, May 7, 2017 at 1:08 PM sjudeng ***@***.***> wrote: ***@***.**** approved this pull request. Thanks for the hard work on this! At some point before merge please squash commits where possible. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#166 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfS5O1v3s0AjZGF6K4NbfEZsp5cuXwFks5r3RmIgaJpZM4MecTi> .

… and prepared statements Signed-off-by: Samant Maharaj <samant.maharaj@orionhealth.com> Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com> Signed-off-by: Jason Plurad <pluradj@us.ibm.com>

Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

pkendall64 · 2017-05-08T20:21:53Z

I have squashed a lot of the commits so it shows the initial implementation included a lot of the fixes we applied along the way, and then the reviews and extra functionality. If there's any other squashing or rearranging, please let us know and we'll get them done.

sjudeng · 2017-05-11T02:00:47Z

Can you squash 4c5672f, 3b00163, 7558097 and 2c11799 into a single commit, and 8415550, fc6fbf4 and 4850899 into a single commit? I think those go together well and the end result would be 7 commits, which seems reasonable to me.

Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com> Signed-off-by: Samant Maharaj <samant.maharaj@orionhealth.com>

…control the thread lifecycle Signed-off-by: Samant Maharaj <samant.maharaj@orionhealth.com>

… deployment Various cleanup as per review comments Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

…esting Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

pkendall64 · 2017-05-11T19:17:47Z

Squashed commits as suggested

sjudeng · 2017-05-11T20:45:11Z

@mbrukman Has all your feedback been resolved here?

@hsaputra,@fppt,@dylanht Are there any objections to merging this PR (pending thumbs up from @mbrukman)? I'm not a Cassandra user myself (@amcp not sure if you are) so current reviews might not have covered all aspects of the contribution. But that said there has been a lot of community interest on this and I think once merged these users will be able to try it out and provide additional feedback that can be tracked in subsequent issues/PRs if necessary.

hsaputra · 2017-05-11T21:14:31Z

@sjudeng The code in general looks good, I didn't have check for correctness on runtime since I am not Cassandra user. But, one main comment is the dependency on io.javaslang which now changed to io.vavr. I am ok to change it later but better be the next PR submission bc there is a good reason why they change from javaslang to vavr =P

sjudeng · 2017-05-11T22:37:58Z

@hsaputra Good call. Looks like the first version under io.vavr just came out yesterday so ... way to be on top of it!

@pkendall64, @samant-orionhealth I added #271 to track work to make this dependency update. Hopefully you'll be able to help out with this work after this PR is merged.

pkendall64 · 2017-05-11T23:32:08Z

Sure, we can start on that now in preparation for the merge.

…

On Fri, May 12, 2017 at 10:38 AM sjudeng ***@***.***> wrote: @hsaputra <https://github.com/hsaputra> Good call. Looks like the first version under io.vavr just same out yesterday so ... way to be on top of it! @pkendall64 <https://github.com/pkendall64>, @samant-orionhealth <https://github.com/samant-orionhealth> I added #271 <#271> to track work to make this dependency update. Hopefully you'll be able to help out with this work after this PR is merged. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#166 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfS5EUJpnY8aZNgu_RFcg9FM1wSlCQVks5r443JgaJpZM4MecTi> .

hsaputra

With tracked issue to update javaslang dependency, LGTM

hsaputra · 2017-05-11T21:12:06Z

janusgraph-cql/pom.xml

+            <artifactId>guava</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.javaslang</groupId>


This one has been changed to vavr.io so not sure if you want to change it now given potential naming conflict from Oracle.

I've updated ticked #271 with a link to a branch in our repo with this migration. I'll create a PR once this is merged.

sjudeng · 2017-05-12T19:22:48Z

@pkendall64 Thanks for jumping on that. Do hold it for another PR though as I think we're about ready to go on this and I wouldn't want to hold it up to wait for another set of reviews on the new update.

sjudeng · 2017-05-16T01:54:57Z

@mbrukman It looks like your feedback has been addressed here. Can you confirm?

sjudeng · 2017-05-21T15:18:40Z

@pkendall64 A minor conflict in .travis.yml was created with merge of separate PR. I resolved it but it created another commit. I think it's fine but you're also welcome to rebase/squash to remove the merge commit if you'd like. Separately we'll need to look at getting the module covered better under Travis.

@mbrukman Unless you have any objections I'd like to move forward with the merge here. We have four approvals and I've confirmed that your feedback has been addressed.

mbrukman

LGTM

pkendall64 · 2017-05-21T19:21:26Z

@sjudeng I'm ok with the merge commit. Thanks for getting this merged and the thorough reviews from everyone involved.

CQL storage backend

janusgraph-bot added cla: no This PR is not compliant with the CLA cla: yes This PR is compliant with the CLA and removed cla: no This PR is not compliant with the CLA labels Mar 15, 2017

mbrukman requested a review from fppt March 16, 2017 03:33

mbrukman requested changes Mar 16, 2017

View reviewed changes

pkendall64 closed this Mar 16, 2017

pkendall64 deleted the feature/cql-backend branch March 16, 2017 20:14

pkendall64 reopened this Mar 16, 2017

mbrukman requested review from amcp, dylanht and hsaputra March 16, 2017 22:07

amcp requested changes Mar 18, 2017

View reviewed changes

janusgraph-bot added cla: no This PR is not compliant with the CLA cla: yes This PR is compliant with the CLA and removed cla: yes This PR is compliant with the CLA cla: no This PR is not compliant with the CLA labels Mar 22, 2017

janusgraph-bot added the cla: no This PR is not compliant with the CLA label Mar 22, 2017

janusgraph-bot removed cla: yes This PR is compliant with the CLA cla: no This PR is not compliant with the CLA labels Mar 22, 2017

sjudeng approved these changes May 7, 2017

View reviewed changes

Samant Maharaj and others added 2 commits May 9, 2017 08:14

Initial implementation of CQL Cassandra backend using Datastax driver…

690956f

… and prepared statements Signed-off-by: Samant Maharaj <samant.maharaj@orionhealth.com> Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com> Signed-off-by: Jason Plurad <pluradj@us.ibm.com>

Update travis build, authors and contributors files

a8736fd

Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

pkendall64 and others added 5 commits May 12, 2017 07:14

Fixes as per reviews and add more unit tests, including tinkerpop tests

d928f5f

Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com> Signed-off-by: Samant Maharaj <samant.maharaj@orionhealth.com>

Modified usages of Future to use a managed ExecutorService so better …

2aac19f

…control the thread lifecycle Signed-off-by: Samant Maharaj <samant.maharaj@orionhealth.com>

Implement missing methods to support ByteOrderedPartitioner and LOCAL…

877fac8

… deployment Various cleanup as per review comments Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

Add option to set cql protocol-version

c3385e6

Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

Allow setting a system property for the cassandra to connect to for t…

d0ea61e

…esting Signed-off-by: Paul Kendall <paul.kendall@orionhealth.com>

sjudeng mentioned this pull request May 11, 2017

Update from javaslang to io.vavr #271

Closed

hsaputra approved these changes May 11, 2017

View reviewed changes

fppt approved these changes May 12, 2017

View reviewed changes

Merge branch 'master' into feature/cql-backend

757b396

mbrukman approved these changes May 21, 2017

View reviewed changes

sjudeng merged commit d139804 into JanusGraph:master May 21, 2017

sjudeng mentioned this pull request May 30, 2017

New Cassandra backend using CQL w/ prepared statements #35

Closed

sjudeng mentioned this pull request Jul 3, 2017

Static cluster/session management in CQL backend #384

Merged

bwatson-rti-org pushed a commit to bwatson-rti-org/janusgraph that referenced this pull request Mar 9, 2019

Merge pull request JanusGraph#166 from orionhealth/feature/cql-backend

bee0a7e

CQL storage backend

micpod pushed a commit to micpod/janusgraph that referenced this pull request Nov 5, 2019

Merge pull request JanusGraph#166 from orionhealth/feature/cql-backend

9b92432

CQL storage backend


		package org.janusgraph.diskstorage.cql;

		import static com.datastax.driver.core.querybuilder.QueryBuilder.*;

Uh oh!

CQL storage backend #166

CQL storage backend #166

Uh oh!

Conversation

pkendall64 commented Mar 15, 2017 • edited by amcp Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janusgraph-bot commented Mar 15, 2017

Uh oh!

mbrukman left a comment

Choose a reason for hiding this comment

Uh oh!

pkendall64 commented Mar 16, 2017

Uh oh!

mbrukman commented Mar 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkendall64 commented Mar 16, 2017

Uh oh!

mbrukman commented Mar 16, 2017

Uh oh!

pkendall64 commented Mar 16, 2017

Uh oh!

mbrukman commented Mar 16, 2017

Uh oh!

amcp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boliza commented Mar 22, 2017

Uh oh!

janusgraph-bot commented Mar 22, 2017

Uh oh!

pkendall64 commented Mar 22, 2017

Uh oh!

janusgraph-bot commented Mar 22, 2017

Uh oh!

sjudeng left a comment

Choose a reason for hiding this comment

Uh oh!

pkendall64 commented May 7, 2017 via email

Uh oh!

pkendall64 commented May 8, 2017

Uh oh!

sjudeng commented May 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkendall64 commented May 11, 2017

Uh oh!

sjudeng commented May 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

pkendall64 commented Mar 15, 2017 •

edited by amcp

Loading

mbrukman commented Mar 16, 2017 •

edited

Loading

sjudeng commented May 11, 2017 •

edited

Loading

sjudeng commented May 11, 2017 •

edited

Loading

sjudeng commented May 11, 2017 •

edited

Loading