Reduce amount of time to run tinkerpop tests from 40 hours to 2.5 hours

[amcp]

Status of builds in janusgraph-codepipelines-ci/pipe.yml/j1:

• bdb - Passed in 15 min 10 secs
• hadoop-lucene-solr - Passed in 21 minutes 12 seconds
• cassandra - Passed in 110 minutes
• test - Passed in 21 minutes 37 seconds

Status of builds in janusgraph-codepipelines-ci/pipe.yml/j2:

• hbase098 - Passed in 65 minutes
• hbase10 - Passed in 39 minutes 44 seconds
• es - Passed in 18 minutes
• cql - Passed in 67 minutes

Status of builds in janusgraph-codepipelines-ci/tp-pipe.yml/jtp1:

• bdb - Passed in 118 minutes
• test - Passed in 32 minutes 4 seconds
• cassandra - Failed in 131 minutes (known failures)

Status of builds in janusgraph-codepipelines-ci/tp-pipe.yml/jtp2:

• hbase098 - Failed in 409 minutes (known failures)
• hbase10 - Passed in 467 minutes (known failures)
• cql - Passed in 2 hours

[amcp]

@sjudeng do you have any insight as to why these tests take so long? any clues on getting the time to complete faster?
@mbrukman our development and release velocity will continue to be adversely affected until we can get this time down.

[sjudeng]

There are a large number of tests (see below). The total time is driven by the Cassandra and HBase modules. If you exclude those the total runtime is around 5 hours.

Module	Test Count	Time (sec)
JanusGraph-Test	4496	2892
JanusGraph-BerkeleyJE	3191	12272
JanusGraph-Cassandra	3109	50577
JanusGraph-ElasticSearch	75	696
JanusGraph-HBase	2164	77801
JanusGraph-Hadoop	17	424
JanusGraph-Lucene	39	107
JanusGraph-Solr	68	455
Total	13159	145224

There's probably not much to do to improve time if all tests need to be run. Potential issues with higher parallelism are the use of embedded Cassandra/HBase (maybe running test groups in Docker containers might be an option if that's an issue) and resource limitations (Travis provides only two cores I think).

In the meantime it might just be on contributors/reviewers to determine which tests (core vs. Tinkerpop and BerkeleyDB vs. Cassandra/HBase) need to be run.

[amcp]

@sjudeng I have been thinking about this issue. From your TP 3.2.3 work, I saw the command mvn clean install -Dtest.skip.tp=false that you used to invoke tp tests. Is there a way to run TP tests without running the tests that normally run? Ie, I want to skip the tests under *janusgraph*diskstorage* and *janusgraph*graphdb*. My idea is to cut the tests into as small pieces as possible, and run each of the pieces in parallel.

[sjudeng]

Yes you can run just the TP tests: mvn clean install -Dtest.skip.tp=false -DskipTests=true (see https://github.com/JanusGraph/janusgraph/blob/master/BUILDING.md). I think hbase/cassandra module tests could even be split into groups if high parallelism, and associated high core/memory resource requirements, is an option.

[amcp]

@sjudeng thanks for the tip. I added a few definitions in #129 based on Travis CI, lets see how far I get.

[amcp]

It seems like HBase TP is the tallest pole at 21.6 hours and Cassandra the next tallest pole at 14.1 hours and then by BDB at 3.4 hours. I want to poke at easy ways to divide TP HBase up into 4 hour chunks and the same for TP Cassandra. You just made me realize that I need to raise the timeouts on my build definitions in #129 , thanks!

[amcp]

@sjudeng The longest tests are in cassandra and hbase10, at around 2 hours and 10 minutes each. Every other individual test class takes less time than that to complete. Seems like that will be our new release build cycle time. I suggest we isolate every single test that took more than 900 seconds (15 minutes) in your build in a separate build plan. After some rearranging, I came up with a total of 39 different test groups in plans.xlsx that can run in parallel. This would require 8 pipelines each with 5 parallel actions in the build stage, save the last one which will have 4 parallel actions.

[sjudeng]

It sounds very promising. I'm not familiar with AWS code pipelines. Not sure what you've found but I wouldn't be surprised if many groups need to be fully isolated (separate instance/container) from each other to avoid unexpected interactions. Many may also require at least a couple cores and 1-2G memory. What would be the size/number of EC2 instances needed to run these pipelines to get all tests to complete under 2.5 hours?

[amcp]

I am now at 6 hours 49 minutes, with the HBase 098 and HBase 10 TinkerPop tests being the long pole. The second longest after TinkerPop HBase tests are TinkerPop Cassandra tests at 2 hours and 11 minutes. Is there any way we can make HBase TinkerPop tests take a shorter amount of time?

[amcp]

@sjudeng TP CQL is the longest pole at 14 hours. TP hbase10 follows at 7 hours 47 minutes and then TP hbase098 at 6 hours 49 minutes. TP Cassandra is the next longest at 2:11, under the initial 2.5 hour goal. All other modules are under the 2.5 hour goal.

[amcp]

@sjudeng Everything is passing now except for TP Cassandra, TP Hbase10 and TP Hbase098. The only builds now over 2.5 hours each are Hbase10 and Hbase098. What about TinkerPop test structure makes it difficult to divide the HBase test runs into multiple definitions?

[sjudeng]

I think there are only three HBase TP test classes, so we could split them up that way (e.g. separate group for each class). It's just that each of these classes are an umbrella for a huge number of TP test classes that run through TP JUnit test runners and I'm not sure it's possible to split those since they're run as part of a single HBase TP test class.

Before we go further with splitting though I'd like to take a look at whether HBase might be being started/stopped for each test (as opposed to once per suite). Glancing at this just now it looks like it might be per-test, which would of course be terrible on performance. I'll take a look and report back. If it is per-test it's possible/likely there's some issue(s) with reusing the instance that will need to be resolved.