Add tooling to enable quick release testing

buildspec.yml

@@ -0,0 +1,19 @@
+version: 0.1
+
+phases:
+  build:
+    commands:
+      - echo Build started on `date`
+      - mvn install --update-snapshots --threads 2.0C --batch-mode -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.compiler.showWarnings=false
+      - chmod -R a+x ./janusgraph-hbase-parent/janusgraph-hbase-098/bin && chmod -R a+x ./janusgraph-hbase-parent/janusgraph-hbase-10/bin
+      - mvn verify --projects $MODULE $ARGS
+      - mvn surefire-report:report --projects $MODULE
+  post_build:
+    commands:
+      - echo Build completed on `date`
+      - mkdir -p results/mvn
+      - echo "$MODULE" | tr "," "\n" | while read MOD; do if [ -d "${MOD}/target" ]; then mkdir -p results/mvn/${MOD} && cp -r ${MOD}/target results/mvn/${MOD}; fi; done
+artifacts:
+  base-directory: results
+  files:
+    - mvn/**/*

janusgraph-codepipelines-ci/README.md

@@ -0,0 +1,146 @@
+# JanusGraph CodePipelines CI
+CodePipelines CI is a mechanism JanusGraph can use to do release testing in massively
+parallel fashion (to the extent of AWS CodePipelines and CodeBuild service limits).
+
+## Prerequisites
+This procedure requires you to have an AWS account and a GitHub account.
+It also requires you to create two service roles in IAM: one for CodePipeline and
+one for CodeBuild. Finally, you need to have the [AWS CLI](https://aws.amazon.com/cli/) installed and on your path.
+
+1. Get a personal access token from [GitHub](https://github.com/settings/tokens) with `repo` and `admin:repo_hook` scopes.
+The `repo` scope is used to push the latest updates on the branch selected below. The `admin:repo_hook` scope is for
+setting up the post-commit hook on GitHub programmatically.
+2. Navigate to the [AWS Console](https://console.aws.amazon.com) and
+[create an IAM User](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) with the following managed
+policies: AmazonS3FullAccess, AWSCodePipelineFullAccess, AWSCodeBuildAdminAccess. This user should be created for
+__Programmatic access__.
+3. For this user, create security credentials and then register them in the `code-pipelines` profile on your computer
+with `aws configure --profile code-pipelines`. Create this profile with a default region that
+[supports CodePipeline and CodeBuild](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/)
+and a default [output format](http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html#controlling-output-format)
+of your choice (`json`, `text`, and `table` are available).
+4. Create an IAM policy for CodeBuild and associate it to a new service role as described
+[here](http://docs.aws.amazon.com/codebuild/latest/userguide/setting-up.html#setting-up-service-role). Select the
+__Amazon EC2__ role type as CodeBuild is not yet available on this page. Edit the new role's trust relationship
+and replace it with the text in step 16 of the CodeBuild user guide.
+5. Create an IAM policy for CodePipelines and associate it to a new service role as described
+[here](http://docs.aws.amazon.com/codepipeline/latest/userguide/iam-identity-based-access-control.html#view-default-service-role-policy).
+Select the __Amazon EC2__ role type as CodePipeline is not yet available on this page. Edit the new role's trust
+relationship and replace the principal `ec2.amazonaws.com` with `codepipeline.amazonaws.com`.
+
+## Setup environment
+Follow the steps below to prepare to create a test pipeline and run tests. First, clean up and set some constant
+environment variables.
+
+```bash
+mvn clean package
+#constants
+export AWS_PROFILE_NAME='code-pipelines'
+export AWS_ACCOUNT_NUMBER=`aws sts get-caller-identity --profile ${AWS_PROFILE_NAME} --output text | cut -f1`
+```
+
+Second, configure the parameters below to customize your test stack.
+
+```bash
+#GitHub personal access token you created in step 1
+export GITHUB_TOKEN=''
+#GitHub organization or user name of the repository to build
+export GITHUB_USERNAME=''
+#GitHub repository name to use for builds
+export GITHUB_REPOSITORY=''
+#GitHub branch in the above repository to use for builds
+export GITHUB_BRANCH=''
+#Name of the AWS CodeBuild service role you created in step 4
+export AWS_CODEBUILD_ROLE=''
+#Name of the AWS CodePipeline role you created in step 6
+export AWS_CODEPIPELINE_ROLE=''
+#Region you want to set up your test stack in
+export AWS_REGION_NAME=''
+#Name of bucket you want to use or create and use for storing your build artifacts.
+export AWS_S3_BUCKET_NAME=''
+#Name of pipeline configuration file to use
+export PIPELINE_CONFIGURATION=''
+```
+
+## Create test pipeline
+This tool uses templates defined in YAML files to configure parallel builds in AWS CodePipeline. By default,
+you can have up to [five parallel actions per stage](http://docs.aws.amazon.com/codepipeline/latest/userguide/limits.html)
+in a pipeline and up to [twenty parallel builds](http://docs.aws.amazon.com/codebuild/latest/userguide/limits.html#limits-builds)
+per region per account. If you need more, you can
+[request to be whitelisted](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for more.
+
+Here is an example of a file that defines two pipelines with parallel build actions.
+
+```yaml
+pipelines:
+- name: j1
+  parallelBuildActions:
+  - name: bdb
+    env:
+    - name: MODULE
+      value: janusgraph-berkeleyje
+  - name: h-l-s
+    env:
+    - name: MODULE
+      value: janusgraph-hadoop-parent/janusgraph-hadoop-2,janusgraph-lucene,janusgraph-solr
+  - name: cassandra
+    env:
+    - name: MODULE
+      value: janusgraph-cassandra
+  - name: test
+    env:
+    - name: MODULE
+      value: janusgraph-test
+- name: j2
+  parallelBuildActions:
+  - name: hbase098
+    env:
+    - name: MODULE
+      value: janusgraph-hbase-parent/janusgraph-hbase-098
+  - name: hbase10
+    env:
+    - name: MODULE
+      value: janusgraph-hbase-parent/janusgraph-hbase-10
+  - name: es
+    env:
+    - name: MODULE
+      value: janusgraph-es
+  - name: cql
+    env:
+    - name: MODULE
+      value: janusgraph-cql
+
+```
+The first pipeline is called `j1` and the second pipeline is called `j2`. Each of these pipelines have four
+parallel build actions defined. Each build action is keyed at the action name and includes the environment
+variables to be passed CodeBuild and used in the buildspec.yml file.
+
+To kick off the regular and TinkerPop tests. Use `export PIPELINE_CONFIGURATION=pipe.yml` for regular tests
+and `export PIPELINE_CONFIGURATION=tp-pipe.yml` for TinkerPop tests.
+
+```bash
+java -jar target/janusgraph-codepipelines-ci-0.2.0-SNAPSHOT.jar \
+--region ${AWS_REGION_NAME} \
+--bucket ${AWS_S3_BUCKET_NAME} \
+--github-owner ${GITHUB_USERNAME} \
+--github-repo ${GITHUB_REPOSITORY} \
+--github-branch ${GITHUB_BRANCH} \
+--profile ${AWS_PROFILE_NAME} \
+--codebuild-role-arn arn:aws:iam::${AWS_ACCOUNT_NUMBER}:role/${AWS_CODEBUILD_ROLE} \
+--codepipeline-role-arn arn:aws:iam::${AWS_ACCOUNT_NUMBER}:role/${AWS_CODEPIPELINE_ROLE} \
+--github-token ${GITHUB_TOKEN} \
+--pipelines ${PIPELINE_CONFIGURATION}
+```
+
+Navigate to the [AWS Console](https://console.aws.amazon.com/codepipeline) and check on the status of your pipeline to see status.
+
+## Cleaning up
+After you follow the steps above, you will end up with resources in the following services:
+1. Build artifacts and source code zips in S3
+2. CloudWatch logs
+3. CodeBuild builds
+4. CodePipeline pipelines, stages, and actions
+5. IAM policies, roles and an IAM user
+6. A GitHub personal access token
+
+You can delete these resources after you are finished running your tests.

janusgraph-codepipelines-ci/pipe.yml

@@ -0,0 +1,37 @@
+pipelines:
+- name: j1
+  parallelBuildActions:
+  - name: bdb
+    env:
+    - name: MODULE
+      value: janusgraph-berkeleyje
+  - name: h-l-s
+    env:
+    - name: MODULE
+      value: janusgraph-hadoop-parent/janusgraph-hadoop-2,janusgraph-lucene,janusgraph-solr
+  - name: cassandra
+    env:
+    - name: MODULE
+      value: janusgraph-cassandra
+  - name: test
+    env:
+    - name: MODULE
+      value: janusgraph-test
+- name: j2
+  parallelBuildActions:
+  - name: hbase098
+    env:
+    - name: MODULE
+      value: janusgraph-hbase-parent/janusgraph-hbase-098
+  - name: hbase10
+    env:
+    - name: MODULE
+      value: janusgraph-hbase-parent/janusgraph-hbase-10
+  - name: es
+    env:
+    - name: MODULE
+      value: janusgraph-es
+  - name: cql
+    env:
+    - name: MODULE
+      value: janusgraph-cql

janusgraph-codepipelines-ci/pom.xml

@@ -0,0 +1,126 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.janusgraph</groupId>
+        <artifactId>janusgraph</artifactId>
+        <version>0.2.0-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <artifactId>janusgraph-codepipelines-ci</artifactId>
+    <name>JanusGraph CodePipelines CI: Distributed release testing.</name>
+    <url>http://janusgraph.org</url>
+    <properties>
+        <top.level.basedir>${basedir}/..</top.level.basedir>
+        <aws.sdk.version>2.0.0-preview-1</aws.sdk.version>
+        <lombok.version>1.16.16</lombok.version>
+    </properties>
+
+    <dependencies>
+        <dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>codepipeline</artifactId>
+            <version>${aws.sdk.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>codebuild</artifactId>
+            <version>${aws.sdk.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>s3</artifactId>
+            <version>${aws.sdk.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>iam</artifactId>
+            <version>${aws.sdk.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>com.google.guava</groupId>
+            <artifactId>guava</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>commons-cli</groupId>
+            <artifactId>commons-cli</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.projectlombok</groupId>
+            <artifactId>lombok</artifactId>
+            <version>${lombok.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-log4j12</artifactId>
+            <version>1.7.12</version>
+            <scope>runtime</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.janusgraph</groupId>
+            <artifactId>janusgraph-test</artifactId>
+            <version>${project.version}</version>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.fasterxml.jackson.dataformat</groupId>
+            <artifactId>jackson-dataformat-yaml</artifactId>
+            <version>2.6.6</version>
+        </dependency>
+        <dependency>
+            <groupId>com.fasterxml.jackson.dataformat</groupId>
+            <artifactId>jackson-dataformat-yaml</artifactId>
+            <version>2.6.6</version>
+        </dependency>
+    </dependencies>
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-shade-plugin</artifactId>
+                <version>3.0.0</version>
+                <executions>
+                    <execution>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>shade</goal>
+                        </goals>
+                        <configuration>
+                            <transformers>
+                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+                                    <mainClass>org.janusgraph.codepipelines.AwsCodePipelinesCi</mainClass>
+                                </transformer>
+                            </transformers>
+                            <createDependencyReducedPom>false</createDependencyReducedPom>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+    <dependencyManagement>
+        <dependencies>
+            <dependency>
+                <groupId>joda-time</groupId>
+                <artifactId>joda-time</artifactId>
+                <version>2.8.1</version>
+            </dependency>
+            <dependency>
+                <groupId>io.netty</groupId>
+                <artifactId>netty-handler</artifactId>
+                <version>4.1.9.Final</version>
+            </dependency>
+            <dependency>
+                <groupId>org.reactivestreams</groupId>
+                <artifactId>reactive-streams</artifactId>
+                <version>1.0.0.final</version>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+</project>