Merge pull request #59 from akhalymon/PLUGIN-72

PLUGIN-72 SAP Hana database Batch source, sink, action and post-action plugins

Author: CuriousVini

README.md

@@ -13,12 +13,20 @@ mvn clean test \
 ```
 Notice that you must change properties for Aurora MySQL and Aurora Postgresql to real before running tests.
 ## Setup Local Environment
-MySQL, Postgresql, MSSQL, DB2, MemSQL are using prebuild images.
+MySQL, Postgresql, MSSQL, DB2, MemSQL, SAP HANA are using prebuild images.
 
 Oracle DB image should be build separately.
 
 MemSQL image should be configure after start.
 
+Note that you should login into docker account to pull SAP HANA image. 
+Account can be created [here](https://hub.docker.com/signup)
+Also, please note SAP HANA is sensitive to some CPU instructions.
+CPU model "host-passthrough" or similar can be required if running inside VM.
+SAP HANA requires that password for DB is provided through url.
+Convenience script ```docker-compose/db-plugins-env/saphana-password-server.sh``` 
+provided for this purpose.
+
 Netezza requires VMware Player for running Netezza emulator.
 
 * [Install Docker Compose](https://docs.docker.com/compose/install/)
@@ -36,8 +44,13 @@ export MEMSQL_LICENSE_KEY=YOUR_LICENSE_KEY
 ```bash
 docker-compose up memsql
 ```
+* Start SAP HANA password service
+```bash
+bash saphana-password-server.sh &
+```
 * Start docker environment by running commands:
 ```bash
+cd docker-compose/db-plugins-env/
 docker-compose up -d
 ```
 * Connect to MemSQL Studio at [http://localhost:8888](http://localhost:8888) 

docker-compose/db-plugins-env/docker-compose.yml

@@ -57,10 +57,32 @@ services:
       - ORACLE_SID=cdap
       - ORACLE_PDB=mydb
       - ORACLE_PWD=123Qwe123
+
   memsql:
     image: memsql/cluster-in-a-box:centos-6.8.10-a53e479edc-1.9.0-1.3.0
     ports:
       - 3308:3306
       - 8888:8080
     environment:
-      - LICENSE_KEY=${MEMSQL_LICENSE_KEY}
+      - LICENSE_KEY=${MEMSQL_LICENSE_KEY}
+
+  saphana:
+    image: store/saplabs/hanaexpress:2.00.040.00.20190729.1
+    hostname: hxehost
+    ports:
+      - 39017:39017
+      - 39013:39013
+    ulimits:
+      nproc: 65535
+      nofile:
+        soft: 1048576
+        hard: 1048576
+    sysctls:
+      - kernel.shmmax=1073741824
+      - net.ipv4.ip_local_port_range=60000 65535
+      - kernel.shmmni=524288
+      - kernel.shmall=8388608
+    extra_hosts:
+      # Alter this if running on non-Linux machine
+      - "host:172.17.0.1"
+    command: --agree-to-sap-license  --passwords-url http://host:1500

docker-compose/db-plugins-env/saphana-password-server.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+# Copyright © 2019 Cask Data, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+
+# Tested with Ubuntu 18.04
+echo -e "HTTP/1.1 200 OK\n\n {\n  \"master_password\" : \"SAPhxe123\"\n } " | nc -q 1 -l 0.0.0.0 1500

pom.xml

@@ -38,6 +38,7 @@
     <module>aurora-mysql-plugin</module>
     <module>aurora-postgresql-plugin</module>
     <module>memsql-plugin</module>
+    <module>saphana-plugin</module>
   </modules>
 
   <licenses>

saphana-plugin/docs/SapHana-action.md

@@ -0,0 +1,32 @@
+# SAP HANA Action
+
+
+Description
+-----------
+Action that runs a SAP HANA command.
+
+
+Use Case
+--------
+The action can be used whenever you want to run a SAP HANA command before or after a data pipeline.
+For example, you may want to run a sql update command on a database before the pipeline source pulls data from tables.
+
+
+Properties
+----------
+**Driver Name:** Name of the JDBC driver to use.
+
+**Database Command:** Database command to execute.
+
+**Host:** Host that SAP HANA is running on.
+
+**Port:** Port that SAP HANA is running on.
+
+**Database:** SAP HANA database name.
+
+**Username:** User identity for connecting to the specified database.
+
+**Password:** Password to use to connect to the specified database.
+
+**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
+will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

saphana-plugin/docs/SapHana-batchsink.md

@@ -0,0 +1,46 @@
+# SAP HANA Batch Sink
+
+
+Description
+-----------
+
+This sink is used whenever you need to write to a SAP HANA table.
+Suppose you periodically build a recommendation model for products on your online store.
+The model is stored in a FileSet and you want to export the contents
+of the FileSet to a SAP HANA table where it can be served to your users.
+
+Column names would be autodetected from input schema.
+
+
+Use Case
+--------
+This sink is used whenever you need to write to a SAP HANA table.
+Suppose you periodically build a recommendation model for products on your online store.
+The model is stored in a FileSet and you want to export the contents
+of the FileSet to a SAP HANA table where it can be served to your users.
+
+Column names would be autodetected from input schema.
+
+
+
+Properties
+----------
+
+**Reference Name:** Name used to uniquely identify this sink for lineage, annotating metadata, etc.
+
+**Driver Name:** Name of the JDBC driver to use.
+
+**Table Name:** Name of the table to export to.
+
+**Host:** Host that SAP HANA is running on.
+
+**Port:** Port that SAP HANA is running on.
+
+**Database:** SAP HANA database name.
+
+**Username:** User identity for connecting to the specified database.
+
+**Password:** Password to use to connect to the specified database.
+
+**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
+will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

saphana-plugin/docs/SapHana-batchsource.md

@@ -0,0 +1,52 @@
+# SAP HANA Batch Source
+
+
+Description
+-----------
+
+Reads from a SAP HANA using a configurable SQL query.
+Outputs one record for each row returned by the query.
+
+Use Case
+--------
+The source is used whenever you need to read from a SAP HANA. For example, you may want
+to create daily snapshots of a database table by using this source and writing to
+a TimePartitionedFileSet.
+
+
+
+Properties
+----------
+**Reference Name:** Name used to uniquely identify this source for lineage, annotating metadata, etc.
+
+**Driver Name:** Name of the JDBC driver to use.
+
+**Host:** Host that SAP HANA is running on.
+
+**Port:** Port that SAP HANA is running on.
+
+**Database:** SAP HANA database name.
+
+**Import Query:** The SELECT query to use to import data from the specified table.
+You can specify an arbitrary number of columns to import, or import all columns using \*. The Query should
+contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDITIONS'.
+The '$CONDITIONS' string will be replaced by 'splitBy' field limits specified by the bounding query.
+The '$CONDITIONS' string is not required if numSplits is set to one.
+
+**Bounding Query:** Bounding Query should return the min and max of the values of the 'splitBy' field.
+For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is set to one.
+
+**Split-By Field Name:** Field Name which will be used to generate splits. Not required if numSplits is set to one.
+
+**Number of Splits to Generate:** Number of splits to generate.
+
+**Username:** User identity for connecting to the specified database.
+
+**Password:** Password to use to connect to the specified database.
+
+**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
+will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.
+
+**Schema:** The schema of records output by the source. This will be used in place of whatever schema comes
+back from the query. However, it must match the schema that comes back from the query,
+except it can mark fields as nullable and can contain a subset of the fields.

saphana-plugin/docs/SapHana-postaction.md

@@ -0,0 +1,39 @@
+# SAP HANA Query Post-run Action
+
+
+Description
+-----------
+Runs a SAP HANA query at the end of the pipeline run.
+Can be configured to run only on success, only on failure, or always at the end of the run.
+
+Use Case
+--------
+The action is used whenever you need to run a query at the end of a pipeline run.
+For example, you may have a pipeline that imports data from a database table to
+hdfs files. At the end of the run, you may want to run a query that deletes the data
+that was read from the table.
+
+
+Properties
+----------
+**Run Condition:** When to run the action. Must be 'completion', 'success', or 'failure'. Defaults to 'success'.
+If set to 'completion', the action will be executed regardless of whether the pipeline run succeeded or failed.
+If set to 'success', the action will only be executed if the pipeline run succeeded.
+If set to 'failure', the action will only be executed if the pipeline run failed.
+
+**Driver Name:** Name of the JDBC driver to use.
+
+**Query:** Query to run.
+
+**Host:** Host that SAP HANA is running on.
+
+**Port:** Port that SAP HANA is running on.
+
+**Database:** SAP HANA database name.
+
+**Username:** User identity for connecting to the specified database.
+
+**Password:** Password to use to connect to the specified database.
+
+**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
+will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.