diff --git a/hub/documentation/building-blocks/environment-variables.rst b/hub/documentation/building-blocks/environment-variables.rst index 40cfedc94a..3d863006a8 100644 --- a/hub/documentation/building-blocks/environment-variables.rst +++ b/hub/documentation/building-blocks/environment-variables.rst @@ -5,6 +5,6 @@ Environment Variables An :ref:`environment variable ` is a named value that you can reference in your configuration. Environment variables are used to parameterize your configuration so that you can easily enable/disable or change certain aspects of your configuration. If you have an environment variable called ``myvariable`` then you can reference it in configuration like this: ``"$ENV(myvariable)"``. -.. NOTE:: +.. important:: Do not use environment variables for sensitive values; use :ref:`secrets ` instead. Environment variables are global only. diff --git a/hub/documentation/service-configuration/configuration-metadata.rst b/hub/documentation/service-configuration/configuration-metadata.rst index 487f6cdb3c..a7480a443d 100644 --- a/hub/documentation/service-configuration/configuration-metadata.rst +++ b/hub/documentation/service-configuration/configuration-metadata.rst @@ -344,9 +344,10 @@ Properties be redacted in the ``pump-completed``/``pump-completed`` events in the execution-log. The purpose of this setting is to redact enough of the secrets to render them safe to log, but to potentially leave some of the secret for debugging purposes. - A value of ``-1`` means to redact all bytes of the secrets. Note that the redaction is only a best-effort - attempt to prevent secrets from ending up in the logs, there may be cases where secrets leak through in any - case, so it is best to always check that what ends up being logged looks ok. + A value of ``-1`` means to redact all bytes of the secrets. + + .. note:: + The redaction is only a best-effort attempt to prevent secrets from ending up in the logs, there may be cases where secrets leak through in any case, so it is best to always check that what ends up being logged looks ok. - 600 - No diff --git a/hub/documentation/service-configuration/pipes/configuration-sources-merge.rst b/hub/documentation/service-configuration/pipes/configuration-sources-merge.rst index b91013f2e4..d8a5904a20 100644 --- a/hub/documentation/service-configuration/pipes/configuration-sources-merge.rst +++ b/hub/documentation/service-configuration/pipes/configuration-sources-merge.rst @@ -64,7 +64,7 @@ Prototype Variant 1: Explicit equality-rules with the ``equality`` property """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -:: +.. code-block :: json { "type": "merge", @@ -80,7 +80,7 @@ Variant 1: Explicit equality-rules with the ``equality`` property Variant 2: Implicit equality-rules with the ``equality_sets`` property """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -:: +.. code-block :: json { "type": "merge", @@ -266,46 +266,56 @@ Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Equality is resolved transitively, so if A is the same as B and B is the same as C then A, B and C are all considered the same. With the ``equality`` property, these rules must be specified -one at a time, like this:: +one at a time, like this: - "equality": [ - ["eq", "a.x", "b.x"], - ["eq", "b.x", "c.y"], - ["eq", "c.z", "d.z"], - ], +.. code-block :: json + + "equality": [ + ["eq", "a.x", "b.x"], + ["eq", "b.x", "c.y"], + ["eq", "c.z", "d.z"], + ], The ``equality_sets`` property was added as a way to makes it clearer which equality-rules belong together. -The equality-rules above could be expressed like this:: +The equality-rules above could be expressed like this: - "equality_sets": [ - ["a.x", "b.x", "c.y"], - ["c.z", "d.z"], - ], +.. code-block :: json + + "equality_sets": [ + ["a.x", "b.x", "c.y"], + ["c.z", "d.z"], + ], Note that the ``equality_sets`` property is just a bit of syntactic sugar; behind the scenes the implicit equality-rules are added to the rules in the ``equality`` property. This means that you can use both the ``equality_sets`` and the ``equality`` property at the same time if you want (although this is not recommended, since it makes it harder to figure out the equality-rules). It also means that you will not get a configuration warning if -if you accidentally specify two equality-sets that are actually overlapping. If you for example specify this:: +if you accidentally specify two equality-sets that are actually overlapping. If you for example specify this: - "equality_sets": [ - ["a.x", "b.x", "c.y"], - ["c.y", "d.y"], - ], +.. code-block :: json -you won't actually get two equality-sets, since behind the scenes you end up with these equality-rules:: + "equality_sets": [ + ["a.x", "b.x", "c.y"], + ["c.y", "d.y"], + ], - "equality": [ - ["eq", "a.x", "b.x"], - ["eq", "b.x", "c.y"], - ["eq", "c.y", "d.y"] - ], +you won't actually get two equality-sets, since behind the scenes you end up with these equality-rules: -, which is equivalent to specifying a single equality-set, like this:: +.. code-block :: json - "equality_sets": [ - ["a.x", "b.x", "c.y", "d.y"], - ], + "equality": [ + ["eq", "a.x", "b.x"], + ["eq", "b.x", "c.y"], + ["eq", "c.y", "d.y"] + ], + +which is equivalent to specifying a single equality-set, like this: + +.. code-block :: json + + "equality_sets": [ + ["a.x", "b.x", "c.y", "d.y"], + ], Continuation support ^^^^^^^^^^^^^^^^^^^^ @@ -336,7 +346,7 @@ that uses the ``merge`` source. Dataset ``A``: -:: +.. code-block :: json [ {"_id": "a1", "f1": 1}, @@ -345,7 +355,7 @@ Dataset ``A``: Dataset ``B``: -:: +.. code-block :: json [ {"_id": "b1", "f1": 1, "f2": "x"}, @@ -354,7 +364,7 @@ Dataset ``B``: Dataset ``C``: -:: +.. code-block :: json [ {"_id": "c1", "f3": "X"}, @@ -365,7 +375,7 @@ Dataset ``C``: Pipe configuration: -:: +.. code-block :: json { "_id": "result", @@ -381,7 +391,7 @@ Pipe configuration: Given the above we should expect an output that looks like this: -:: +.. code-block :: json [ {"$ids": ["a1", "b1", "c1"], "_id": "0|a1|1|b1|2|c1", "_updated": 0, diff --git a/hub/documentation/service-configuration/pipes/configuration-sources.rst b/hub/documentation/service-configuration/pipes/configuration-sources.rst index cba7e403f0..9689930b15 100644 --- a/hub/documentation/service-configuration/pipes/configuration-sources.rst +++ b/hub/documentation/service-configuration/pipes/configuration-sources.rst @@ -16,13 +16,13 @@ Prototype The following *JSON* snippet shows the general form of a source definition. -..code-block::json +.. code-block:: json - { - "type": "a-source-type", - "comment": "This is a comment" + { + "type": "a-source-type", + "comment": "This is a comment" - } + } The only universally required property is ``type``. @@ -64,9 +64,9 @@ the Sesam components and it is assumed to be interpretable *only by the source*. Within an entity the marker is carried in the ``_updated`` property if supported by its source. - .. important:: +.. important:: - When using continuation support, Sesam will not be able to do automatic deletion tracking. If you wish to include deleted entities in your import, make sure you regularly set a full sync on the imported data. + When using continuation support, Sesam will not be able to do automatic deletion tracking. If you wish to include deleted entities in your import, make sure you regularly set a full sync on the imported data. @@ -201,9 +201,9 @@ The table below shows which strategy is chosen depending on the value of the pro :header-rows: 1 :widths: 25, 25, 25, 25 - * - ``supports_since`` - - ``is_since_comparable`` - - ``is_chronological`` + * - supports_since + - is_since_comparable + - is_chronological - Strategy * - ``false`` @@ -298,6 +298,8 @@ If you wish to activate continuation support for a :ref:`microservice +| + The microservice needs to pass on an entity property named "_updated" to Sesam for each entity from the source. This property should take the value corresponding to the time-stamp or sequence value of the source data representing the last data update for that entity (the same column as for the "updated_column" for SQL type sources). When the entities have been passed on into Sesam, the inbound pipe will go through all these "_updated" values and pick the max value as the new "pipe_offset". The first time the inbound pipe runs (or if the pipe is reset), the "pipe_offset" will not have a value, resulting in a complete import of all the data from the endpoint. Once data has been imported, the new "pipe_offset" will get passed to the microservice as the query parameter "since". This parameter can in turn be used as a query parameter to the API ensuring that only data updated after the last "since" value will be included in the GET request. An example of this is shown in the Python code snippet below. @@ -335,6 +337,8 @@ The first time the inbound pipe runs (or if the pipe is reset), the "pipe_offset +| + In this case the data from the source is not ordered chronologically, which means we can not use the "is_chronological" tag. The benefit of chronologically ordered data in the source system is that if the pipe's pump for some reason should fail in the middle of a request, Sesam can use the chronological order of the source data to continue requesting data from the last received entity. If the data is not ordered, Sesam has to re-run the whole last request. Type of sources diff --git a/hub/documentation/service-configuration/systems/configuration-systems-elasticsearch.rst b/hub/documentation/service-configuration/systems/configuration-systems-elasticsearch.rst index dbb3a69363..bdb2b039d8 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-elasticsearch.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-elasticsearch.rst @@ -10,7 +10,7 @@ used in conjunction with the :ref:`Elasticsearch sink `. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "id-of-system", @@ -62,7 +62,7 @@ Properties Example configuration ^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block :: json { "_id": "our-elasticsearch-server", @@ -75,7 +75,7 @@ Example configuration Example configuration with authentication ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block :: json { "_id": "our-elasticsearch-server", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-ldap.rst b/hub/documentation/service-configuration/systems/configuration-systems-ldap.rst index a386af64d7..012e288faf 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-ldap.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-ldap.rst @@ -14,7 +14,7 @@ It supports the following properties: Prototype ^^^^^^^^^ -:: +.. code-block :: json { "host": "FQDN of LDAP host", @@ -108,7 +108,7 @@ Properties Example configuration ^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block :: json { "_id": "example_ldap", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-microservice.rst b/hub/documentation/service-configuration/systems/configuration-systems-microservice.rst index 89a477f948..1451c9fc8c 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-microservice.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-microservice.rst @@ -261,7 +261,7 @@ service: Example configuration ^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block :: json { "_id": "our-http-server", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-mssql-legacy.rst b/hub/documentation/service-configuration/systems/configuration-systems-mssql-legacy.rst index ac43e7a860..b48ae9a34e 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-mssql-legacy.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-mssql-legacy.rst @@ -15,7 +15,7 @@ are supported and how they are mapped to :ref:`Sesam types `. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "sql_system_id", @@ -97,7 +97,7 @@ Example configuration Example MS SQL Server configuration: -:: +.. code-block :: json { "_id": "sqlserver_db", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-mssql.rst b/hub/documentation/service-configuration/systems/configuration-systems-mssql.rst index 2150ed0c4a..b3502e3e37 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-mssql.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-mssql.rst @@ -17,7 +17,7 @@ are supported and how they are mapped to :ref:`Sesam types `. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "sql_system_id", @@ -85,7 +85,7 @@ Example configuration Example MS SQL Server configuration: -:: +.. code-block :: json { "_id": "sqlserver_db", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-mysql.rst b/hub/documentation/service-configuration/systems/configuration-systems-mysql.rst index 6ac6e71bf6..f107e7f434 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-mysql.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-mysql.rst @@ -11,7 +11,7 @@ how they are mapped to :ref:`Sesam types `. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "sql_system_id", @@ -72,7 +72,7 @@ Example configuration Example MySQL configuration: -:: +.. code-block :: json { "_id": "sqlserver_db", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-oracle-tns.rst b/hub/documentation/service-configuration/systems/configuration-systems-oracle-tns.rst index 3bba369cef..57da7f9894 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-oracle-tns.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-oracle-tns.rst @@ -11,7 +11,7 @@ and how they are mapped to :ref:`Sesam types `. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "sql_system_id", @@ -69,7 +69,7 @@ Example configuration Example Oracle TNS configuration: -:: +.. code-block :: json { "_id": "oracle_db", @@ -77,7 +77,7 @@ Example Oracle TNS configuration: "type": "system:oracle_tns", "username": "$ENV(username-variable)", "password": "$SECRET(password-variable)", - "tns_name": "(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = foo)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = BAR)))"", + "tns_name": "(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = foo)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = BAR)))", "coerce_to_decimal": true } diff --git a/hub/documentation/service-configuration/systems/configuration-systems-oracle.rst b/hub/documentation/service-configuration/systems/configuration-systems-oracle.rst index b4734dc846..b4e8f12901 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-oracle.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-oracle.rst @@ -10,7 +10,7 @@ and how they are mapped to :ref:`Sesam types `. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "sql_system_id", @@ -81,7 +81,7 @@ Example configuration Example Oracle configuration: -:: +.. code-block :: json { "_id": "oracle_db", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-postgresql.rst b/hub/documentation/service-configuration/systems/configuration-systems-postgresql.rst index 5180150d49..eca0d1e99f 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-postgresql.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-postgresql.rst @@ -10,18 +10,18 @@ and how they are mapped to :ref:`Sesam types `. Prototype ^^^^^^^^^ -:: - - { - "_id": "sql_system_id", - "type": "system:postgresql, - "name": "The PostgreSQL Database", - "username":"$ENV(username-variable)", - "password":"$SECRET(password-variable)", - "host":"fqdn-or-ip-address-here", - "port": 5432, - "database": "database-name", - "sslmode": "prefer" +.. code-block :: json + + { + "_id": "sql_system_id", + "type": "system:postgresql", + "name": "The PostgreSQL Database", + "username":"$ENV(username-variable)", + "password":"$SECRET(password-variable)", + "host":"fqdn-or-ip-address-here", + "port": 5432, + "database": "database-name", + "sslmode": "prefer" } Properties @@ -80,7 +80,7 @@ Example configuration Example PostgreSQL configuration: -:: +.. code-block :: json { "_id": "postgresql_db", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-rest.rst b/hub/documentation/service-configuration/systems/configuration-systems-rest.rst index cfee4d50c8..2364300fb7 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-rest.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-rest.rst @@ -18,7 +18,7 @@ apply. We'll only cover the REST system specific properties in this section. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "id-of-system", @@ -652,7 +652,7 @@ rendered template, then the property is omitted from its parent object. Note tha An example: -:: +.. code-block :: json { "_id": "our-rest-service", @@ -670,7 +670,6 @@ An example: "some_other_key{% if entity.other_conditional_property is not defined %}sesam:markskip{% endif %}": "other_value" } } - .. (experimental) @@ -678,7 +677,7 @@ You can use the special marker ``"sesam:markjson"`` to construct JSON objects, l An example: -:: +.. code-block :: json { "_id": "our-rest-service", @@ -695,28 +694,30 @@ An example: "some_other_key": "[{{ properties.arg1, \"literal value \"}}]sesam:markjson" } } - .. + + + Result payload object: -:: +.. code-block :: json - .. + { "payload": { "key": 10, - "some_other_key": [1.2, \"literal value \"] + "some_other_key": [1.2, "literal value"] } - .. + } When using the ``custom_auth`` feature, the response properties from the authentication request(s) are available under the ``token`` object. Use this to construct the payload/headers/parameters for the operations, e.g. for a system that uses the bearer token format: -:: +.. code-block :: json { "_id": "webcrm", @@ -725,7 +726,6 @@ system that uses the bearer token format: "headers": { "Authorization": "Bearer {{ token.access_token }}" }, - .. See the :ref:`example configurations ` for more examples on systems that use ``custom_auth``. diff --git a/hub/documentation/service-configuration/systems/configuration-systems-smtp.rst b/hub/documentation/service-configuration/systems/configuration-systems-smtp.rst index c6dee45917..59cee695c7 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-smtp.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-smtp.rst @@ -10,7 +10,7 @@ and send emails based on the entities it receives. Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "id-of-system", @@ -86,7 +86,7 @@ Example configuration "smtp_server": "localhost", "smtp_port": 25, "smtp_username": "$ENV(username-variable)", - "smtp_password": $SECRET(password-variable) + "smtp_password": $SECRET(password-variable), "max_per_hour": 100000 } diff --git a/hub/documentation/service-configuration/systems/configuration-systems-solr.rst b/hub/documentation/service-configuration/systems/configuration-systems-solr.rst index a1d25074c6..39075dbdc8 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-solr.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-solr.rst @@ -9,7 +9,7 @@ server for indexing JSON documents. It is used in conjunction with the :ref:`Sol Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "id-of-system", @@ -47,7 +47,7 @@ Properties Example configuration ^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block :: json { "_id": "our-solr-server", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-sql.rst b/hub/documentation/service-configuration/systems/configuration-systems-sql.rst index ae739d9406..bcdb8bf835 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-sql.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-sql.rst @@ -14,7 +14,7 @@ The common properties for all SQL systems are: Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "sql_system_id", diff --git a/hub/documentation/service-configuration/systems/configuration-systems-twilio.rst b/hub/documentation/service-configuration/systems/configuration-systems-twilio.rst index 8619d36b6a..39a701a198 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems-twilio.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems-twilio.rst @@ -11,7 +11,7 @@ It has the following properties: Prototype ^^^^^^^^^ -:: +.. code-block :: json { "_id": "system-id", @@ -65,7 +65,7 @@ Properties Example configuration ^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block :: json { "_id": "twilio_service", diff --git a/hub/documentation/service-configuration/systems/configuration-systems.rst b/hub/documentation/service-configuration/systems/configuration-systems.rst index d8b82d35e5..1f38617863 100644 --- a/hub/documentation/service-configuration/systems/configuration-systems.rst +++ b/hub/documentation/service-configuration/systems/configuration-systems.rst @@ -16,7 +16,7 @@ All systems share a number of common properties: Prototype --------- -:: +.. code-block :: json { "_id": "a_system_id",